TCP, UDP and ICMP — the protocols every firewall rule decides on

The firewall does not understand your application. It understands the protocol your application uses. Almost everything on the network is one of three: TCP, UDP, or ICMP. The firewall operator who knows what those protocols actually do — not just “TCP is reliable, UDP is unreliable” — diagnoses problems faster.

TCP: stateful by design

TCP establishes a connection with a three-way handshake. The firewall sees the SYN, SYN-ACK, ACK and creates state for that flow. Every subsequent packet matches the state. When the connection ends, the firewall removes the state (after a timeout).

Client                                         Server
  | --- SYN seq=x ----------------------->       |
  | <-- SYN-ACK seq=y, ack=x+1 ---------         |
  | --- ACK seq=x+1, ack=y+1 ---------->       |
  |  (connection established; data flows)        |

PF creates the state on the SYN-ACK (the second packet from the server) when the rule allows the packet. A rule that permits “any → any” creates state on the SYN; a rule that permits only established connections allows the SYN-ACK back because there is already a state.

This is why the standard OPNsense LAN rule is “allow LAN net → any”. That rule creates state on the first SYN. The matching return packets come back as state-matched, and the firewall allows them without a separate inbound rule.

TCP flags the firewall can match on

PF can match on individual TCP flags. The most common reason to do so is to defeat stateless scans (nmap, etc.) that send packets with unusual flag combinations:

A SYN with no ACK is a connection attempt. A SYN+ACK is the server’s response. A FIN is a clean close. A RST is an abrupt close. OPNsense’s default state-matching rules treat any flag combination as part of the state; specific rule matching on flags is for unusual cases (blocking outbound SYN that doesn’t match an established connection, for example).

UDP: connectionless

UDP has no handshake. A UDP packet is just a payload with a source and destination port. The firewall cannot tell from a single UDP packet whether it is the start of a flow, the middle, or the end.

PF still creates “state” for UDP, but it is a soft state: PF records the first UDP packet’s tuple (src IP, src port, dst IP, dst port) and allows subsequent packets in either direction that match the tuple. The state times out quickly (default 60 seconds) and is not refreshed by application-level activity. There is no “UDP connection” — the firewall is inferring one.

$ pfctl -s state | grep udp

all udp 192.0.2.50:51820 -> 8.8.8.8:53       SINGLE:NO_TRAFFIC

ICMP: more than ping

ICMP is not “ping”. ICMP carries:

• Echo request / echo reply (type 8 / type 0) — the ping protocol.
• Destination unreachable (type 3) — returned by a router when the destination cannot be reached. Includes subtypes: network unreachable (code 0), host unreachable (code 1), port unreachable (code 3, from UDP), fragmentation needed (code 4, also known as “ICMP must fragment” — used in MTU discovery).
• Time exceeded (type 11) — returned by a router when the TTL hits zero (traceroute works because of this).
• Redirect (type 5) — sent by a router to tell a host about a better next hop.
• Source quench (type 4) — deprecated, used for congestion control in the 1980s, ignored now.
• Router advertisement / solicitation (type 9 / type 10) — used by IPv4 router discovery, mostly deprecated.

The firewall operator’s job is to remember that dropping ICMP breaks more than ping. Specifically:

• Dropping ICMP type 3 code 4 (fragmentation needed) breaks Path MTU Discovery. The result is large TCP connections silently fail.
• Dropping ICMP type 11 (time exceeded) breaks traceroute.
• Dropping all ICMP often breaks QUIC, IPsec NAT-T, and certain multi-path protocols.

The safe default is to rate-limit ICMP, not block it. OPNsense lets you permit ICMP to the firewall itself and from internal hosts, while limiting its rate to something reasonable.

Why the firewall operator must read protocol evidence

When a firewall incident lands at 03:00, the operator needs to know which protocol is in trouble to know where to look. The rules of thumb:

• TCP hangs → routing, MTU, NAT, state, asymmetric path
• UDP “works then stops” → state timeout, asymmetric path
• ICMP “doesn’t work” → usually intentional, but check PMTUD if large TCP also fails
• “Some hosts work, some don’t” → look at source or destination — usually one host is on the wrong VLAN