TCP, UDP and ICMP — the transport protocols the routing decisions affect

The routing engineer needs to know the transport protocols because the firewall does, the NAT engine does, and the routing policy does. The IP packet that arrives at the router contains a TCP or UDP segment, and the firewall’s decision to accept or drop the packet depends on that segment’s port numbers and flags. The NAT engine rewrites those ports when traffic crosses the boundary. The routing engine, when configured to do policy routing on source port, matches against them.

This lesson is the transport-level grounding the routing engineer must have before configuring firewall, NAT, or policy routing. It is not a TCP tutorial — those exist elsewhere — but a working operator’s reference for the parts of TCP, UDP, and ICMP that surface in routing and firewall incidents.

Three protocols the routing engineer handles daily

flowchart TB
  subgraph "IP packet"
    IP["IP header\n(protocol)"]
    L4["L4 header"]
    PAY["Payload"]
  end

  IP --> P6["6 = TCP"]
  IP --> P17["17 = UDP"]
  IP --> P1["1 = ICMP"]

The IP header’s protocol field selects what the kernel does with the packet:

• 1 (ICMP): handled by the kernel’s ICMP subsystem. Most ICMP messages are diagnostic and must often be permitted through the firewall.
• 6 (TCP): the connection-oriented protocol. Stateful firewalls track TCP connections in conntrack (NEW, ESTABLISHED, RELATED, INVALID).
• 17 (UDP): the connectionless protocol. Connection tracking is per-flow pseudo-state; the firewall times out UDP flows faster than TCP because there is no FIN/RST to mark end-of-stream.

There are other protocols (GRE 47, ESP 50 for IPsec, ICMPv6 58, OSPF 89, VRRP 112) but TCP, UDP, and ICMP account for the vast majority of operator-visible traffic and incident reports.

What the routing engineer needs to know about TCP

TCP is the protocol that “looks like” a connection. It is not a connection at the IP layer — IP is connectionless — but TCP makes it look like one to the application by maintaining state at both ends.

The three-way handshake establishes a TCP connection:

sequenceDiagram
  autonumber
  participant C as Client
  participant S as Server

  C->>S: SYN (seq=x)
  S->>C: SYN, ACK (seq=y, ack=x+1)
  C->>S: ACK (seq=x+1, ack=y+1)
  Note over C,S: Connection ESTABLISHED

A TCP segment carries sequence numbers, acknowledgement numbers, flags (SYN, ACK, FIN, RST, PSH, URG), and a window size. The stateful firewall on VyOS tracks each of these — it knows whether a packet is the start of a new connection (SYN with no prior state), part of an established connection (matching ACK in conntrack), or an attempt to close (FIN) or reset (RST) it.

ss -tan state established
ss -tan state time-wait
sudo conntrack -L -p tcp --src 10.0.0.5

The four states the operator must remember:

• NEW — a packet that does not match an existing conntrack entry but is valid. The firewall decides whether to accept the new flow.
• ESTABLISHED — a packet that matches an existing entry. The firewall usually accepts ESTABLISHED traffic automatically.
• RELATED — a packet that is part of a different but related connection (e.g. an ICMP error response to a TCP SYN, or an FTP data connection to an FTP control). The firewall accepts RELATED traffic.
• INVALID — a packet that does not match any conntrack state. The firewall should drop INVALID by default.

What the routing engineer needs to know about UDP

UDP is connectionless. Each segment is independent. There is no handshake, no sequence number, no FIN. The kernel’s conntrack subsystem tracks UDP flows as if they were connections — it creates a NEW entry on the first packet and removes the entry after a timeout — but this is best-effort, not authoritative.

UDP is used by:

• DNS (port 53)
• NTP (port 123)
• SNMP (port 161 / 162)
• Syslog (port 514)
• WireGuard (port 51820)
• IPsec IKE (UDP 500, UDP 4500 for NAT-Traversal)
• OSPF does not use UDP; it uses IP protocol 89 directly
• BGP uses TCP (port 179), not UDP
• VRRP uses IP protocol 112 directly

A firewall rule for UDP is more permissive than for TCP because the firewall cannot validate the connection’s legitimacy — there is no handshake. The trade-off is to make UDP timeouts short so a conntrack entry does not accumulate indefinitely.

sudo conntrack -L -p udp | head
sudo conntrack -L -p udp | wc -l

What the routing engineer needs to know about ICMP

ICMP is the protocol that carries diagnostic and error messages for IP. It is not a transport protocol — there are no ports — but the routing engineer handles it daily because:

• ping uses ICMP Echo Request (type 8) and Echo Reply (type 0).
• traceroute uses ICMP Time Exceeded (type 11) — every router that decrements the IP TTL to zero replies with this message.
• Destination Unreachable (type 3) tells the sender why the packet could not be delivered. The code field gives the reason: 0 = net unreachable, 1 = host unreachable, 2 = protocol unreachable, 3 = port unreachable, 4 = fragmentation needed and DF set.
• ICMP is also used by routing protocols (OSPF uses OSPF-specific messages, not ICMP, but other protocols use ICMP).

flowchart LR
  H["Source host"] -->|"ICMP Echo Request"| R["VyOS router"]
  R -->|"ICMP Echo Reply"| H
  H -->|"traceroute packet, TTL=1"| R
  R -->|"ICMP Time Exceeded"| H

The routing engineer must permit ICMP carefully. The firewall must allow:

• Echo Request / Echo Reply to/from the router’s own addresses (for monitoring).
• Time Exceeded for traceroute to work.
• Destination Unreachable from the router (for the host to know the path failed).
• Fragmentation Needed (type=3, code=4) for PMTUD to work — this is the message that tells the sender to reduce its packet size.

A common production mistake is to drop all ICMP at the firewall. This breaks PMTUD: hosts cannot learn that the path MTU is smaller than they assumed, and large packets are silently dropped without any error feedback. The lesson on MTU covers this in detail.

MSS clamping, PMTUD, and the transport-layer MTU

The IP packet can be up to 64 KB, but Ethernet has a 1500-byte MTU, and every tunnel adds overhead. The kernel needs to know the path MTU to avoid fragmentation, and the standard mechanism is Path MTU Discovery (PMTUD).

PMTUD works by setting the “Don’t Fragment” (DF) bit on every packet. If a router along the path cannot forward the packet because the next-hop’s MTU is smaller, the router sends back an ICMP Fragmentation Needed (type=3, code=4) and discards the packet. The sender reads the MTU from the ICMP payload and reduces its packet size.

sequenceDiagram
  autonumber
  participant H as Host (MTU 1500)
  participant R1 as VyOS Router 1
  participant T as Tunnel (MTU 1400)
  participant R2 as VyOS Router 2
  participant S as Server (MTU 1500)

  H->>R1: packet, size=1460, DF=1
  R1->>T: packet, size=1460, DF=1
  T->>R2: packet, size=1460 (MTU exceeded)
  R2->>H: ICMP Fragmentation Needed, MTU=1400
  Note over H: PMTUD: path MTU is 1400
  H->>R1: packet, size=1360, DF=1
  R1->>T: packet, size=1360, DF=1
  T->>R2: packet, size=1360
  R2->>S: packet, size=1360
  S-->>H: reply

PMTUD is fragile because the ICMP feedback can be filtered. Most operators configure MSS clamping on the tunnel interface as a workaround: the firewall rewrites the TCP MSS option in the SYN packet to a value that fits inside the tunnel MTU, so the connection never attempts a packet larger than the tunnel can carry.

The VyOS configuration lives on the interface, under its ip sub-tree:

configure
set interfaces wireguard wg0 ip adjust-mss '1380'
set interfaces wireguard wg0 ipv6 adjust-mss '1360'
commit
save

Three things about that command are worth holding onto:

• adjust-mss is a per-interface node, not a firewall rule. Every interface type that carries the common ip sub-tree has it — ethernet, bonding, tunnel, vti, wireguard — so you clamp on whichever interface is the constrained hop, which is usually the tunnel rather than the LAN.
• ip and ipv6 are separate nodes with separate values. Clamping v4 and forgetting v6 produces a dual-stack router where IPv4 works and IPv6 hangs on large transfers, which is a genuinely unpleasant afternoon.
• The value can be the literal clamp-mss-to-pmtu instead of a number, which tells the kernel to derive the clamp from the route’s PMTU rather than from a number you calculated by hand. Use it when the overhead is not fixed; use a number when you know it and want it stable.

On VyOS 1.3 this was set firewall options interface wg0 adjust-mss, under the firewall rather than the interface. 1.4 moved it to the interface and 1.5 kept it there. If you find firewall options in an old config archive, that is what it was.

This is a workaround for the underlying PMTUD failure, not a replacement for it. The lesson on MTU covers the full treatment.

What “filtering ICMP” actually does to the network

A blanket ICMP drop at the firewall breaks:

• PMTUD (the host cannot learn the path MTU)
• Traceroute (the host cannot see the path)
• The host’s own diagnostic of routing failure (no Destination Unreachable feedback)
• The router’s ability to communicate control-plane ICMP

The correct firewall posture is selective. On VyOS 1.5 that is a named ruleset plus a rule in a base hook that jumps to it:

set firewall ipv4 name INBOUND-INTERNET default-action 'drop'

set firewall ipv4 name INBOUND-INTERNET rule 10 action 'accept'
set firewall ipv4 name INBOUND-INTERNET rule 10 description 'ping the router'
set firewall ipv4 name INBOUND-INTERNET rule 10 protocol 'icmp'
set firewall ipv4 name INBOUND-INTERNET rule 10 icmp type-name 'echo-request'

set firewall ipv4 name INBOUND-INTERNET rule 11 action 'accept'
set firewall ipv4 name INBOUND-INTERNET rule 11 description 'traceroute return path'
set firewall ipv4 name INBOUND-INTERNET rule 11 protocol 'icmp'
set firewall ipv4 name INBOUND-INTERNET rule 11 icmp type-name 'time-exceeded'

set firewall ipv4 name INBOUND-INTERNET rule 12 action 'accept'
set firewall ipv4 name INBOUND-INTERNET rule 12 description 'PMTUD — do not remove'
set firewall ipv4 name INBOUND-INTERNET rule 12 protocol 'icmp'
set firewall ipv4 name INBOUND-INTERNET rule 12 icmp type-name 'fragmentation-needed'

set firewall ipv4 name INBOUND-INTERNET rule 13 action 'accept'
set firewall ipv4 name INBOUND-INTERNET rule 13 description 'path failure feedback'
set firewall ipv4 name INBOUND-INTERNET rule 13 protocol 'icmp'
set firewall ipv4 name INBOUND-INTERNET rule 13 icmp type-name 'destination-unreachable'

That ruleset is inert on its own. Nothing reaches it until a base hook sends traffic there, and which hook you pick decides which ICMP you are actually filtering:

set firewall ipv4 input filter rule 100 action 'jump'
set firewall ipv4 input filter rule 100 jump-target 'INBOUND-INTERNET'
set firewall ipv4 input filter rule 100 inbound-interface name 'eth0'

input filter is traffic addressed to the router — the echo request someone sends to the WAN address, the ICMP error a distant router sends back about a packet the router itself originated. Traffic passing through the router is forward filter, and that is where the PMTUD case lives: the fragmentation-needed message travelling back to a host behind you is transit traffic, not traffic to the router. A rule set that only ever gets jumped to from input filter will not save a downstream host’s PMTUD.

The specific ICMP types to allow depend on the network’s operational posture; a router handling ICMP traffic destined for itself needs Echo Request, a router passing traffic needs Time Exceeded and Fragmentation Needed.

Operational commands

The operator verifies transport-layer state with:

ss -tan                        # TCP sockets
ss -uan                        # UDP sockets
ss -tulpn                      # listening sockets with process info
sudo conntrack -L              # the conntrack table, one line per flow
sudo conntrack -L expect       # the expectation table, which is what RELATED consults
sudo conntrack -E              # live event stream: entries as they are created and destroyed
sudo tcpdump -ni eth0 'tcp and dst port 22'
sudo tcpdump -ni eth0 'icmp'

ss is the modern replacement for netstat. conntrack is the conntrack subsystem’s debug tool — note that -L expect and -E are different things, and the pair is easy to mix up. -L expect dumps the expectation table, the short-lived entries a helper module creates so that a not-yet-seen connection will be classified RELATED when it arrives. -E is an event stream: it prints nothing until something changes, then prints every create and destroy as it happens, which is what you want when the question is “is this flow being tracked at all”. tcpdump is the canonical packet capture.

From the VyOS CLI rather than the shell, the same ground is covered by show conntrack table ipv4, show conntrack statistics, and monitor traffic interface eth0 for the capture.

Validation

The diagnostic checklist when traffic is not flowing:

1. Confirm the routing lookup returns the expected path (ip route get).
2. Confirm the neighbour entry is REACHABLE (ip neigh show).
3. Confirm the firewall allows the traffic in both directions — and confirm the ruleset you are reading is one something actually jumps to (show firewall ipv4 forward filter, then the named set).
4. Confirm the conntrack table is not full and the connection is tracked (show conntrack statistics).
5. Capture the wire to prove packets are arriving and leaving (monitor traffic interface eth0).
6. If TCP: confirm the three-way handshake completed and the connection is in ESTABLISHED state (ss -tan state established).
7. If UDP: confirm conntrack has an entry for the flow (sudo conntrack -L -p udp).
8. If ICMP: confirm the firewall is not dropping the message type you need, and that the drop is not the kernel’s own icmp_ratelimit budget rather than a rule.

Cross-course references

• The Linux course’s XIX-Linux-NetFoundations and XXII-Linux-NetTroubleshoot cover the same primitives from the host perspective.
• The OPNsense course covers the FreeBSD-specific transport-layer state tracking, including pf state table inspection.
• The Observability course covers Prometheus blackbox exporter, which uses ICMP/TCP probes as a synthetic monitoring primitive.

Quiz

Production discipline

Transport-layer understanding is what separates the operator who sees “traffic not flowing” as a routing problem from the operator who sees it as a firewall problem or a NAT problem or a conntrack problem or an MTU problem. The right first move is to read the transport layer of the failing packets, not to “fix the routing”.

Plan the firewall rules once, including the ICMP types the path needs. Plan the conntrack sizing once. Plan the MSS clamping once. Then the path either works or fails predictably, and the diagnostic surface is small.