PMTUD — RFC 1191, RFC 8201, ICMP Frag Needed, black hole detection, MTU 1280 floor

Path MTU Discovery (PMTUD) is the protocol by which an end-host learns the maximum packet size that can traverse the path to a destination without fragmentation. Without PMTUD, the host sends the largest packet it can (often 1500 bytes) and hopes for the best; with PMTUD, the host sends a packet with the Don’t Fragment (DF) flag set, and if a router in the path cannot forward the packet, that router sends back an ICMP Fragment Needed message with the bottleneck MTU. The host then reduces its effective MTU to the bottleneck value.

This lesson is the production reference for PMTUD on VyOS 1.5 LTS: the protocol details, the IPv4 vs IPv6 differences, the “PMTUD black hole” failure mode, and the canonical diagnostic and fix.

The PMTUD protocol

sequenceDiagram
  participant Host as Source host
  participant R1 as Router 1 (MTU 9000)
  participant R2 as Router 2 (MTU 1500)
  participant Dest as Destination

  Host->>R1: Packet, size=9000, DF=1
  R1->>R2: Forward
  R2->>Host: ICMP Frag Needed, MTU=1500
  Host->>R1: Packet, size=1500, DF=1
  R1->>R2: Forward
  R2->>Dest: Forward
  Dest-->>Host: ACK

The protocol:

1. The source sends a packet with DF=1 (Don’t Fragment) at its interface MTU.
2. Every router along the path checks whether the packet exceeds its outgoing interface MTU.
3. If a router cannot forward the packet (because the packet exceeds the outgoing MTU), it sends back an ICMP Fragment Needed message (IPv4 type 3 code 4) or ICMPv6 Packet Too Big (type 2), including the bottleneck MTU in the message.
4. The source receives the ICMP, reduces its effective MTU to the bottleneck value, and retransmits.
5. The source caches the path MTU in its routing cache for the destination.

For IPv4 (RFC 1191), fragmentation is allowed at the source; the source can also choose to fragment before sending. For IPv6 (RFC 8201), routers do not fragment; if the packet exceeds the outgoing MTU, the router drops it and sends ICMPv6 Packet Too Big. The source must reduce the packet size.

The IPv6 PMTUD differences

IPv6 has stricter PMTUD semantics:

• No router-side fragmentation. IPv6 routers do not fragment packets. If a packet exceeds the outgoing MTU, the router drops it and sends ICMPv6 Packet Too Big (RFC 4443 type 2).
• Minimum MTU of 1280. RFC 8200 requires every link to support a minimum MTU of 1280. PMTUD cannot reduce the path MTU below 1280 for IPv6; if the discovered path MTU is below 1280, the IPv6 layer must fragment locally (or drop the packet).
• PLPMTUD (Packetization Layer PMTUD). RFC 8899 specifies a PMTUD variant that does not rely on ICMP — useful when ICMP is filtered. PLPMTUD is increasingly supported in modern TCP stacks.

The discipline: for IPv6 traffic, PMTUD is mandatory; if ICMPv6 is filtered, PMTUD fails, and the operator must use PLPMTUD or MSS clamping.

The PMTUD black hole

A PMTUD black hole is a router or firewall that silently drops packets exceeding the outgoing MTU without sending back an ICMP Fragment Needed. The source never learns the bottleneck MTU; it keeps sending large packets that are silently dropped.

flowchart LR
  S["Source<br/>sends 1500-byte packets"] --> R1["R1<br/>forwards"]
  R1 --> R2["R2<br/>(bottleneck MTU 1400)<br/>DROPS without ICMP"]
  R2 --> D["Destination<br/>(never receives)"]

The classic causes:

1. Firewall drops ICMP. An over-zealous firewall rule blocks all ICMP, including Fragment Needed. The source never learns the bottleneck.
2. Router does not generate ICMP. A misconfigured router (or a router under CPU pressure) drops packets but does not generate the ICMP.
3. Tunnel endpoint swallows the ICMP. A packet that is too big for a tunnel is dropped at the tunnel’s outer hop, so the ICMP Frag Needed is addressed to the tunnel endpoint, not to the original source. The endpoint has to translate that into a fresh ICMP aimed at the inner source, and a tunnel implementation that does not track enough state to do so drops the news on the floor. There is no VyOS knob for this; the fix is on the tunnel interface (see below).
4. ISP filters ICMP for security. Some ISPs filter ICMP at their edge for DDoS protection; the ICMP never reaches the source.

The discipline: ICMP Fragment Needed is essential for PMTUD. A firewall that blocks all ICMP will break PMTUD for any path that traverses it. The fix is to allow ICMP Fragment Needed explicitly.

Diagnostic: a do-not-fragment ping

The canonical PMTUD diagnostic is a ping that sets DF and names a payload size. From the VyOS operational shell that is:

ping 198.51.100.1 size 1472 do-not-fragment count 3

size is the ICMP payload, so 1472 is the number that makes a 1500-byte frame: 1472 + 8 bytes of ICMP header + 20 bytes of IPv4 header. VyOS wraps iputils ping, so the same probe from any Linux host — including one behind the router, which is usually the host that is actually suffering — is -M do (set DF, never fragment) with -s for the same payload:

$ ping -M do -s 1472 198.51.100.1
PING 198.51.100.1 (198.51.100.1) 1472(1500) bytes of data.
From 10.0.0.1 icmp_seq=1 Frag needed and DF set (mtu = 1400)

Read that line twice, because both halves matter. Frag needed and DF set is the ICMP the bottleneck router sent; mtu = 1400 is the size it can actually forward, and the source now caches 1400 as the path MTU for this destination. Read the address too — From 10.0.0.1 names the hop whose MTU is the constraint, which is the device to go and fix. An error attributed to your own address instead (ping: local error: message too long) means the outgoing interface on the box you are typing on is the limit, and nothing upstream is involved at all.

If the ping hangs without the ICMP reply, the path has a PMTUD black hole:

$ ping -M do -s 1472 198.51.100.1
PING 198.51.100.1 (198.51.100.1) 1472(1500) bytes of data.
<no output, hangs>

The operator’s diagnostic ladder:

flowchart TD
  S["Symptom<br/>(large transfers stall,<br/>small ones fine)"]
  S --> Q1{"size 1472 do-not-fragment<br/>succeeds?"}
  Q1 -->|yes| Q5["Path MTU is at least 1500<br/>look elsewhere"]
  Q1 -->|no| Q2{"Frag Needed<br/>came back?"}
  Q2 -->|yes| Q3["PMTUD is working<br/>read the MTU it reports<br/>and fix that hop"]
  Q2 -->|no| Q4["PMTUD black hole"]
  Q4 --> Q5a{"Where is the ICMP lost?"}
  Q5a -->|"own firewall<br/>(counter proves it)"| F1["Permit icmp type 3 code 4<br/>in the chain that drops it"]
  Q5a -->|"upstream, not yours"| F2["ip adjust-mss on the WAN side"]
  Q5a -->|"a tunnel hop"| F3["Set tunnel mtu + adjust-mss<br/>on the tunnel interface"]

The canonical fix: MSS clamping

When PMTUD fails (because ICMP is filtered), the canonical fallback is MSS (Maximum Segment Size) clamping. MSS is the TCP-layer concept of “the largest TCP payload I can receive in a single segment”. MSS clamping rewrites the MSS in the TCP SYN handshake to fit the path MTU, so the sender never sends a TCP segment larger than the path allows.

sequenceDiagram
  participant Client
  participant Server
  participant Router as VyOS router<br/>(ip adjust-mss on eth0)

  Client->>Router: SYN, MSS=1460
  Router->>Server: SYN, MSS=1380 (clamped)
  Server->>Router: SYN-ACK, MSS=1380
  Router->>Client: SYN-ACK, MSS=1380
  Note over Client,Server: Both sides use MSS=1380<br/>(1380 + 40 TCP/IP = 1420 fits tunnel MTU)

On VyOS 1.5, clamping is a property of an interface, not of a firewall rule. It lives in the ip and ipv6 sub-trees that every routed interface type shares:

configure
set interfaces ethernet eth0 ip adjust-mss 1380
set interfaces ethernet eth0 ipv6 adjust-mss 1360
commit

adjust-mss takes either a number or the keyword clamp-mss-to-pmtu, which clamps against whatever the kernel currently believes the path MTU to be instead of a constant you have to recompute every time the encapsulation changes:

set interfaces ethernet eth0 ip adjust-mss clamp-mss-to-pmtu

Three consequences follow from clamping being per-interface:

• You clamp on the constrained hop. That is nearly always the tunnel — tunnel, vti, wireguard — rather than the LAN-facing Ethernet, because the tunnel is where the smaller MTU is. Clamping the LAN side of a router whose WAN is the bottleneck fixes nothing.
• It applies to every TCP flow crossing that interface. There is no source, port or ruleset selector. If you need MSS rewriting for some flows and not others, VyOS does not expose that; the honest workaround is to put those flows on their own interface.
• The two address families are separate nodes. ip adjust-mss does nothing for IPv6 TCP. The IPv6 value is normally 20 bytes lower, because the IPv6 header is 40 bytes against IPv4’s 20.

Part LI-04 covers MSS clamping in detail, including how to pick the number.

The kernel knobs, and which of them exist

VyOS has no CLI nodes for PMTUD behaviour. What it has is set system sysctl parameter <key> value <value>, which writes the sysctl through the configuration tree so it survives reboot and shows up in the configuration archive. The knobs worth knowing are the ones the Linux ip-sysctl documentation defines:

net.ipv4.ip_no_pmtu_disc      # 0 = PMTUD on (the default). 1/2/3 are degradation modes.
net.ipv4.route.min_pmtu       # 552 by default: the floor the kernel accepts from an ICMP.
net.ipv4.ip_forward_use_pmtu  # 0 by default: forwarded traffic ignores cached PMTU. Leave it.
net.ipv4.tcp_mtu_probing      # 0 off / 1 on when a black hole is detected / 2 always on.
net.ipv4.tcp_base_mss         # 1024: the MSS probing starts from in mode 2.
net.ipv4.tcp_mtu_probe_floor  # 48: the smallest MSS probing will search down to.

Read that list for what is not in it. There is no ip_dont_fragment_ignore, and there is no net.ipv6 counterpart to ip_no_pmtu_disc — IPv6 PMTUD is not switchable system-wide, because RFC 8200 leaves a router no legal alternative to Packet Too Big. A process can opt out for its own sockets with IPV6_MTU_DISCOVER, and that is the only lever there is.

The one that looks misfiled is net.ipv4.tcp_mtu_probing, and it is not: the net.ipv4.tcp_* knobs configure the whole TCP stack, IPv6 included, because there is one TCP implementation under both address families. Setting it fixes IPv6 black holes too, despite the name.

tcp_mtu_probing is PLPMTUD (RFC 4821): rather than believe an ICMP that may never arrive, TCP searches for the largest segment that gets acknowledged. Mode 1 arms it only after the stack has decided a connection looks black-holed, which is the conservative setting and the one to reach for on a router whose own management sessions cross a filtered path:

configure
set system sysctl parameter net.ipv4.tcp_mtu_probing value 1
commit

The operator validates against the running kernel rather than the config:

sysctl net.ipv4.tcp_mtu_probing

which prints net.ipv4.tcp_mtu_probing = 1.

Production failure modes

The PMTUD failure modes the operator encounters:

• PMTUD black hole from firewall. The local firewall drops ICMP Fragment Needed. Fix: allow ICMP type 3 code 4 explicitly.
• PMTUD black hole from ISP. The upstream ISP filters ICMP for DDoS protection. Fix: MSS clamping on the local router (the operator cannot fix the ISP).
• PMTUD black hole from tunnel. The ICMP is generated toward the tunnel endpoint rather than the inner source and is never translated inward. Fix: there is no “relay ICMP” option on a VyOS tunnel — set the tunnel interface MTU to the real ceiling and apply ip adjust-mss / ipv6 adjust-mss on that same interface, so TCP never builds a segment that needs the message.
• IPv6 minimum violation. A path with MTU below 1280 prevents IPv6. Fix: ensure every link in the IPv6 path has MTU ≥ 1280.
• PLPMTUD failure. tcp_mtu_probing is 0 and PMTUD is black-holed, so the router’s own TCP sessions stall. Fix: set system sysctl parameter net.ipv4.tcp_mtu_probing value 1. This helps the router, not the hosts behind it.
• DF=0 in application. An application sets DF=0, disabling PMTUD. Fix: ensure the application does not disable DF; use MSS clamping at the network layer.

Rollback

PMTUD-related changes are simple to roll back:

• delete interfaces ethernet eth0 ip adjust-mss (and the ipv6 sibling) — clamping is rewritten in the SYN, so removing it affects only new connections. Established sessions keep the clamped MSS until they are torn down, which means a rollback that “did not work” is usually a rollback you have not tested with a fresh connection.
• delete system sysctl parameter net.ipv4.tcp_mtu_probing — restores the kernel default (0) at commit; nothing needs to restart. Deleting is better than setting it back to 0, because it leaves nothing in the configuration for the next operator to wonder about.
• commit-confirm 5 for any of these, and for the firewall change especially — a rule that widens an inbound chain is exactly the kind of edit worth being able to lose contact through.

Production discipline

Cross-course references

• Part LI-01 (LI-VyOS-MTU / MTU basics) covers the canonical MTU values.
• Part LI-02 (LI-VyOS-MTU / tunnel overhead) covers the overhead calculations for tunnels, which PMTUD must negotiate around.
• Part LI-04 (LI-VyOS-MTU / MSS clamping) covers the canonical fix when PMTUD fails.
• Part LI-05 (LI-VyOS-MTU / MTU and fragmentation troubleshoot) covers the operational diagnostic for MTU mismatches.
• Part XXXVII (XXXVII-VyOS-Firewall) covers firewall configuration, including the ICMP rules that affect PMTUD.

Quiz