Runbook: Investigate an MTU Issue

An MTU incident almost never arrives described as one. It arrives as four reports that read like four different problems:

• The VPN is up, but the file share on the other side is unusable.
• The site loads, and the login page hangs forever.
• SSH is fine; copying anything large stalls at the same point every time.
• The nightly replication to the DR site started failing on Tuesday, and nobody changed anything.

They are one fault. Something on the path cannot carry a packet above a certain size, and everything below that size works perfectly — which is precisely why none of the four reports mentions size, and why the first hour of these incidents is usually spent on DNS, on the application, and on the firewall.

The whole job of this runbook is to convert an adjective into a number in the first ten minutes. “Large packets fail” is not evidence. “1372 bytes of payload succeeds and 1373 does not” is evidence, and it is most of the diagnosis.

When to use this runbook

Small packets reach the destination and large ones do not, on a path that this estate has a router on. The classic tells are a protocol that completes its handshake and then stalls on the first substantial payload, and a failure that reproduces at exactly the same size every time.

When not to use it

• A default-size ping already fails. Nothing here applies; the path is broken and the entry point is vyos-rb-troubleshoot-missing-route.
• Loss is proportional at every size, or comes and goes. That is vyos-rb-investigate-packet-loss — MTU faults are a cliff, not a slope.
• The tunnel is new and has never carried a large packet. Sizing the tunnel is part of turn-up: vyos-rb-configure-wireguard and vyos-rb-configure-ipsec both end with a sized ping for this reason.

Blast radius

Every step up to the fix is read-only and can be run during business hours from any session. The fix is not. An interface MTU change bounces the link, an MSS clamp changes every TCP connection that later crosses the interface, and permitting ICMP widens a filter. Treat the remediation as a change with its own review, on a path that has just proved it can surprise you.

Step 1: turn the adjective into a number

ping 192.0.2.50 count 3
ping 192.0.2.50 size 1472 do-not-fragment count 3
ping 192.0.2.50 size 1372 do-not-fragment count 3
ping 192.0.2.50 size 1422 do-not-fragment count 3

size is the ICMP payload, not the packet. The IP packet is the payload plus 28 bytes — 20 of IP header and 8 of ICMP echo header — so payload 1472 is a 1500-byte packet and payload 1372 is a 1400-byte packet. Subtract 28 from the MTU you are testing for; this is the arithmetic operators most often get wrong, and getting it wrong shifts every conclusion by 28 bytes.

Four or five probes bisect the whole range. Start at 1472, and when it fails, halve the gap rather than stepping down by ten — a linear walk from 1472 is forty probes and an hour you do not have.

If you prefer the underlying tool, ping -M do -s 1472 192.0.2.50 is the same test; the course’s Part LI lessons use that form throughout.

Step 2: run the same size again without the DF bit

This is the step that gets skipped, and it is the step that tells you what kind of MTU fault you have.

ping 192.0.2.50 size 1472 count 3

Two probes at one size produce four outcomes, and each one points at a different fix:

The second row is the benign case and it is worth pausing on. If the path returns an ICMP fragmentation-needed carrying a usable MTU, the mechanism is working: senders that respect Path MTU Discovery will size themselves correctly on their own. If an application is still failing, the application is not honouring PMTUD, and clamping MSS is a workaround for that application rather than a repair of the network.

The third row is the one that generates incidents. Nothing is broken visibly; a router or firewall somewhere silently discards the oversized packet without generating the ICMP that would have told the sender why.

Step 3: find the hop, or accept that you cannot

Where PMTUD is working — the second row of the Step 2 table — the path has already told you the number and the hop that owns it. The ICMP fragmentation-needed carries the next-hop MTU, and a capture is what reads it:

monitor traffic interface eth0 filter 'icmp or icmp6'

IP 198.51.100.9 > 198.51.100.1: ICMP 192.0.2.50 unreachable -
     need to frag (mtu 1450), length 556

Two things come out of one line: the constraining MTU (1450) and the address of the hop that announced it (198.51.100.9). That is the part of a tracepath report you were actually going to act on — the binding constraint and its owner — read from the router you are already logged into. What it will not give you is the MTU of hops that are not currently the constraint, which is a real limitation only if you intend to raise the ceiling twice.

If nothing arrives, that is the black-hole case and it is a finding, not a failed test. Run traceroute to establish the topology — the hops, and where the path leaves your estate — and pair it with the measured number from Step

1. Knowing “1372 fits, 1373 does not, and hop 4 is the provider’s” is enough to choose a fix and enough to open a ticket with.

Do IPv6 separately and expect a different answer. The two families do not share a path MTU, they do not share an MSS, and they do not share a firewall rule set. A fix applied to one leaves the other exactly as it was.

Step 4: is the constraining hop yours?

Everything after this depends on the answer, so answer it explicitly and write it into the incident record.

The hop is an interface this estate configures — a tunnel, a VLAN sub-interface, a link you own both ends of. The MTU is probably simply wrong, and correcting it is the honest fix. Check the arithmetic below before you change it.

The hop is inside a provider or partner network. You are not going to change it. Your options are to size your own traffic to fit, or to get the provider to stop filtering the ICMP that would let PMTUD do it for you. Both are legitimate; only one is under your control today.

The hop cannot be identified at all because the path goes dark partway. Then the measured number from Step 1 is still valid and still actionable — you know what fits, even if you do not know why. Clamp to what fits, record that the constraining hop is unknown, and open the provider ticket anyway.

If PMTUD is black-holed, check that this router is not the one doing the filtering before blaming anyone else:

show firewall ipv4 input filter
show firewall ipv4 forward filter
show firewall ipv6 input filter
show firewall ipv6 forward filter
show firewall statistics

A default-deny chain with no rule permitting ICMP fragmentation-needed (IPv4) or ICMPv6 packet-too-big is a PMTUD black hole that this team owns. Read the counters: an accept rule sitting at zero packets while the default-action counter climbs means your traffic is being matched by something other than the rule you believe is in effect.

The arithmetic, before you change anything

MSS is the TCP payload only: it excludes both the TCP header and the IP header. So IPv4 MSS is MTU minus 40, and IPv6 MSS is MTU minus 60. The twenty-byte difference between the families is the single most common arithmetic error in this whole subject, and its symptom is an IPv6 path that still stalls after “the MTU was fixed”.

IPsec is deliberately left as “measure it”. The overhead depends on the cipher, on whether NAT traversal has added its eight bytes of UDP encapsulation, and on the length of the integrity check value, which is why Part LI-02 lands on a conservative VTI MTU of 1438 rather than a computed one. Read the negotiated proposal out of the Proposal column of show vpn ipsec sa — that is the cipher actually in use, which is not necessarily the one at the top of your proposal list — and then measure.

The second half of that row matters as much as the arithmetic. A VTI comes up at MTU 1500. Nothing about set interfaces vti vti0 sizes it for the overhead the tunnel is about to add, so an IPsec tunnel that was built without an explicit mtu is not “probably fine” — it is the exact configuration this runbook exists to diagnose, and it will present as an application fault. Confirm it with show interfaces vti vti0 before theorising about the path beyond it.

Step 5: choose the fix, and say what it costs

Permit the ICMP that PMTUD needs. This repairs the mechanism rather than working around it, and it fixes every protocol at once rather than only TCP. It only works where you own the filter, and it widens an inbound chain, so it is a security-reviewed change.

set firewall ipv4 name WAN-IN rule 5 action accept
set firewall ipv4 name WAN-IN rule 5 protocol icmp
set firewall ipv4 name WAN-IN rule 5 icmp type 3
set firewall ipv4 name WAN-IN rule 5 icmp code 4

set firewall ipv6 name WAN6-IN rule 5 action accept
set firewall ipv6 name WAN6-IN rule 5 protocol icmpv6
set firewall ipv6 name WAN6-IN rule 5 icmpv6 type-name packet-too-big

compare
commit-confirm 5

Correct the interface MTU. The right fix when a tunnel or a link is simply configured wrong. It bounces the link — briefly for an Ethernet interface, and long enough to notice — and it is only correct if the far end agrees.

Clamp MSS. The cheapest fix and the one that works through a provider you cannot change, because it needs nothing from anybody else. It is TCP-only.

set interfaces wireguard wg01 ip adjust-mss clamp-mss-to-pmtu
set interfaces wireguard wg01 ipv6 adjust-mss clamp-mss-to-pmtu

compare
commit-confirm 5

clamp-mss-to-pmtu derives the value from the interface MTU, which is why setting the interface MTU deliberately matters even when the clamp is doing the work. VyOS documents the derivation precisely: the MSS becomes the interface MTU minus 40 for IPv4 — 20 bytes of IP header and 20 of TCP — so an interface at 1468 advertises 1428. Where the underlay is itself constrained — PPPoE, a tunnel inside a tunnel — ip adjust-mss <value> takes an explicit number from your Step 1 measurement, which is safer than a derived one that is derived from the wrong MTU.

Note that ip adjust-mss and ipv6 adjust-mss are separate nodes on the same interface. Setting one does nothing for the other, which is the mechanical reason behind “fixed for IPv4, still broken for IPv6” appearing twice in this runbook.

Hold. A first-class option. If the path is degraded rather than down, and the change window is hours away, holding with a named owner and an end time is better than a rushed MTU change on a live provider link. Write the owner and the time into the incident record, not into somebody’s memory.

Step 6: apply it with a net, then verify on traffic

compare
commit-confirm 5
exit
ping 192.0.2.50 size 1372 do-not-fragment count 3

Leave configure mode and test from the operational shell while the timer runs. If the result is right, go back in and confirm, then save. If it is not, do nothing and let the timer restore the previous state — the operator who has just been wrong once is not the best judge of the second attempt.

monitor traffic interface eth0 filter 'tcp[tcpflags] & tcp-syn != 0'

monitor traffic is VyOS’s wrapper around tcpdump, so the filter syntax is tcpdump’s and the output is tcpdump’s; Ctrl-C stops it. The SYN carries the MSS option, and this capture is the only direct evidence that the clamp is on the interface the SYNs actually cross. A clamp bound to the wrong interface commits cleanly, validates cleanly, and does nothing.

The interface to watch is the one the SYN crosses, which is not always the one you clamped. On a tunnel the SYN enters on the LAN side and leaves on the tunnel; the clamp rewrites it on egress, so capture on the tunnel interface to see the clamped value and on the LAN interface to see what it was before.

Then verify the thing that was actually broken. Re-run the failed upload, the file copy, the backup job. A sized ping that now passes is necessary and not sufficient, and the gap between the two is where “we fixed it” turns into a second page an hour later.

Common patterns

Escalation

Escalate when:

• The constraining hop is outside the estate.
• The link is one half of a redundant pair.
• The fix requires widening a default-deny chain.
• The failing traffic is UDP and the clamp cannot reach it.
• The interface refuses the MTU, or does not come back after the change.
• The path crosses a tunnel whose far end this team cannot see.