MTU Causes Large Packets to Fail

Reported symptoms

edge2 is the single edge router for a mid-sized office. One WAN handoff on eth0 at 203.0.113.2/30, a LAN at 10.20.0.0/22, a stateful WAN chain that has not been touched in over two years, and no tunnels. Every interface is at the default MTU of 1500.

Over ten days the service desk collected four tickets that nobody connected:

• Uploads to a partner API stall part-way through and time out. Small requests to the same endpoint answer instantly. The partner has been asked twice to check their side and has twice reported their service healthy.
• A developer’s git clone over SSH hangs at “Receiving objects: 12%” and stays there. The SSH connection itself is fine; an interactive shell to the same host works normally. Filed as a Git server problem.
• The nightly database dump to an off-site target has failed three nights running with a transfer timeout, after a year of clean runs. Filed against the backup job.
• One team reports that a particular SaaS console “sometimes doesn’t load”, which was closed as a browser issue.

Everything anyone reached for came back clean. ping to every affected destination succeeds. traceroute completes. The monitoring dashboard is entirely green — the ICMP checks answer and the synthetic HTTP check is a HEAD request. DNS, VoIP and interactive SSH have no complaints at all.

An MTU audit was run early, because “large transfers fail” is a familiar phrase. It found every interface on edge2 at 1500 and closed. No configuration has been committed on the router in eleven weeks. The ISP was asked and confirmed no customer-facing change to the circuit, which is true.

Evidence provided

user@lan-host:~$ ping -M do -s 1472 198.51.100.25

PING 198.51.100.25 (198.51.100.25) 1472(1500) bytes of data.
^C
--- 198.51.100.25 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4085ms

user@lan-host:~$ ping -M do -s 1422 -c 2 198.51.100.25

PING 198.51.100.25 (198.51.100.25) 1422(1450) bytes of data.
1430 bytes from 198.51.100.25: icmp_seq=1 ttl=57 time=8.41 ms
1430 bytes from 198.51.100.25: icmp_seq=2 ttl=57 time=8.33 ms

vyos@edge2:~$ sudo tcpdump -i eth0 -nn 'icmp and icmp[icmptype] == 3'

10:14:02.117 IP 203.0.113.9 > 10.20.0.31: ICMP 198.51.100.25 unreachable - need to frag (mtu 1450), length 556
10:14:04.229 IP 203.0.113.9 > 10.20.0.31: ICMP 198.51.100.25 unreachable - need to frag (mtu 1450), length 556
10:14:08.441 IP 203.0.113.9 > 10.20.0.31: ICMP 198.51.100.25 unreachable - need to frag (mtu 1450), length 556

vyos@edge2:~$ show log firewall | match 'icmp'

10:14:02 edge2 kernel: [WAN-IN-default-D] IN=eth0 OUT=eth1 SRC=203.0.113.9 DST=10.20.0.31 PROTO=ICMP TYPE=3 CODE=4
10:14:04 edge2 kernel: [WAN-IN-default-D] IN=eth0 OUT=eth1 SRC=203.0.113.9 DST=10.20.0.31 PROTO=ICMP TYPE=3 CODE=4
10:14:08 edge2 kernel: [WAN-IN-default-D] IN=eth0 OUT=eth1 SRC=203.0.113.9 DST=10.20.0.31 PROTO=ICMP TYPE=3 CODE=4

vyos@edge2:~$ show configuration commands | match 'firewall ipv4 name WAN-IN'

set firewall ipv4 name WAN-IN default-action 'drop'
set firewall ipv4 name WAN-IN enable-default-log
set firewall ipv4 name WAN-IN rule 10 action 'accept'
set firewall ipv4 name WAN-IN rule 10 state 'established'
set firewall ipv4 name WAN-IN rule 20 action 'accept'
set firewall ipv4 name WAN-IN rule 20 protocol 'icmp'
set firewall ipv4 name WAN-IN rule 20 icmp type-name 'echo-reply'
set firewall ipv4 name WAN-IN rule 30 action 'accept'
set firewall ipv4 name WAN-IN rule 30 protocol 'icmp'
set firewall ipv4 name WAN-IN rule 30 icmp type-name 'time-exceeded'

vyos@edge2:~$ sudo tcpdump -i eth0 -nn host 198.51.100.25 and tcp

10:14:01.982 IP 10.20.0.31.51402 > 198.51.100.25.443: Flags [.], seq 1:1449, ack 1, length 1448
10:14:02.196 IP 10.20.0.31.51402 > 198.51.100.25.443: Flags [.], seq 1:1449, ack 1, length 1448
10:14:04.310 IP 10.20.0.31.51402 > 198.51.100.25.443: Flags [.], seq 1:1449, ack 1, length 1448
10:14:08.520 IP 10.20.0.31.51402 > 198.51.100.25.443: Flags [.], seq 1:1449, ack 1, length 1448

Work the evidence before reading on

Before reading on, account for every green result as well as every red one. The green ones are doing more work in this incident than the red ones.

1. ping -M do -s 1472 hangs. ping -M do -s 1422 succeeds. There are two very different situations that produce a size boundary, and only one of them produces silence at the top. Which is this, and what is the other one’s signature?
2. ping works and traceroute works. Name the ICMP types each of those depends on, then read rule 20 and rule 30 again. Is that a coincidence?
3. The router’s every interface is at 1500 and the boundary is at 1450. Where can the constriction be, given that constraint, and does that location have any bearing on whether this router is involved in the fault?
4. Rule 10 accepts state established. An ICMP error generated by a router in the middle of a path, referring to one of this site’s flows — what state does conntrack put it in, and does rule 10 admit it?
5. The firewall’s default-deny log has been recording these drops at a steady low rate for as long as the retention goes back, and the tickets started ten days ago. Reconcile those two facts.

Then the question that decides which fix you choose: if you make every stalled transfer succeed, have you necessarily made discovery work?

Root cause

1. The path is small and, worse, silent about it

Two distinct things can go wrong with a large packet. A path with a smaller bottleneck than the sender assumes is ordinary and self-correcting: the router at the bottleneck drops the oversized packet and sends back an ICMP fragmentation-needed carrying the bottleneck’s MTU, the sender lowers its estimate for that destination, and the flow continues. That failure lasts one round trip and nobody files a ticket.

The failure that produces tickets is the second one: the report never gets back. The sender’s estimate is never corrected, so it retransmits the identical oversized segment, which is dropped identically, forever. That is the retransmission pattern in the capture — the same segment, the same length, at doubling intervals, with nothing coming back for it — and it is why the application-layer symptom is a stall rather than an error.

The bisect established the number: the path to this destination carries 1450 bytes. The capture established the more important fact: the report saying so is arriving at eth0 and is not reaching the host that needs it.

2. The allow-list permits exactly the two types that make diagnostics look fine

WAN-IN accepts inbound ICMP by type, and the list is echo-reply and time-exceeded. Those are not arbitrary choices — they are the two types someone added when ping and traceroute stopped working after the chain was tightened. Both are legitimately useful, and permitting them made the obvious tests pass.

Fragmentation-needed is ICMP type 3. It is not on the list. First match wins in a VyOS chain, so a type 3 packet falls past rule 20 and rule 30, reaches the default-deny, and is dropped and logged.

This is why every diagnostic anyone reached for came back green, and it is worth sitting with. The engineers testing this network were using the two ICMP types the firewall had been configured to permit, and concluding from their success that ICMP was fine. The test and the defect were tuned to each other by accident.

3. established is not related

There is a second door that should have admitted these packets and does not. Connection tracking classifies an ICMP error that references a tracked flow as related — it is not a new connection and it is not part of the established stream, it is a message about the stream. A rule matching state established does not match it. A rule matching state established, related does.

The omission is easy to make and completely invisible until an ICMP error matters. In two years, one never did.

4. Nothing changed at the site, and the path changed anyway

The constriction is at 1450 inside the provider’s network, on a path that used to carry 1500. Both statements the incident kept running into are true: the site committed no configuration in eleven weeks, and the ISP made no customer-facing change to the circuit. The handoff is still 1500. The end-to-end path is not, because a path is not a property of either end.

That also explains the scatter that made this look like an application problem. A destination is affected if and only if its path has a sub-1500 bottleneck. Destinations reached over provider backbones that encapsulate somewhere, or that sit behind a cloud overlay, are affected; direct-peered destinations are not. The pattern is real and it lives entirely outside anything visible from the site, which is why nobody at the site could find it.

Resolution

1. Decide which of the two fixes you are applying, and say what each does not cover. MSS clamping makes TCP fit; restoring ICMP makes discovery work. They are not the same repair and one of them leaves UDP-based traffic — QUIC and HTTP/3, UDP-encapsulated VPN, large DNS responses — exactly as broken and exactly as silent as it is now.
2. If you need the transfers working before the ICMP policy can be changed, clamp to the measured value: set interfaces ethernet eth0 ip adjust-mss 1410, which is the 1450-byte path minus 40 bytes of IPv4 and TCP header. Do not use clamp-mss-to-pmtu here; the local path MTU is 1500 and the clamp it derives would fix nothing.
3. Treat that clamp as a stopgap with an expiry rather than a resolution, and record it as one: the number is specific to a 1450-byte path, and a different destination with a smaller bottleneck will fail again without anything looking different.
4. Fix the chain properly by letting connection tracking decide: change rule 10 to match state established, related so that an ICMP error referring to a flow this router already knows about is admitted, and an unsolicited one still is not. This is narrower than permitting ICMP type 3 outright and it is the change to argue for.
5. If the security owner will not accept an ICMP change inside an incident, that is a legitimate hold rather than an obstruction — but book it, with a named owner and a date, and write down that UDP-based protocols remain exposed until then.
6. Do not raise an MTU anywhere on this router. Every interface is already at 1500 and the constriction is not here; a change that makes the symptom move without explaining it is the thing you will have to undo during the next incident.
7. Tell the partner and the backup owner what actually happened, since both were asked to investigate services that were working correctly, and one of them has been changing their configuration in response.

Verification

1. The mechanism is restored, not merely the transfer. From an affected host, ping -M do -s 1472 198.51.100.25 must now return Frag needed and DF set (mtu = 1450). Silence with a working transfer means you clamped and nothing else.
2. The ICMP is arriving and no longer being dropped: the fragmentation-needed packets are visible in a capture on eth0 and absent from show log firewall.
3. The firewall default-deny counter has stopped advancing at the rate it had been advancing at for years. If it has not, something above your new rule is still shadowing it, and first-match order is the place to look.
4. A second affected destination behaves the same way. Path MTU differs per path; a fix that works because you clamped to 1410 proves nothing about a 1400-byte path, whereas a fix that restores discovery needs no number at all.
5. Nothing was solved by changing an MTU. Confirm every interface is still at 1500 and that no MTU was lowered during the incident to make a test pass.
6. The synthetic monitoring now contains a probe that would have caught this: a DF-set check at full size, not only the small checks that stayed green throughout.

Prevention

• Express ICMP policy as a deny-list of what you object to, not an allow-list of what you use. An allow-list fails silently and late, and the two types most people remember to permit are exactly the two that make ping and traceroute succeed — which is why this defect survives every casual test of the network.
• Better still, let conntrack carry it: state established, related admits ICMP errors that reference a flow the router already knows about and admits nothing else. That is both narrower than a type allow-list and impossible to leave a gap in by forgetting a type.
• Alert on fragmentation-needed drops. This evidence was in the firewall log the whole time, at a rate low enough that nobody had a reason to read it. The normal value for that counter is zero, which makes it a very cheap alert with almost no false positives.
• Put a DF-set probe at full interface MTU into monitoring, next to the ordinary ping check. Every probe in this estate fit inside one segment, and every one of them stayed green for the entire incident, which is a fact about the probes rather than about the network.
• Baseline path MTU to the destinations that matter — the partner API, the backup target, the main SaaS endpoints — so that a change in one is an event with a timestamp rather than a mystery that starts as four unrelated tickets.
• Retire “nothing changed on our side” as an answer. Both parties were telling the truth about their configuration, and the path changed anyway, because a path is not owned by either end of it.