Break/Fix: small requests succeed, large ones hang, and ping says the network is fine · Break/Fix · Linux

Reported symptoms

The build agents cannot pull artefacts from the artefact server. curl hangs after a few hundred kilobytes and eventually times out.
The artefact server’s /healthz endpoint, which returns 200 bytes, responds instantly from the same agents.
SSH from an agent to the artefact server connects, prints the banner, gives a prompt — and then freezes the first time a command produces a page or more of output. ls on a small directory is fine. ls -lR is not.
ping between them is clean: 0% loss, sub-millisecond latency.
Monitoring has been green throughout, on both hosts.
Agents on the same VLAN as the artefact server work perfectly. Agents in the other rack do not.
Someone noticed the transfer always stalls around the same offset.

Evidence provided

$ ping -c4 198.51.100.20
PING 198.51.100.20 (198.51.100.20) 56(84) bytes of data.
64 bytes from 198.51.100.20: icmp_seq=1 ttl=63 time=0.412 ms
64 bytes from 198.51.100.20: icmp_seq=4 ttl=63 time=0.398 ms
--- 198.51.100.20 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3050ms

$ ping -M do -c2 -s 1472 198.51.100.20
1480 bytes from 198.51.100.20: icmp_seq=1 ttl=63 time=0.451 ms
--- 198.51.100.20 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss

$ ping -M do -c2 -s 8972 198.51.100.20
--- 198.51.100.20 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1013ms

$ ping -M do -c2 -s 8972 192.0.2.30
8980 bytes from 192.0.2.30: icmp_seq=1 ttl=64 time=0.702 ms
--- 192.0.2.30 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss

$ ip link show eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT
    link/ether 52:54:00:9a:0b:1c brd ff:ff:ff:ff:ff:ff

$ tracepath -n 198.51.100.20
 1?: [LOCALHOST]                      pmtu 9000
 1:  192.0.2.1                         0.380ms
 1:  192.0.2.1                         0.291ms
 2:  192.0.2.1                         0.302ms pmtu 1500
 2:  no reply
 3:  no reply

$ ss -ti dst 198.51.100.20 | tail -2
	 cubic wscale:7,7 rto:1920 rtt:0.44/0.2 mss:8948 pmtu:9000
	 bytes_sent:1449216 bytes_acked:262144 retrans:0/47 lastsnd:8112

$ sudo tcpdump -ni eth0 host 198.51.100.20 and not icmp | tail -4
09:41:02.118 IP 192.0.2.44.51882 > 198.51.100.20.443: Flags [.], seq 262145:271093, length 8948
09:41:03.142 IP 192.0.2.44.51882 > 198.51.100.20.443: Flags [.], seq 262145:271093, length 8948
09:41:05.190 IP 192.0.2.44.51882 > 198.51.100.20.443: Flags [.], seq 262145:271093, length 8948
09:41:09.286 IP 192.0.2.44.51882 > 198.51.100.20.443: Flags [.], seq 262145:271093, length 8948

$ sudo tcpdump -ni eth0 icmp -c 5 -w /dev/null
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C0 packets captured

Work the evidence before reading on

The transcript contains the answer twice over, and the second occurrence is the more important one:

ping succeeds and ping -M do -s 8972 to the same address fails completely. Work out what those two commands differ in — there are two differences, and both matter.
The 8972-byte probe to the same-VLAN host succeeds. So the local interface can send a 9000-byte frame.
tracepath reports pmtu 1500 at hop 2.
tcpdump shows the same segment, same sequence numbers, sent four times with doubling gaps. bytes_acked has stopped at 262144 while bytes_sent keeps rising.
The ICMP capture recorded nothing at all.

Before continuing: what is TCP supposed to receive that would tell it to send a smaller segment, and what does an empty ICMP capture imply about it?

Root cause

1. `ping` and `ping -M do -s` ask different questions

A default ping sends 56 bytes of payload — an 84-byte packet — with fragmentation permitted. It will traverse almost any path. A clean ping result means the addresses are reachable and the routing is sane. It says nothing whatsoever about how large a packet the path can carry, which is why “ping is fine” is one of the least useful facts in a troubleshooting session.

ping -M do -s 8972 sets the Don’t Fragment bit and sends a 9000-byte packet (8972 payload + 8 ICMP header + 20 IP header). That is the actual question.

The two failure modes look different and the difference is diagnostic:

Local error (ping: local error: message too long, mtu=1500) — your own interface refused it. The problem is on this host.
100% loss with no error — the packet left, and something on the path silently discarded it. The problem is out there.

Here it is the second, which points at the path.

2. Path MTU Discovery is supposed to fix this automatically

The mechanism exists precisely for a path with a smaller link in the middle:

TCP sends a segment with the Don’t Fragment bit set (Linux sets it by default).
The router whose next hop cannot carry it drops the packet and sends back ICMP type 3, code 4 — destination unreachable, fragmentation needed — including the MTU it can carry.
The sender caches a lower MTU for that destination and retransmits in smaller segments.

Step 3 never happens if step 2 never arrives.

3. The ICMP is being filtered

The capture on icmp recorded nothing during an active stall. A site firewall rule blocks ICMP as a class — almost always added years ago as “hardening”, and almost always without anyone considering which ICMP types the network depends on to function.

With that message filtered, the sender’s model of the path is simply wrong and nothing can correct it. ss -ti shows the consequence plainly: mss:8948 and pmtu:9000, both of which are false, and a retrans counter climbing while bytes_acked stands still. TCP is behaving exactly as designed — retransmit, back off, retransmit — against a segment that will never be delivered no matter how many times it is sent.

This is a PMTUD black hole, and the name is apt: packets go in, no information comes out.

4. Why size is the axis and everything else is noise

Anything that fits inside 1500 bytes works perfectly:

The TCP handshake, and usually the TLS handshake.
The SSH banner and key exchange, and small commands.
DNS.
A 200-byte health check — hence three weeks of green monitoring.

Anything that fills the window stalls at the first full-size segment. The transfer always dies at roughly the same offset because that offset is where the sender’s congestion window first grows past a single under-1500-byte segment.

Same-VLAN traffic is fine because it never reaches the router. So the fault appears to be about which rack, when it is really about which size, and the rack correlation sends the investigation straight to the network team’s routing configuration.

Blocking all ICMP is not hardening

ICMP is not an optional diagnostic protocol that can be switched off for safety. Parts of it are load-bearing:

Type 3 code 4 (fragmentation needed) — without it, Path MTU Discovery cannot work and every MTU mismatch on the path becomes a silent hang instead of an automatic adjustment.
ICMPv6 type 2 (packet too big) — for IPv6 this is not merely helpful, it is mandatory. IPv6 routers do not fragment at all, so PMTUD is the only mechanism. Filtering it breaks large IPv6 transfers completely.
ICMPv6 types 133-136 (neighbour and router discovery) — filtering these breaks IPv6 address resolution outright, the equivalent of blocking ARP.

If echo request is genuinely unwelcome on a boundary, block echo request. Blocking the whole protocol trades a small reduction in reconnaissance for a class of failure that is very hard to diagnose and that will eventually cost more hours than it ever saved.

Resolution

Measure the path MTU rather than assuming it. Binary-search with the do-not-fragment bit; the largest size that succeeds plus 28 is the path MTU:
``# Substitute your own values before running: PEER=198.51.100.20 for s in 1472 2972 4972 6972 8972; do printf '%s: ' "$s" ping -M do -c1 -W1 -s "$s" "$PEER" >/dev/null 2>&1 && echo ok || echo fail done``
Confirm where it drops. tracepath -n "$PEER" names the hop that reduces the pmtu, which is the hop to take to the network team
Restore service now by matching the interface to the path. This is a workaround, and it works immediately:
``sudo ip link set eth0 mtu 1500``
Re-test the large transfer before doing anything else, to confirm the hypothesis was right
Persist the MTU. A runtime ip link set is gone at the next reboot or netplan apply. On netplan:
``network: ethernets: eth0: mtu: 1500``
Fix the ICMP filtering, which is the defect that turned a configuration mismatch into a silent hang. At minimum permit destination-unreachable inbound:
``sudo nft add rule inet filter input icmp type destination-unreachable accept sudo nft add rule inet filter input icmpv6 type packet-too-big accept``
Take the same finding to whoever owns the boundary firewall. A host-level rule does not help if the message is dropped two hops away, and it usually is
Decide what the MTU policy actually is. Either make the path jumbo end to end — every switch, inter-switch link, router interface and hypervisor bridge — or standardise on 1500. A jumbo configuration that is right on the hosts and wrong on one hop is worse than not having it
Clamp MSS at the L3 boundary as a belt-and-braces measure, so TCP never offers a segment the path cannot carry even if PMTUD fails again:
``tcp flags syn tcp option maxseg size set rt mtu``

Verification

A full-size do-not-fragment probe succeeds to a far-side host. ping -M do -c2 -s 1472 198.51.100.20 at MTU 1500, or the jumbo equivalent if you fixed the path. Testing only a same-VLAN peer reproduces the original mistake
tracepath is consistent end to end. tracepath -n 198.51.100.20 reports one pmtu with no reduction at an intermediate hop
A large transfer completes under a time bound. This is the check that can fail and the one the users care about:
``curl -sS --max-time 30 -o /dev/null -w '%{size_download} bytes in %{time_total}s\n' https://svc.example.com/artifact.tar``
No retransmissions during that transfer. Run ss -ti dst 198.51.100.20 while it is in flight and confirm retrans stays at zero and bytes_acked tracks bytes_sent
PMTUD is working again, not merely unneeded. After a large transfer, ip route get 198.51.100.20 should show a cached MTU learned from the path. A route with no cached MTU on a path that needs one means the ICMP is still not arriving
ICMP actually arrives. Capture during a deliberately oversized transmission: sudo tcpdump -ni eth0 icmp should now show the fragmentation-needed message rather than nothing
SSH behaves. Run ls -lR /usr over the SSH session that used to freeze. It is a crude test and it is the one the reporter will repeat
It survives a reboot. Reboot or re-apply the network configuration and re-run the do-not-fragment probe. An MTU set only at runtime regresses silently
Monitoring can now see this class of fault. Add a large-packet probe and confirm it goes red when you temporarily lower an MTU somewhere on the path

Where MTU mismatches actually come from

Almost nobody sets a wrong MTU on purpose. They appear at boundaries:

Tunnels. WireGuard, IPsec, GRE, VXLAN and OpenVPN each add encapsulation overhead, so the usable MTU inside the tunnel is smaller than the MTU outside it. A tunnel interface left at 1500 on a 1500-byte underlay is guaranteed to produce this failure.
Overlay networks. Container and cloud overlays subtract their own header. This is why so many Kubernetes clusters have a 1450 or 1400 MTU on the pod network, and why “DNS works but pulling an image hangs” is such a common first-day symptom.
Partial jumbo rollouts. Jumbo frames get enabled on the hosts and the top-of-rack switches, and the inter-switch link or the router interface is missed. Everything inside the rack works, so it ships.
A new hop. Adding a firewall, an inspection appliance or a provider circuit into an existing path can quietly reduce the MTU.
PPPoE and some provider links, which have carried a 1492-byte MTU for decades and will continue to surprise people.

The common thread is that the host configuration is usually right and the path is not, which is why per-host audits find nothing.

Prevention

Test MTU end to end, with ping -M do -s, as part of network acceptance. A jumbo rollout that has not been probed across every path it claims to cover has not been verified.
Never filter ICMP as a class. Permit destination-unreachable, and permit ICMPv6 packet-too-big and neighbour discovery unconditionally. Write the reason in a comment next to the rule so the next hardening pass does not remove it.
Clamp MSS to the path MTU at L3 boundaries and on every tunnel endpoint. It is a one-line rule that makes TCP correct even when PMTUD is broken by someone else’s firewall.
Include a large-payload check in monitoring. A 200-byte health probe is structurally incapable of detecting this, and the three weeks of green dashboards in this incident were not a monitoring gap so much as a monitoring design that only ever asked one question.
Document the MTU for every VLAN and every tunnel, and treat a change to it as a change to every host on it.
When a report contains the words “small requests work, large ones hang”, check MTU before anything else. That sentence has essentially one cause.

What if lowering the MTU does not fix it?

Then the size dependence is real but the mechanism is different, and there are three other candidates worth knowing.

TCP segmentation offload interacting with a broken NIC or driver. The kernel hands the NIC a large buffer and the NIC splits it into frames. A firmware bug can corrupt the result only for large payloads. Test by disabling offloads temporarily:

sudo ethtool -K eth0 tso off gso off gro off

If the transfer completes with offloads off, it is not an MTU problem. Re-enable them after testing — they are worth real CPU — and take the finding to the driver or firmware.

An application-layer buffer or proxy limit. A reverse proxy with a small buffer, an ingress body-size limit, or a middlebox doing deep inspection will also fail on size. The distinguishing evidence is that these usually produce a response — a 413, a RST, a truncated body — rather than a hang with retransmissions and silence.

Asymmetric routing through a stateful firewall. Large flows take a different path than small ones through ECMP hashing, and one path traverses a firewall that never saw the handshake. The tell is packets arriving at the far end and replies returning by a different route.

The way to tell these apart quickly is the same evidence used above: tcpdump for what leaves and what comes back, and ss -ti for whether the sender is retransmitting into silence or receiving an actual answer. A hang with retransmissions and no ICMP is an MTU black hole. A hang with a RST, or a completed transfer that is simply truncated, is not.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. ping and ping -M do -s ask different questions

2. Path MTU Discovery is supposed to fix this automatically

3. The ICMP is being filtered

4. Why size is the axis and everything else is noise

Resolution

Verification

Prevention

1. `ping` and `ping -M do -s` ask different questions