MTU mismatch: the most baffling slow-ops cause

Why this matters in production

MTU mismatch is the failure most likely to consume a full day of the wrong investigation. It presents as slow ops and hung operations, it survives every ordinary connectivity check, and it is confirmed or eliminated in about ten seconds by a test almost nobody runs first.

The symptom pattern

Learn this pattern — it is highly specific:

• Small operations succeed, large ones hang. Heartbeats fine, metadata fine, data transfers stall.
• Idle cluster looks healthy. The problem appears under load, when large transfers occur.
• Slow ops accumulate with no device or CPU saturation.
• Recovery stalls — backfill moves large blocks and is hit hardest.
• ping succeeds. Default 56-byte pings pass through a 1500-byte bottleneck without difficulty.
• OSDs flap under load and recover when load drops.

Any two of those together should prompt an MTU test before anything else.

The confirmation

# succeeds if the path carries 9000-byte frames
ping -M do -s 8972 -c 3 10.30.0.47

# for comparison, standard MTU
ping -M do -s 1472 -c 3 10.30.0.47

If the second succeeds and the first fails, you have your answer.

Then localise by bisecting: test to intermediate hops, and check interfaces along the path.

ip link show | grep -E 'mtu|state UP'

# every OSD host, both networks
for h in $(ceph osd metadata | jq -r '.[].hostname' | sort -u); do
  printf '%-24s ' "$h"
  ping -M do -s 8972 -c 1 -W 1 "$h" >/dev/null 2>&1 && echo OK || echo FAIL
done

Common causes

The VLAN case is a frequent one: setting the MTU on a bond does not always propagate to sub-interfaces created on it, so bond0 shows 9000 and bond0.30 shows 1500.

Immediate mitigation

If you cannot fix the mismatched device immediately, lowering MTU to 1500 everywhere restores correct behaviour at a small performance cost. That is usually preferable to leaving a cluster in a state where large transfers hang.

Quiz

Production discipline

Put ping -M do at the top of the slow-ops runbook for any cluster using jumbo frames, and script the full-mesh sweep so it can be run in one command. Add MTU verification to the network change checklist on both the storage and network sides — nearly every occurrence follows a network change, and the check is faster than the incident.

Cross-course references

• Kubernetes: overlay MTU mismatches produce identical hang-on-large-payload symptoms
• Linux: path MTU discovery failure with filtered ICMP is a general and well-documented hazard