MTU end to end

Why this matters in production

An MTU mismatch produces intermittent failures on large transfers while small packets work perfectly — which is one of the most confusing failure modes in networking, and Ceph exercises it constantly.

Verifying end to end

# with a 9000 MTU: 9000 - 20 (IP) - 8 (ICMP) = 8972
PEER=peer
ping -M do -s 8972 -c 5 ${PEER}

PING peer (10.0.1.12) 8972(9000) bytes of data.
8980 bytes from 10.0.1.12: icmp_seq=1 ttl=64 time=0.187 ms

Success means every hop carries a 9000-byte frame. Failure:

ping: local error: message too long, mtu=1500
# or
--- peer ping statistics ---
5 packets transmitted, 0 received, 100% packet loss

The second form — silent loss rather than an error — is the dangerous one, because it means a switch in the path is dropping oversized frames without notifying anyone.

# check every path, not just one
for h in $(ceph orch host ls --format json | python3 -c 'import sys,json;[print(x["addr"]) for x in json.load(sys.stdin)]'); do
  printf '%-16s ' "$h"
  ping -M do -s 8972 -c 2 -W 2 "$h" >/dev/null 2>&1 && echo OK || echo FAIL
done

The symptoms of a mismatch

IFACE=bond0
ip link show ${IFACE} | grep -o 'mtu [0-9]*'
ethtool -S ${IFACE} | grep -iE 'oversize|too_long|frag'

Configuring correctly

Every element in the path must agree:

host NIC          → 9000
host bond         → 9000
host bridge/VLAN  → 9000
switch port       → 9216 (typically, to allow for headers)
switch fabric     → jumbo enabled
router, if any    → 9000

# persistent, on the host
ip link set dev eth0 mtu 9000
# and in the network configuration so it survives a reboot

Switches often need a larger value than the hosts because their figure includes framing overhead, which is a common source of an off-by-a-few mismatch that fails only for maximum-size frames.

Is it worth it?

On 25 GbE and above with modern offload, the benefit is smaller than it was on 1 GbE. A cluster that is not throughput-bound gains little, and the failure mode is unpleasant — so the honest answer is that jumbo frames are worth configuring when the fabric is fully under your control and not worth the risk when it is not.

Quiz

Production discipline

Test MTU across every host pair with ping -M do, not one representative path; a single misconfigured port produces intermittent recovery stalls that persist for months. Configure switches with framing headroom above the host MTU, and treat jumbo frames as optional unless the fabric is fully under your control.

Cross-course references

• Kubernetes: overlay network MTU mismatches produce the identical silent large-packet failure
• Linux: path MTU discovery failures behind ICMP filtering are a classic hard-to-find fault