Jumbo frames: benefit, risk, and verification

Why this matters in production

Jumbo frames are a modest, real improvement with a severe failure mode. The benefit is a few percent of CPU and some throughput on large transfers; the cost of getting it wrong is intermittent hangs that take days to diagnose. Whether that trade is worth it depends on your discipline more than on your workload.

The benefit

A 9000-byte MTU carries six times the payload of 1500 bytes per packet, so a large transfer needs roughly one-sixth the packets:

The gain is largest where per-packet cost dominates: recovery, backfill, large sequential RGW transfers, and any host where softirq processing is near saturation. It is negligible for small random RBD I/O, where the packets are small anyway.

Everything that must agree

• Host NIC on every Ceph node, and every bond member
• The bond interface itself, and any VLAN sub-interface
• Every switch port in the path
• Switch fabric and inter-switch links
• Any router in a routed design
• Client hosts, for public-network traffic

One device left at 1500 produces a path that carries small frames and drops large ones. Ceph then works for heartbeats and small operations and hangs on large ones.

Verifying properly

ip link show bond0 | grep mtu
ip link show bond0.30 | grep mtu

# the only test that proves anything
ping -M do -s 8972 -c 3 10.30.0.47

8972 + 28 = 9000. -M do prevents fragmentation so the test fails loudly on a path that cannot carry the frame. Run it between every pair of racks, on both networks, and after every network change.

# quick sweep across all OSD hosts
for h in $(ceph osd metadata | jq -r '.[].hostname' | sort -u); do
  ping -M do -s 8972 -c 1 -W 1 "$h" >/dev/null 2>&1 \
    && echo "OK   $h" || echo "FAIL $h"
done

Deciding

Use jumbo frames when: the cluster is throughput-oriented, the network is under your control end to end, and MTU verification is part of your change process.

Stay at 1500 when: the network is shared or managed by another team, the workload is small-random, or you cannot guarantee verification after every change. A correctly-working 1500 MTU cluster outperforms a mis-configured 9000 one by a wide margin.

Quiz

Production discipline

If you deploy jumbo frames, make ping -M do verification a mandatory post-change step for both storage and network teams, and script the full-mesh sweep so it can be run in seconds. If you cannot guarantee that discipline, staying at 1500 is the better engineering decision and worth stating explicitly so it is not revisited annually.

Cross-course references

• Kubernetes: overlay MTU must account for encapsulation overhead, with the same silent-failure risk
• Linux: path MTU discovery behaviour and ICMP filtering are general networking concerns