Changing MTU across a live cluster

Why this matters in production

An MTU change is the network maintenance most likely to produce a partial failure, because a mismatched pair works for small packets and fails for large ones.

The problem with the obvious sequence

Raising MTU host by host:
  host A at 9000, host B at 1500
  A sends a 9000-byte frame to B
  B's interface drops it
  small packets work; large ones do not
  recovery stalls; heartbeats continue
  the cluster appears healthy and moves no data

This intermediate state exists for however long the rollout takes, and it is the worst possible failure mode.

Sequencing to avoid it

The switch fabric must support the larger MTU first.
A switch configured for 9216 carries both 1500 and 9000 frames.
Hosts at 1500 send 1500-byte frames, which the fabric carries.
Hosts at 9000 send larger frames, which the fabric also carries.
The mismatch is between hosts, not with the fabric.

1. raise the MTU on every switch port and the fabric
2. verify the fabric carries 9000-byte frames between two test hosts
3. raise the MTU on hosts one at a time
4. between each, verify connectivity to every other host at both MTUs

# after each host, verify it can still reach every other
for h in $(ceph orch host ls --format json | python3 -c \
  'import sys,json;[print(x["addr"]) for x in json.load(sys.stdin)]'); do
  printf '%-16s 1500:' "$h"
  ping -M do -s 1472 -c 2 -W 2 "$h" >/dev/null 2>&1 && printf ' OK' || printf ' FAIL'
  printf '  9000:'
  ping -M do -s 8972 -c 2 -W 2 "$h" >/dev/null 2>&1 && echo ' OK' || echo ' FAIL'
done

Why the intermediate state still needs care

Host A at 9000 sending to host B at 1500:
  A's stack knows its own MTU, not B's
  it sends a 9000-byte frame
  B drops it

Path MTU discovery would resolve this, and it depends on ICMP that is frequently filtered.

# so verify both directions after each host
OTHER_HOST=stor-04
ping -M do -s 8972 -c 2 ${OTHER_HOST}
# and from the other host back

The safest approach on a cluster that can tolerate it is a maintenance window where every host is changed together and the cluster is briefly stopped.

Verifying at each stage

ip link show | grep -oE 'mtu [0-9]+'
ceph -s
ceph -s | grep -E 'slow|degraded'

Rolling back

ip link set dev bond0 mtu 1500

Rolling back a host is immediate and restores compatibility with the
1500-byte hosts, which is why an incremental rollout with per-step
verification is recoverable at any point.

Quiz

Production discipline

Raise the fabric MTU before any host — a fabric carrying the larger size accommodates both, so the only possible mismatch is host-to-host and the per-step verification catches it. Verify each host against every other at both sizes after each change.

Cross-course references

• Kubernetes: overlay MTU changes need the same underlay-first sequencing
• Linux: MTU mismatches are silent because the failure mode produces no error