Skip to main content
RunBook Academy

CephXXVIII · Ceph NetworkingCeph Networking

MTU, bonding, and VLANs: the link layer under Ceph

Advanced⏱ ~18 minippingethtool

What you'll learn

  • Configure and verify MTU consistently end to end
  • Choose a bonding mode appropriate to Ceph traffic
  • Apply VLAN separation without breaking daemon binding
  • Diagnose link-layer faults from Ceph symptoms

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

Link-layer misconfiguration produces Ceph symptoms that look like storage problems: hung large writes, flapping OSDs, unexplained latency. All three of the settings below are easy to get subtly wrong and easy to verify properly, and the verification takes minutes.

MTU

Jumbo frames (9000) reduce per-packet overhead and help throughput-heavy workloads. The requirement is absolute: every device in the path must agree. One switch port left at 1500 produces a path that carries small packets fine and silently drops large ones.

ip link show ens1f0 | grep mtu

# the definitive test — do not omit the -M do
ping -M do -s 8972 -c 3 10.30.0.47

-s 8972 plus 28 bytes of headers is exactly 9000. -M do sets don’t-fragment, so a path that cannot carry the frame fails rather than silently fragmenting. Without -M do the test proves nothing.

The Ceph symptom of an MTU mismatch is characteristic: small operations succeed, large ones hang, OSDs flap under load, and everything looks fine when the cluster is idle.

Bonding

ModeSuitability for Ceph
active-backup (1)safe, no aggregation — half the capacity idle
802.3ad LACP (4)the standard choice; needs switch configuration
balance-alb (6)no switch config, unpredictable under load
balance-rr (0)reordering breaks TCP performance — avoid

For LACP, the hash policy matters. The default layer2 hashes on MAC addresses, so all traffic between two hosts lands on one link — useless when OSD hosts talk mostly to each other.

cat /proc/net/bonding/bond0
# use layer3+4 so per-connection hashing spreads across members

xmit_hash_policy=layer3+4 hashes on IP and port, which spreads Ceph’s many concurrent connections across the bond members properly.

VLANs

VLAN separation for the public and cluster networks is standard practice. The Ceph-specific pitfall is binding: daemons select their address by matching against the configured CIDR, so a host whose VLAN interface is missing or misnumbered fails to start rather than falling back.

ip -br addr show
ceph config get global public_network
ceph config get global cluster_network

Verify that each host has exactly one address matching each CIDR.

The diagnostic table

SymptomLikely link-layer cause
Large writes hang, small ones workMTU mismatch
Throughput capped at one link’s ratebond hash policy
OSD fails to start, bind errorVLAN/addressing mismatch
Intermittent flapping under loadMTU or a marginal bond member
One host slow, others finebond member down — check /proc/net/bonding

Quiz

Knowledge check · 4 questions

  1. Q1. Small Ceph operations succeed but large writes hang, and OSDs flap under load while the cluster is fine when idle. What should you check first?

  2. Q2. On an LACP bond carrying cluster-network traffic, xmit_hash_policy must be set to layer3+4 before the second member carries any of it.

  3. Q3. Investigate throughput that will not exceed a single link.

    OSD hosts each have a 2 × 25 Gb LACP bond on the cluster network. During recovery, aggregate throughput between any pair of hosts never exceeds about 25 Gb/s, though many hosts are involved and total cluster throughput is higher. Switch counters show one bond member carrying almost all traffic per host pair.

  4. Q4. Why can an MTU mismatch cause hangs rather than a clear error?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Put the ping -M do MTU test and a bond member check into the post-change verification list for every network maintenance, on both the public and cluster networks. Record the expected MTU, bond mode, hash policy, and VLAN assignments in the host build configuration so a reprovisioned host cannot silently come back with different link-layer settings.

Cross-course references

  • Kubernetes: overlay network MTU mismatches produce the identical hang-on-large-payload symptom
  • Linux: these are ordinary bonding and VLAN configuration concerns, verified with the same tools