Bonding for redundancy and capacity

Intermediate⏱ ~16 minipethtool

What you'll learn

Compare bonding modes for Ceph traffic
Configure LACP correctly for a Ceph host
Understand what a bond does and does not protect against
Verify bond health operationally

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A bond makes a single NIC, cable, or switch port failure invisible to Ceph — no OSDs marked down, no recovery, no incident. That is worth a great deal, and the configuration details determine whether you get it.

The modes that matter

802.3ad (LACP, mode 4) — active-active aggregation with switch cooperation. Both members carry traffic, and a failure removes one member from the hash. This is the standard choice.

active-backup (mode 1) — one member carries everything, the other waits. No switch configuration needed, works across switches that cannot form a peer relationship. Half the capacity sits idle.

Avoid balance-rr (packet reordering destroys TCP throughput) and treat balance-alb as a fallback where switch configuration is impossible.

Configuring LACP

# /etc/systemd/network or your distribution's equivalent
# key settings:
#   Mode=802.3ad
#   LACPTransmitRate=fast
#   TransmitHashPolicy=layer3+4
#   MIIMonitorSec=100ms

cat /proc/net/bonding/bond0

Three settings deserve attention:

Setting	Value	Why
`xmit_hash_policy`	`layer3+4`	spreads Ceph’s many connections; layer2 pins host pairs to one member
`lacp_rate`	`fast`	1-second LACPDUs, so failures are detected in ~3 s rather than ~90 s
`miimon`	100	link-state checks every 100 ms

The switch must agree on the LACP configuration, and the hash policy should match on both ends for symmetric distribution.

What a bond protects against

Failure	Protected?
NIC port failure	yes
Cable or transceiver failure	yes
Switch port failure	yes
Single switch failure	only with MLAG across two switches
Whole-NIC-card failure	only if members are on different cards
Switch fabric misconfiguration	no

The last three are the ones people assume are covered and are not. Members on the same physical NIC card share a failure domain; a bond to a single switch protects against port failures and not switch failures.

Verifying

cat /proc/net/bonding/bond0 | grep -E 'MII Status|Slave Interface|Speed'
ip -br link show type bond

Both members should show MII Status: up. A bond running degraded — one member down — delivers half the throughput silently, which shows up as a slow host with no other symptom.

Quiz

Knowledge check · 4 questions

Q1. A 2 × 25 Gb LACP bond is benchmarked with a single iperf3 stream and reports 25 Gb/s. What does this indicate?

A bond member has failed, so the remaining one is carrying the whole stream on its own The hash policy is layer2, which pins this host pair to a single member of the bond The switch is not configured for LACP, so the bond has fallen back to active-backup Correct behaviour — LACP pins each connection to one member, so a single flow never exceeds one link
Q2. A bond to a single top-of-rack switch protects against that switch failing.

True False
Q3. Investigate one slow host among many identical ones.

One OSD host in a 20-host cluster shows consistently higher latency and lower throughput than its peers. The hardware is identical, the OSDs report no device problems, `ceph osd perf` shows all twelve of its OSDs slightly elevated, and the host is reachable and healthy.
Q4. Why does xmit_hash_policy layer2 perform poorly for Ceph cluster traffic?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Monitor bond member state and negotiated speed on every Ceph host — it is the cheapest check available and it catches a failure mode nothing else reports. Standardise the bond configuration in your host build so hash policy and LACP rate cannot drift between hosts, and verify them after any reprovisioning.

Cross-course references

Kubernetes: node-level network degradation shows the same uniform-slowness signature across all pods
Linux: bond configuration and monitoring are standard host administration with Ceph-specific stakes