Bonded links and what happens when one fails

Advanced⏱ ~17 minipethtooliperf3

What you'll learn

Explain how bond modes distribute traffic
Predict capacity after a member fails
Verify a bond is actually distributing traffic
Design bonds for the failure case

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A 2×25 GbE bond is described as 50 GbE and behaves as 25 GbE for any single flow. Both the aggregate and the per-flow figure matter, and confusing them produces capacity plans that do not hold.

How bonds distribute

cat /proc/net/bonding/bond0
ip -d link show bond0 | grep -A3 'bond '

Mode	Distribution	Per-flow limit
`active-backup` (1)	one member, others idle	one member’s speed
`balance-xor` (2)	by hash	one member’s speed
`802.3ad` / LACP (4)	by hash, with switch coordination	one member’s speed
`balance-alb` (6)	by hash, receive-load balanced	one member’s speed

The critical property shared by every mode: a single TCP flow uses one member. Aggregate bandwidth scales with member count; per-connection bandwidth does not.

# the hash policy determines how well flows spread
cat /proc/net/bonding/bond0 | grep -i 'hash policy'
# layer3+4 spreads by IP and port, which is what Ceph needs

layer2 hashing spreads by MAC address, which for Ceph means all traffic to a given host uses one member regardless of how many connections exist. layer3+4 includes ports and spreads properly.

Capacity after a failure

2×25 GbE LACP, both up:    50 Gb/s aggregate, 25 Gb/s per flow
2×25 GbE LACP, one down:   25 Gb/s aggregate, 25 Gb/s per flow

If the cluster’s normal traffic exceeds 25 Gb/s aggregate, the failure of one member produces congestion — during exactly the period when the cluster may also be recovering.

# normal utilisation
ip -s link show bond0
sar -n DEV 1 10 | grep bond0

The planning rule: size the bond so that normal traffic plus recovery traffic fits within n-1 members.

Verifying distribution

# per-member counters
cat /proc/net/bonding/bond0 | grep -A5 'Slave Interface'
ip -s link show eth0
ip -s link show eth1

Wildly uneven counters mean the hash is not spreading — usually a layer2 policy, or too few distinct flows.

# PEER is another cluster node running `iperf3 -s`; substitute your own:
PEER=198.51.100.12

# force multiple flows to test
iperf3 -c "$PEER" -P 8 -t 30

A single-stream iperf3 test on a bond measures one member and is routinely misreported as the bond’s capacity.

Designing for the failure case

Requirement	Design
Survive a member failure without congestion	size for `n-1`
Survive a switch failure	members on different switches (MLAG)
Maximum per-flow bandwidth	a faster single link, not more members
Simplicity	`active-backup` with adequate single-member capacity

# alert when a bond is degraded
cat /proc/net/bonding/bond0 | grep -c 'MII Status: up'

Quiz

Knowledge check · 4 questions

Q1. Why does `layer2` bond hashing fail to distribute Ceph traffic?

It cannot carry jumbo frames, so Ceph's large transfers fall back onto a single member It is only honoured in active-backup mode, where one member carries everything anyway All connections between two hosts share the same MAC pair, so they hash to one member It is chosen by the switch rather than the host, so setting the policy on the host has no effect
Q2. A 2×25 GbE LACP bond gives a single TCP connection 50 Gb/s.

True False
Q3. Size a bond for the failure case.

A cluster uses 2×25 GbE LACP bonds. Normal cluster network traffic peaks at 34 Gb/s aggregate per host. A member failure is considered acceptable because "the bond is redundant".
Q4. What is the sizing rule for a bond that must survive a member failure?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Set the bond hash policy to layer3+4 at both host and switch; layer2 puts all traffic between a host pair on one member. Size bonds so normal plus recovery traffic fits within n-1 members, and use iperf3 -P 8 — a single-stream test measures one member and under-reports by the member count.

Cross-course references

Kubernetes: node network capacity after a link failure needs the same n-1 sizing
Linux: multipath I/O presents the identical aggregate-versus-per-flow distinction