Bond member failure versus bond failure

Why this matters in production

A degraded bond is the most under-monitored failure in a typical Ceph deployment. Ceph does not see it, host reachability checks do not see it, and the only symptom is that one host is quietly slower than its peers — which is easy to attribute to almost anything else.

The two cases

One member fails. The bond continues on the remaining member.

• Host reachable, OSDs up, no Ceph events
• Throughput halved for that host
• Redundancy gone — the next failure takes the host offline
• Nothing alerts

All members fail. The bond goes down.

• Host unreachable on that network
• OSDs marked down within the heartbeat grace period
• Recovery begins after the down-out interval
• Very visible

Detecting the silent case

cat /proc/net/bonding/bond0

Slave Interface: ens1f0
MII Status: up
Speed: 25000 Mbps

Slave Interface: ens1f1
MII Status: down          ← this line
Speed: Unknown

For monitoring, the useful checks are:

# member count that is actually up
grep -c 'MII Status: up' /proc/net/bonding/bond0

# negotiated speed per member
for i in ens1f0 ens1f1; do
  echo -n "$i: "; ethtool "$i" | awk -F': ' '/Speed/{print $2}'
done

# carrier changes, which reveal flapping members
ip -s -d link show ens1f1 | grep -A1 'link/ether'

Alert on the up-member count dropping below the configured member count. It is a single number and it catches the case nothing else does.

The capacity consequence

A host on half its network capacity affects more than itself: it is slower to accept replication, so writes to any PG it participates in are slower, and with a few hundred PGs per OSD that is a substantial share of the pool.

ceph osd perf | sort -k2 -n | tail

The signature is every OSD on one host being uniformly somewhat elevated, with no single device standing out — which is exactly what a host-level network constraint looks like.

Quiz

Production discipline

Add bond member count and per-member negotiated speed to host monitoring on every Ceph node; it is a few lines of check and closes a gap that nothing else covers. Treat a degraded bond with disk-replacement urgency, because the host is running without network redundancy until it is fixed.

Cross-course references

• Kubernetes: node-level degradation invisible to the scheduler needs host-layer monitoring the same way
• Linux: multipath and bonding both require monitoring at their own layer for exactly this reason