Switch maintenance with redundant uplinks

Why this matters in production

A redundant fabric is only redundant if the redundancy has been verified, and switch maintenance is where an untested assumption becomes an outage.

Verifying the redundancy first

# on every Ceph host: which members does the bond have, and are both up?
cat /proc/net/bonding/bond0 | grep -E 'Slave Interface|MII Status|Speed'

# and which switch is each connected to
lldpctl 2>/dev/null | grep -E 'Interface|SysName'

The check that matters:
  each bond has members on both switches
  both members are up
  the surviving member alone can carry the load

# the last is the one usually untested
sar -n DEV 1 30 | awk '/bond0/ {rx+=$5; tx+=$6; n++} END {print (rx+tx)/n*8/1000000, "Mb/s average"}'
ethtool bond0 | grep Speed

If normal traffic exceeds a single member’s capacity, the maintenance produces congestion regardless of the link staying up.

Testing before committing

# take one member down deliberately, in a controlled way
ip link set eth0 down

# and observe
ceph -s
watch -n 5 'ceph -s | grep -E "osds|health"'
sar -n DEV 1 10

ip link set eth0 up

Doing this on one host first establishes whether the failover works before a whole switch is taken out.

Sequencing the maintenance

1. verify the cluster is HEALTH_OK
2. verify every host's bond has both members up
3. test failover on one host
4. set noout if the maintenance risks OSDs being marked down
5. take the switch out
6. monitor throughout
7. return the switch
8. verify every bond has both members up again

ceph -s | grep HEALTH_OK
ceph osd set noout
# switch maintenance
ceph osd unset noout

noout is prudent even with working redundancy: a brief packet loss during a failover can cause OSDs to be marked down.

Monitoring during it

watch -n 5 'ceph -s | grep -E "osds|health|slow"'

When redundancy is not sufficient

# find single-homed hosts
for h in $(ceph orch host ls --format json | python3 -c \
  'import sys,json;[print(x["hostname"]) for x in json.load(sys.stdin)]'); do
  printf '%-14s ' "$h"
  ssh "$h" 'grep -c "Slave Interface" /proc/net/bonding/bond0 2>/dev/null || echo 0'
done

Quiz

Production discipline

Measure actual traffic against a single bond member’s capacity before a switch maintenance — links staying up while every host is congested is the failure mode that redundancy does not prevent. Set noout regardless; LACP convergence can lose enough heartbeats to trigger a down-marking.

Cross-course references

• Kubernetes: verifying node redundancy actually absorbs the load before relying on it
• Linux: untested redundancy is an assumption, not a property