Individual link and NIC maintenance

Why this matters in production

Replacing a cable or transceiver on a live host is routine when the bond carries the traffic, and produces an outage when it does not.

Taking a member out

# check the current state
cat /proc/net/bonding/bond0 | grep -A4 'Slave Interface: eth0'

# take it down cleanly
ip link set eth0 down

# verify the bond adapted
cat /proc/net/bonding/bond0 | grep -E 'Slave Interface|MII Status'
ip -s link show bond0

Taking the member down administratively is cleaner than unplugging it: the bond converges on a link-down event rather than on a timeout.

Verifying traffic has moved

sar -n DEV 1 10 | grep -E 'eth0|eth1|bond0'

eth0 should show zero traffic
eth1 should show the bond's full traffic
bond0 should be unchanged

# and Ceph is unaffected
ceph -s
ceph osd perf | sort -k2 -rn | head -3

A latency increase at this point means the surviving member cannot carry the load, and the replacement should be deferred to a quieter window.

Replacing the component

# identify the physical port
ethtool -p eth0 30    # blink the LED for 30 seconds
ethtool -i eth0 | grep -E 'driver|bus-info'
lldpctl eth0 2>/dev/null | grep -E 'PortDescr|SysName'

The LLDP output names the switch and port, which confirms which cable to
pull at the far end.

# after replacing
ip link set eth0 up

Restoring and verifying

cat /proc/net/bonding/bond0 | grep -E 'Slave Interface|MII Status|Speed|Duplex'
ethtool eth0 | grep -E 'Speed|Duplex|Link detected'

# error counters should be clean on the new component
ethtool -S eth0 | grep -iE 'err|drop|crc'

# and traffic distributes across both again
sar -n DEV 1 10 | grep -E 'eth0|eth1'

# throughput restored
# a peer on the same cluster network, running `iperf3 -s`:
PEER=192.0.2.23

iperf3 -c "$PEER" -t 20 -P 8

A replacement that links at a lower speed, or negotiates half duplex, or accumulates errors, is worse than the component it replaced.

Quiz

Production discipline

Take a bond member down administratively before unplugging it — the convergence is immediate where a physical removal may wait for an LACP timeout. Verify speed, duplex, and error counters on a replacement component; linking up is not the same as working correctly.

Cross-course references

• Kubernetes: draining before removing is the same convergence principle
• Linux: clean removal of a redundant component beats abrupt removal everywhere