Network change rollback

Why this matters in production

A network change can remove the access needed to reverse it, which makes the rollback path a prerequisite rather than a contingency.

Preserving the path

Before any network change:
  a management path independent of the change
  the previous configuration captured
  out-of-band access to every affected device
  a colleague who can act if you lose access

# capture the current state on every host
{
  ip -br addr
  ip -br link
  ip route
  cat /proc/net/bonding/bond0 2>/dev/null
} > /tmp/net-before-$(hostname -s).txt

# and the switch configuration, from the switch
# show running-config > backup

Out-of-band access — IPMI, a console server, a separate management
network — is what makes a change that removes in-band access
recoverable.

Executing with reduced connectivity

# a change that may cut your session, applied with a timer
( sleep 300; ip addr flush dev bond0; ip addr add 10.0.2.23/24 dev bond0 ) &
# apply the change
# if it works, cancel the revert

# or, on network devices, a commit-confirm mechanism where available
# commit confirmed 5

The pattern is to arrange the reversal to happen automatically unless actively cancelled, so losing access results in restoration rather than a permanent outage.

When the change removes your access

Symptoms:
  the session hangs
  the host is unreachable
  the cluster reports the OSDs down

# from another host
AFFECTED_HOST=stor-04
ping ${AFFECTED_HOST}
ceph -s

The last row is why the first three are prerequisites.

Verifying the restoration

# connectivity restored
for h in $(ceph orch host ls --format json | python3 -c \
  'import sys,json;[print(x["addr"]) for x in json.load(sys.stdin)]'); do
  printf '%-16s ' "$h"
  ping -c 2 -W 2 "$h" >/dev/null 2>&1 && echo OK || echo FAIL
done

# and against the captured state
diff <(ip -br addr) /tmp/net-before-$(hostname -s).txt

# the cluster
ceph -s
ceph osd tree | grep -c down
ceph health detail

Quiz

Production discipline

Arm an automatic revert before applying a network change over the connection it affects — losing access then results in restoration rather than an unreachable host. Verify out-of-band access works before starting; it is the one path the change cannot remove.

Cross-course references

• Kubernetes: changes to the network the control plane uses need the same precautions
• Linux: firewall and routing changes applied remotely are the classic self-lockout