OOB and DR - the out-of-band layer in disaster recovery

Out-of-band management is the layer that makes disaster recovery possible when the production network is down. This lesson covers the role of OOB in DR and the testing discipline.

The role of OOB in DR

In a DR scenario:

• The production network is down or partitioned.
• Hosts may be unreachable via production.
• OOB is the only way to access the hosts.
• The OOB network and BMCs are still up.

Without OOB, recovery is impossible. With OOB, recovery is possible (and may be straightforward).

Use cases

• Power cycle: bring a host back after a hang.
• Console access: see what is on the screen when the OS is unresponsive.
• Firmware update: install a critical fix without OS access.
• iPXE boot: boot from a network image for a clean install.
• Reset BMC: recover from a hung BMC.

DR drill with OOB

A full DR drill should include OOB:

1. Simulate the production network down.
2. Verify OOB network is still up.
3. Connect to a host’s BMC.
4. Power cycle the host.
5. Verify the host comes back.
6. Access the serial console.
7. Update firmware if needed.

If any step fails, the OOB is not ready.

OOB dependencies

The OOB network itself has dependencies:

• Switches (must be powered, configured).
• Power (must be on separate circuits).
• Operator access (must be available).
• BMC firmware (must be current).

The dependency runs the other way too, and it is the one most often missed: cluster fencing depends on OOB. Pacemaker fences through the BMC - fence_ipmilan, fence_redfish, fence_ilo5, fence_idrac all reach the target node over the OOB network. If that network is down, no node can be fenced, and Pacemaker will not recover resources from a node it cannot fence. The cluster does not fail over; it stops and waits.

So the OOB network is not only the recovery path for operators, it is a live production dependency of every HA cluster on the estate. Treat an OOB outage as a cluster incident, not a management-plane inconvenience:

• Monitor each node BMC continuously from the peers that would fence with it, not only from a central monitoring host.
• Check pcs stonith status alongside BMC reachability - a fence device that cannot reach its BMC still shows Started until it is used.
• Include a real fence action in the DR drill, not only a console login.

linux-checklist-cluster-monitoring carries these as items.

Document these dependencies in the DR plan, in both directions. Each is a potential failure point for OOB itself, and OOB is a failure point for the clusters above it.

Testing

Quarterly, test the full chain:

• Power cycle: can we power cycle a host via OOB?
• Console access: can we see the serial console?
• Firmware update: can we update BMC firmware?
• Network access: is the OOB network reachable?
• Disaster simulation: simulate a production outage and verify OOB still works.

Knowledge check