OOB architecture - the out-of-band management layer

Out-of-band (OOB) management is the network and hardware for accessing hosts when the production network is unavailable. This lesson covers the design.

What OOB is

OOB is the dedicated network and hardware for:

• Power on / off (IPMI / iDRAC / iLO).
• Serial console access.
• Firmware updates.
• Hardware sensors.
• KVM (keyboard, video, mouse) over IP.

The OOB network is separate from the production network. If the production network is down, the OOB network is still up.

Design the OOB network

A production OOB design:

• Dedicated physical network (separate switches).
• Each host BMC is on the OOB network, on a dedicated BMC port rather than a shared LOM where the hardware allows it.
• Operator workstations have OOB access.
• The OOB network is not routed from the production network.
• The OOB network has its own monitoring.

Dedicated hardware for OOB

• Managed switches for the OOB network.
• A jump host for operator access.
• A VPN for off-site operator access.
• A dedicated firewall if the OOB is internet-connected.

Dedicated vs shared BMC port

Every server presents one of two BMC network layouts, and the choice is made when the machine is racked. It is the single most consequential OOB hardware decision.

A dedicated BMC port has its own PHY and its own switch port. It stays up when the host operating system panics, when the host NIC driver is reloaded, when the host is powered off, and when somebody reconfigures host networking badly.

A shared (sideband) port rides the host LOM over NC-SI. The BMC has no PHY of its own; it borrows the host NIC. That means it is lost when that NIC fails, when its cable or switch port fails, and sometimes when the host NIC is reset by a driver reload or a firmware update.

Read those failure lists side by side. The shared port drops out in precisely the scenarios OOB exists to survive. A rack wired that way looks like it has out-of-band management on the diagram and does not have it during the outage.

Production standard: dedicated BMC port, on a physically separate management switch, with the BMC on a separate power feed where the platform supports it. Where a shared port is unavoidable - some dense or edge hardware offers nothing else - record it as a known risk against those hosts and keep a serial console or a switched PDU as the real fallback.

OOB is a cluster dependency, not just an operator convenience

The obvious use of OOB is a human logging in when the production network is down. The less obvious use is the one that runs unattended.

Every fence agent in this course - fence_ipmilan, fence_idrac, fence_ilo4 and fence_ilo5, fence_redfish - reaches the target node’s BMC over the OOB network. Fencing is OOB traffic. If the OOB network is down, fencing fails.

A Pacemaker cluster that cannot fence will not recover resources. That is deliberate, and it is covered in linux-why-fencing-exists and linux-stonith-and-data-integrity: the cluster cannot prove the node is dead, so starting the resource elsewhere risks two writers on one dataset. It refuses. pcs status shows the peer as UNCLEAN (offline) and the resources stopped.

Three consequences follow.

Monitor BMC reachability from every cluster node, not only from the operator jump host. The jump host is on the operator path; the cluster nodes are on the fencing path, and those are not the same route. A firewall rule or VLAN change can break one and leave the other working.

# Run from each cluster node, alert on non-zero exit
ipmitool -I lanplus -H "$BMC" -U "$USER" -f /etc/ipmi/bmc.pw \
  chassis power status

Better still, ask the cluster itself. A stonith resource with a monitor operation checks the exact path fencing will use, from the node that will use it:

sudo pcs stonith status          # every fence device Started, none Failed
sudo pcs stonith config          # confirm each has a monitor operation

A fence device whose monitor is failing is an outage that has not happened yet. Alert on it with the same urgency as a failed resource, and add the check to the cluster monitoring checklist.

Security

OOB is critical infrastructure. The risks:

• Default credentials: change immediately.
• Outdated firmware: update regularly.
• Untrusted access: lock down to operator networks.
• Network exposure: keep OOB off the public internet.

For internet-connected OOB (e.g. cloud BMC), use:

• Strong authentication (certificates, MFA).
• Per-host credentials (not shared).
• Audit logging.
• Network segmentation.

Test OOB access

Quarterly, verify OOB works:

• Connect to a host BMC.
• Power cycle the host.
• Verify the host comes back.
• Access the serial console.
• Reach every node’s BMC from every other cluster node, not just from the jump host.
• Confirm each stonith device is Started and its monitor is passing.

If any of these fail, the OOB is not ready - and if the last two fail, the cluster cannot fail over, whatever the application dashboards say.

Knowledge check