OOB credentials and break-glass - access that survives the outage

Out-of-band management has three requirements, and most designs only think hard about the first two. The network has to be separate. The hardware has to be independent of the host. And the credential has to be reachable at the moment the host is not.

That third requirement is where designs quietly fail. It is easy to build a beautiful management network, put every BMC on it, generate strong unique passwords, store them properly — and end up with an estate you cannot log into during the exact incident the estate was designed for.

The dependency loop

Draw the path from “I need to power-cycle node1” to “I have node1’s BMC password” and look for anything on it that depends on the systems being recovered.

A common and entirely well-intentioned arrangement:

Operator
  -> VPN concentrator            (authenticates against AD)
  -> jump host                   (SSH keys from the config-management repo)
  -> password vault              (runs on the Kubernetes cluster)
  -> BMC password
  -> BMC                         (authenticates against AD)
  -> node1

Every arrow is reasonable in isolation. Together they mean that an Active Directory outage, or a failure of the cluster hosting the vault, removes your ability to reach the hardware that would let you fix it. The BMC is up, the management network is up, and you cannot log in.

The failure is not hypothetical or rare. It has a name in incident reports — “we could not get to the consoles” — and it extends outages by hours.

Auditing the accounts on a BMC

ipmitool reads and writes the BMC’s own user table. Start read-only.

$ BMC=192.0.2.50
CRED="-I lanplus -H $BMC -U admin -f /etc/ipmi/bmc.pw"

ipmitool $CRED user list 1
ipmitool $CRED channel getaccess 1 2

ID  Name      Callin  Link Auth  IPMI Msg   Channel Priv Limit
1                ard    false      false       true       NO ACCESS
2   admin           false      false       true       ADMINISTRATOR
3   monitoring      true       false       true       USER
4   root            false      false       true       ADMINISTRATOR

Maximum User IDs     : 16
Enabled User IDs     : 3

User ID              : 2
User Name            : admin
Fixed Name           : No
Access Available     : callback
Link Authentication  : enabled
IPMI Messaging       : enabled
Privilege Level      : ADMINISTRATOR

Three findings from a table like that one, in order of severity:

• A root account with ADMINISTRATOR that nobody uses. It is either a vendor default or a leftover. Disable it.
• A monitoring account with more privilege than it needs. Reading sensors requires USER; it does not require ADMINISTRATOR, and an over-privileged monitoring credential is stored in a monitoring configuration file on a host, which is the least protected place any BMC credential lives.
• User ID 1 with any access at all. ID 1 is the anonymous user. It should be NO ACCESS on every channel.

The write operations, for when you have a finding to fix:

# Substitute your own values before running:
BMC=192.0.2.50
CRED="-I lanplus -H $BMC -U admin -f /etc/ipmi/bmc.pw"
USER_ID=5

# Create a break-glass account. The password is read from a
# prompt, never from argv - see linux-ipmi-and-bmc-basics.
ipmitool $CRED user set name "$USER_ID" breakglass
read -rs -p 'new BMC password: ' NEWPW; echo
ipmitool $CRED user set password "$USER_ID" "$NEWPW"
unset NEWPW

# Grant it ADMINISTRATOR (privilege level 4) on LAN channel 1
ipmitool $CRED user priv "$USER_ID" 4 1
ipmitool $CRED user enable "$USER_ID"

# Verify, read-only
ipmitool $CRED user list 1
ipmitool $CRED channel getaccess 1 "$USER_ID"

Privilege levels are numeric: 1 CALLBACK, 2 USER, 3 OPERATOR, 4 ADMINISTRATOR. USER can read sensors and query power state; OPERATOR adds power control; ADMINISTRATOR adds user and channel management. Give the monitoring account 2 and stop there.

What a break-glass path needs

Four properties, and it is the fourth that most designs skip.

1. Offline. A copy that does not depend on the network, the directory, the vault, or any host in the estate. A sealed envelope in the data centre safe and a second one off-site is not old-fashioned; it is the only storage medium with no dependency on the thing you are recovering. An encrypted file on a laptop that is never joined to the domain is an acceptable modern equivalent, provided the decryption key is not in the vault.

2. Complete. The password alone is useless. The offline copy needs the BMC address, the account name, the password, and the network path — including which VLAN, which jump host, and which route works when the production network is down. An operator who has the password and does not know the BMC is on 192.0.2.0/24 reachable only from the management jump host is still locked out.

3. Audited. Opening the envelope has to leave a trace, and the credential has to be rotated afterwards. Break-glass access that leaves no record is indistinguishable from a compromise, and a break-glass password that is never rotated becomes a shared password that half the team knows.

4. Tested. This is the one that gets skipped, because testing it means opening the envelope, which means rotating the credential, which is work. Untested break-glass credentials fail for mundane reasons: the password was truncated by firmware, the BMC was re-imaged during a firmware update and reset to defaults, the address changed when the rack moved, the account was disabled by a hardening playbook that ran across the fleet.

The drill

Quarterly, and after any firmware campaign, prove the whole path rather than any single link of it. The rule is that the person running the drill uses only what the offline copy contains — no shell history, no saved SSH config, no vault.

1. Retrieve the offline copy. Record who opened it and when.
2. From the management jump host, and using only the address in
   that copy:
       ipmitool -I lanplus -H <bmc> -U breakglass -f <pw file> chassis status
   Expect: power state and chassis flags. Read-only, changes nothing.
3. Open the console with the same credential:
       ipmitool ... sol activate
   Expect: a login prompt or the current console output.
4. On one non-production host only, exercise a state change:
       ipmitool ... chassis power cycle
   Watch it come back on the serial console.
5. Rotate the break-glass password. Verify the new one from a
   second terminal before closing the first.
6. Re-seal, re-file both copies, and record the drill.

Step 2 is what finds a truncated password. Step 3 is what finds a BMC where SOL was never enabled. Step 4 is what finds the account with USER privilege that can read but not act — a finding that would otherwise surface during a real incident, at the point of no return.

Do not let the inventory live only on the fleet

The last dependency loop is the quietest one. The list of BMC addresses is usually in the configuration management repository, or a CMDB, or generated from the DHCP leases — all of which are services running on the estate.

Keep a flat, boring, exported copy somewhere that does not depend on any of it:

# Run from the control node, on a schedule; ship the output
# off-site alongside the break-glass envelopes.
ansible-inventory -i /etc/ansible/prod --list \
  | jq -r '._meta.hostvars | to_entries[]
           | [.key, .value.bmc_address, .value.rack] | @csv' \
  > /var/backups/oob/bmc-inventory-$(date -u +%Y%m%d).csv

A CSV in the safe next to the passwords costs nothing and removes the last reason an operator cannot reach the hardware.

And give the BMC network a static address plan. A BMC that gets its address from DHCP is a BMC whose address is unknown when the DHCP server is down, which is a state strongly correlated with “the hosts are unreachable”.

Knowledge check