OOB power control - the commands that change state

Everything else in this part is diagnosis. This lesson is the part that acts, and it deserves a different level of care than anything else you run from a terminal.

An IPMI power command does not ask the operating system for permission, does not wait for filesystems to flush, and does not appear in any log on the host it kills. It is indistinguishable from someone walking to the rack and holding the button in. The BMC does exactly what it is told, immediately, to whichever address is in the -H argument.

Read-only first, always

Every one of these changes nothing. Run them before you run anything else, both because they tell you whether the BMC path works and because they are how you confirm you are talking to the host you think you are.

$ BMC=192.0.2.50
CRED="-I lanplus -H $BMC -U admin -f /etc/ipmi/bmc.pw"

ipmitool $CRED chassis status
ipmitool $CRED fru print 0
ipmitool $CRED mc info

System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Restore Policy : previous
Last Power Event     : command
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive

FRU Device Description : Builtin FRU Device (ID 0)
Product Manufacturer  : ACME
Product Name          : R740xd
Product Serial        : ABC1234

Device ID                 : 32
Firmware Revision         : 4.40
IPMI Version              : 2.0
Manufacturer Name         : ACME

Two fields in chassis status earn a second look.

Last Power Event tells you whether the host you are investigating went down by itself or was told to. command means somebody or something issued a power command — which, during an unexplained outage, is the difference between a hardware fault and a fence action nobody expected.

Power Restore Policy decides what happens after a power cut. previous restores whatever the state was; always-off means a rack that loses power comes back with every host dark and needing a manual power-on. Audit it across the fleet before the first power event, not after.

The narrower query, for scripting and monitoring:

# Substitute your own values before running:
BMC=192.0.2.50

ipmitool -I lanplus -H "$BMC" -U admin -f /etc/ipmi/bmc.pw chassis power status
# Chassis Power is on

The power verbs

These change state. Each line below is a production outage if it is aimed at the wrong host.

The line that divides that table is whether the operating system is given a chance to act.

soft is the polite one: the BMC raises an ACPI power-button event, systemd-logind handles it, services stop, filesystems sync and unmount, and the host powers off in the same way as systemctl poweroff. It is what you want for a planned action on a host that is still healthy.

Everything below it in the table is a hard stop. Dirty page cache is lost, journals replay on the next boot, a database recovers from its write-ahead log, and any filesystem without journalling needs a full check. That is sometimes exactly the right answer — on a hung host, soft does nothing, because there is no responsive kernel to receive the ACPI event.

Targeting: the discipline that matters more than the command

The failure mode here is not choosing the wrong verb. It is choosing the right verb and the wrong -H. BMC addresses are consecutive, hostnames differ by one character, and the command is usually reached with a shell history search at two in the morning.

Three habits make a wrong-host power command hard to issue.

1. Confirm identity from the BMC, not from your notes. Before any state change, print the serial the BMC reports and compare it against the asset record for the host you intend to act on:

# Substitute your own values before running:
BMC=192.0.2.50
EXPECT_SERIAL=ABC1234

got=$(ipmitool -I lanplus -H "$BMC" -U admin -f /etc/ipmi/bmc.pw fru print 0 \
      | awk -F': *' '/Product Serial/ {print $2; exit}')
if [ "$got" != "$EXPECT_SERIAL" ]; then
  echo "REFUSING: $BMC reports serial '$got', expected '$EXPECT_SERIAL'" >&2
  exit 1
fi
echo "confirmed $BMC is $EXPECT_SERIAL"

That check costs one second and turns a class of outage into an error message. Wrap your power tooling in it.

2. One credential file per host, named after the host. A single shared bmc.pw means every BMC in the estate is one -H typo away. Per-host files make the wrong target fail authentication instead of succeeding:

sudo install -d -m 0700 /etc/ipmi
sudo install -m 0600 /dev/null /etc/ipmi/node1.pw

3. Say the state out loud before and after. Query, act, query. The pattern is three commands, never one, and the before-query is what catches the case where the host you were about to reboot is already off because a colleague got there first.

Boot device selection

Occasionally you need the host to come up somewhere other than its usual disk — a PXE-booted rescue image, or the firmware setup screen to fix the console redirection from the previous lesson.

# Substitute your own values before running:
BMC=192.0.2.50
CRED="-I lanplus -H $BMC -U admin -f /etc/ipmi/bmc.pw"

# What are the current boot flags?
ipmitool $CRED chassis bootparam get 5

# Next boot only: network boot, then back to normal
ipmitool $CRED chassis bootdev pxe

# Next boot only: stop in the firmware setup screen
ipmitool $CRED chassis bootdev bios

chassis bootdev sets a flag the firmware consumes on the next boot and clears, in the same one-shot way that grub-reboot sets a single boot entry. That is deliberate and it is what you want: a host that PXE-boots once to be re-imaged, not a host that PXE-boots forever because somebody made it permanent during an incident.

It also means the flag expires. Set bootdev pxe, get distracted for an hour, and some BMCs will have timed the flag out before the host actually reboots — so verify with chassis bootparam get 5 immediately before the power command, not fifteen minutes earlier.

The runbook shape

Before
  ipmitool ... chassis status            state, last power event, restore policy
  ipmitool ... fru print 0               serial matches the asset record
  pcs node standby <node>                if this is a cluster node
  pcs status nodes                       hosting nothing

Act
  ipmitool ... chassis power soft        healthy host, planned action
  ipmitool ... chassis power off         only when soft did not take effect
  ipmitool ... chassis power on          bring it back

After
  ipmitool ... chassis power status      confirm the state changed
  <serial console>                       watch it boot, do not assume
  pcs node unstandby <node>
  pcs status --full                      no Failed Resource Actions

The step people drop is “watch it boot”. A host that was hard-powered-off may come back into an fsck, a firmware setup screen, or a degraded array rebuild, and all three are visible on the serial console and invisible from a power status that cheerfully reports on.

Knowledge check