Runbook: Fencing failure - recover a fenced node

This runbook recovers from a fencing failure. Fencing can fail because of a broken STONITH device, a false positive, or a network issue. The recovery is to bring the node back into the cluster.

When to use this runbook

Use this runbook when:

• A node is fenced and will not rejoin.
• STONITH device is broken.
• Fencing is happening repeatedly without quorum loss.

Inputs

Gather before starting:

• Cluster name and node names.
• The BMC / iLO / iDRAC credential file, /etc/pacemaker/bmc-pw, mode 0600 and owned by root. Not the password itself.
• Network access to the BMC.
• Recent cluster logs.

Procedure

Step 1: Identify the fenced node

# Cluster status
pcs status

# Or corosync status
corosync-cmapctl | grep members

Identify which node is offline.

Step 2: Check STONITH logs

sudo journalctl -u pacemaker | grep -i stonith
sudo journalctl -u corosync | grep -i member

Look for:

• Which device was used.
• Why it triggered.
• Errors.

Step 3: Check the STONITH device

Read the BMC password from the root-only credential file the cluster already uses. Do not type -P <password>: mid-incident is exactly when a password gets pasted into a shared terminal, where it enters ~/.bash_history and is visible in ps output to everyone else on the box.

# Test the device manually
sudo ipmitool -I lanplus -H <bmc-ip> -U admin \
  -f /etc/pacemaker/bmc-pw chassis power status

If the file is missing on this node, create it here rather than falling back to -P:

sudo install -d -m 0700 /etc/pacemaker
sudo install -m 0600 /dev/null /etc/pacemaker/bmc-pw
read -rs -p 'BMC password: ' BMC_PW; echo
printf '%s' "$BMC_PW" | sudo tee /etc/pacemaker/bmc-pw >/dev/null
unset BMC_PW

If the device is unresponsive:

• Network issue (can the cluster reach the BMC?).
• Credentials issue.
• BMC hang (may need to be rebooted).

Step 4: Power on the node manually

sudo ipmitool -I lanplus -H <bmc-ip> -U admin \
  -f /etc/pacemaker/bmc-pw chassis power on

If the device is broken, use out-of-band access (rack PDU, cloud console, etc.) to power cycle.

Step 5: Wait for the node to rejoin

sleep 60
pcs status

The node should rejoin the cluster once it is up and corosync is running.

Step 6: Verify

# Cluster status
pcs status

# Resource status
pcs resource status

# Quorum
pcs quorum status

pcs resource show was removed in pcs 0.11. Use pcs resource status for state and pcs resource config for configuration; the same split applies to pcs stonith and pcs constraint.

Verify:

• All nodes online.
• Quorum is healthy.
• Resources are correctly placed.

Step 7: Document

In the incident log:

• Which node was fenced.
• Why (false positive, device failure, real failure).
• Recovery time.
• Improvements (replace STONITH device, fix false positive cause, etc.).

If the cluster is inquorate and will not start resources

This is the point where an operator, under pressure, searches for “force quorum” and finds pcs quorum unblock. Read this before running it.

Common patterns

Knowledge check