high riskcluster affecting~45 min

Runbook: Cluster node failure - triage, recover, return to service

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm from a surviving node whether the cluster is still quorate
· Confirm every resource is running on exactly one node right now
· Confirm out-of-band console/BMC access to the failed node before touching it
· Confirm whether a fence action has already fired for this node
· Confirm nobody else is already recovering the same node

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Establish the survivors view first: pcs status, quorum, and stonith history
2Classify: hardware/OS failure, fenced by the cluster, or network partition
3Confirm resources have migrated and are running exactly once
4Investigate the node itself via BMC console, not by rebooting it blindly
5Collect evidence before returning the node - the journal ends at the failure
6Return the node in standby, so it rejoins membership without taking resources
7Verify it is healthy under no load, then unstandby
8Clear failed actions and fence history, and confirm fencing is armed
9Write the timeline

4 · Verification

Confirm the procedure actually fixed the problem.

✓pcs status shows every node Online and no failed actions
✓pcs quorum status shows the full expected vote count
✓Every resource appears exactly once in pcs status resources
✓All fence devices report Started and stonith-enabled is true
✓No standby flag remains on the recovered node
✓The returned node has passed its own health checks under real load

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the node destabilises the cluster after rejoining, put it back in standby immediately: pcs node standby <node>
↶If a resource fails on the returned node, move it back to a known-good node and clear the constraint afterwards
↶If a configuration change made things worse, restore the CIB captured in the pre-checks

6 · Escalation

When the runbook isn't enough, contact:

· Resources are running on more than one node against shared storage: active split brain. Escalate immediately and stop the cluster on one side.
· The node was fenced repeatedly: escalate rather than returning it - a fence loop damages data and availability
· Hardware fault indicated by the BMC event log: escalate to the hardware vendor and do not return the node
· Fencing did not complete: escalate; the cluster cannot safely recover resources from a node it could not fence

A node has gone. The urgent question is not “why” — it is “is every resource running exactly once, and is the cluster still able to act”. Answer that first, then diagnose.

Step 1: The survivors’ view, first

Always start from a node that stayed in the majority. The failed node’s own view, if you can get it, is a partition’s view.

Read-only / Safepcs status— Reads cluster state, quorum and fence history.

pcs status --full
pcs quorum status
sudo corosync-quorumtool -s

# Has the cluster already fenced this node?
sudo pcs stonith history show
pcs stonith status

# What the survivors logged at the moment of the loss
sudo journalctl -u pacemaker -u corosync --since "30 min ago" --no-pager | tail -80

Step 2: Confirm resources are running exactly once

This takes precedence over diagnosis. A resource running in two places on shared storage is a data-loss event in progress.

Read-only / Saferesource census— Reads resource placement across the cluster.

pcs status resources

# On any node you can still reach that is NOT in the cluster's view,
# check whether it is still running the resource locally
sudo systemctl is-active <app>.service
sudo findmnt /srv/shared
ps -eo pid,comm,args | grep -i <app>

Cluster-wide riskstop the losing side— Stops cluster services on one partition. Use only to end an active split brain.

# On the INQUORATE side only, confirmed from quorum status
sudo pcs cluster stop
sudo umount /srv/shared
findmnt /srv/shared || echo 'unmounted'

Step 3: Classify the failure

Evidence	Classification
`pcs stonith history show` records a completed fence for this node; the node’s journal stops abruptly with no shutdown sequence	Fenced by the cluster
No fence record; survivors report links down; the node is up and healthy on its console	Network partition
No fence record; the node is powered off, panicked, or unresponsive at the console	Node/hardware failure
Fence record, then another, then another	Fence loop — escalate, do not return the node

The distinction matters because it decides what you fix. A fenced node is a symptom; the cause is whatever made the survivors lose it.

Step 4: Investigate the node itself

Go via the BMC console. Rebooting the node destroys the evidence, and if it was fenced, its journal already ends at the moment of the reset.

Read-only / SafeBMC evidence— Reads hardware event and power state via the BMC. -f keeps the password out of the process list.

# Power state and hardware events - read-only
sudo ipmitool -I lanplus -H <bmc-ip> -U admin -f /etc/pacemaker/bmc-pw \
chassis power status
sudo ipmitool -I lanplus -H <bmc-ip> -U admin -f /etc/pacemaker/bmc-pw \
sel list | tail -30
sudo ipmitool -I lanplus -H <bmc-ip> -U admin -f /etc/pacemaker/bmc-pw \
sdr type Temperature

Never pass a BMC password with -P. It lands in /proc/<pid>/cmdline, readable by every local user, and in your shell history. That credential is power control for every node in the cluster.

Once the node is reachable, collect before you change:

Read-only / Safenode evidence— Reads local logs and hardware state on the recovered node.

# The previous boot - the one that ended
sudo journalctl -b -1 -p err --no-pager | tail -60
sudo journalctl -k -b -1 --no-pager | grep -iE 'panic|oom|hardware error|mce|watchdog'

# Did the machine reset without a clean shutdown
last -x reboot shutdown | head

# Ring and membership from this node's point of view
sudo corosync-cfgtool -s
sudo ss -lunp | grep corosync

Step 5: Return the node in standby

Do not let a node that just failed take resources back the moment it rejoins. Standby lets it join membership and contribute its quorum vote while running nothing.

Cluster-wide riskstandby then start— Adds the node back to membership without placing resources on it.

# From a surviving node, BEFORE the failed node's cluster stack starts
sudo pcs node standby <node> --wait=300

# On the recovered node
sudo pcs cluster start
sudo pcs status

# Confirm it is Online (standby) and holds no resources
pcs status --full | grep -A2 '<node>'

Let it sit there while you check it. Memory, disk, the network path that failed, the service dependencies:

Read-only / Safenode health— Reads local health on the standby node.

uptime
free -h
df -h /
df -i /
sudo corosync-cfgtool -s
systemctl --failed --no-pager
sudo journalctl -p err -b --no-pager | tail -30

Step 6: Return it to service

Cluster-wide riskunstandby— Makes the node eligible to run resources again.

sudo pcs node unstandby <node> --wait=300
pcs status --full

Watch what the cluster does next. If a resource moves back immediately and then fails, put the node straight back into standby rather than letting it flap.

Step 7: Clean up and re-arm

Cluster-wide riskcleanup— Clears failed-action records and fence history cluster-wide.

sudo pcs resource cleanup
sudo pcs stonith history cleanup

# Fencing must be armed and every device Started
pcs property config stonith-enabled
pcs stonith status

# No leftover placement constraints from the incident
sudo pcs constraint config --full | grep -i 'cli-ban\|cli-prefer'

Step 8: Write the timeline

When the survivors lost the node, from the pacemaker log.
Whether a fence fired, when, and whether it completed.
Which resources moved, and how long recovery took.
What the BMC event log and the node’s previous boot show.
The cause, and the change that prevents a recurrence.

A node failure with no recorded cause will happen again, and the second time nobody will remember the first.

Common patterns

Symptom	Likely cause	Resolution
Node fenced, comes back, fenced again	Fence loop: the underlying fault is still present	Keep it in standby; escalate
Node lost, no fence record, node healthy at console	Network partition	Fix the ring path; check `corosync-cfgtool -s`
Node Online but resources will not run on it	Leftover `cli-ban`, or node still in standby	`pcs constraint config --full`; `pcs node unstandby`
Resource running on two nodes	Fencing failed or was disabled	Stop the inquorate side now; escalate
Node rejoins, cluster becomes unstable	Returned too early, still faulty	`pcs node standby` immediately
Fence completed but resources did not migrate	Constraint or dependency prevents placement	`pcs constraint config --full`, check colocation/order
Journal on the failed node ends mid-sentence	It was reset, not shut down	Expected after a fence; use `journalctl -b -1`