Runbook: Pacemaker failover - test and recover

This runbook tests and recovers from Pacemaker failover. The test verifies that the cluster handles failure correctly; the recovery is the actual failover.

When to use this runbook

Use this runbook when:

• A node has failed and resources need to be migrated.
• Testing failover in a quarterly drill.
• Recovering from a planned downtime.

Inputs

Gather before starting:

• Cluster name and node names.
• The BMC credential file, /etc/pacemaker/bmc-pw, mode 0600 and owned by root. Not the password itself.
• Resource names and their dependencies.
• A maintenance window (for testing).
• Communication channels (war room, on-call).

Procedure

Step 1: Capture the baseline

sudo pcs status
sudo pcs resource status
sudo pcs resource config

Document the current state. The test will compare against this. pcs resource show was removed in pcs 0.11: use pcs resource status for state and pcs resource config for configuration. The same split applies to pcs stonith and pcs constraint.

Step 2: Choose the failover method

Two methods:

• Graceful move: pcs resource move <resource> <node>. The resource is stopped on the current node and started on the new node. Used for planned failover.
• STONITH: pcs stonith fence <node>. The node is powered off; the cluster considers it dead; resources migrate. Used for unplanned failover.

Both belong in a quarterly test, but they are not interchangeable and they answer different questions. The graceful move proves the resources can run on the other node. Only a real fence proves the fence device works — the credential, the network path to the BMC, and the host mapping — and that is the component whose failure blocks every recovery the cluster would otherwise perform.

The cost of that proof is a hard power-off. Do the graceful move first; if it fails, fix that before fencing anything.

Step 3: Test graceful move

pcs resource move does not “push” a resource anywhere. It creates a location constraint - a cli-ban rule with score -INFINITY against the node the resource is currently on - and the cluster then has nowhere else to run it but the target. The move is a side effect of a rule that is still in the CIB afterwards.

Whether that rule clears itself depends on the version: older pcs 0.10 leaves it in place until you remove it, while newer Pacemaker auto-clears a completed move. Do not rely on either. Bound it explicitly:

# Preferred: the constraint expires on its own.
sudo pcs resource move web node2 --lifetime=PT30M

# Or move without a lifetime, then clear it yourself in this same step.
sudo pcs resource move web node2

Verify, and look at the constraints as well as the status:

sudo pcs status
sudo pcs constraint config --full | grep -i 'cli-ban\|cli-prefer'

The resource should be on node2, and you should be able to name every cli-ban or cli-prefer entry that command prints. Anything you cannot account for is left over from an earlier move.

Step 4: Test STONITH

There is no dry run. pcs stonith fence cuts power to the node immediately — no graceful stop, no resource shutdown, every process killed where it stands. Confirm all four preconditions before running it:

sudo pcs quorum status          # survivors must stay quorate without node1
sudo pcs status                 # no resource already Stopped or failed
sudo pcs stonith status         # the device for node1 reads Started
sudo fence_ipmilan --ip=<node1-bmc> --username=<user> \
   --password-script=/etc/cluster/bmc-pw.sh --lanplus --action=status

You also need console or BMC access to power node1 back on, and a maintenance window. One node at a time.

sudo pcs stonith fence node1

Verify:

• The node is actually powered off (check via BMC).
• Resources have migrated to other nodes.
• sudo pcs stonith history show records the action as completed, not failed.
• After powering node1 back on, it rejoins and sudo pcs status shows no failed actions once you have run sudo pcs resource cleanup.

Step 5: Verify the service, then clear the constraint

After the failover, verify the service works:

# Smoke test
curl -I http://<vip>/health

# Synthetic monitoring (if configured)

# Real traffic (if in production)

The moment the smoke test passes, remove the move constraint. This is not a tidy-up task for the end of the runbook; it is part of verifying the failover:

sudo pcs resource clear web
sudo pcs constraint config --full | grep -ci 'cli-ban\|cli-prefer'   # must be 0

The resource stays where it is - clearing the ban only restores the cluster’s freedom to place it. If it immediately moves back, that tells you the cluster’s own preference differs from where you put it, which is worth knowing before an incident does the experiment for you.

If the service does not work:

• Check the resource agent logs.
• Check that constraints are correctly applied.
• Check that the new node has the prerequisites.

Step 6: Test reverse

Move the resource back:

sudo pcs resource move web node1 --lifetime=PT30M

pcs resource clear removes the constraint from the earlier move; it does not move the resource. To relocate it deliberately you issue a new move, and that move creates a new constraint - so it needs the same --lifetime or the same explicit clear afterwards:

sudo pcs resource clear web
sudo pcs constraint config --full | grep -ci 'cli-ban\|cli-prefer'   # must be 0

Verify the service works on the original node.

Step 7: Bring the failed node back

If a node was fenced:

# Power on via BMC. -f reads the password from the root-only
# credential file; -P would put it in /proc/<pid>/cmdline for
# every local user, and in this shell's history.
sudo ipmitool -I lanplus -H <bmc-ip> -U admin \
  -f /etc/pacemaker/bmc-pw chassis power on

# Wait for the node to rejoin
sleep 60
sudo pcs status

That credential is power control for every node in the cluster. A leak turns a routine failover drill into someone else’s ability to fence your cluster at will.

Step 8: Document

In the run log:

• Time of test.
• Method (graceful or STONITH).
• Time to recover.
• Findings.
• Improvements.

Common patterns

Knowledge check