Cluster IR: fencing failure and the fence loop

Fencing is the mechanism that makes cluster recovery safe, and it is the mechanism whose failure is hardest to see. When it breaks, nothing turns red. The cluster is quorate, no resource shows as failed, every node the cluster can see is online — and the service has been down for forty minutes.

linux-why-fencing-exists and linux-fencing-devices-and-agents cover the mechanism and the agents. This lesson covers the three shapes it takes in an incident, and what a fenced node does to your ability to find out what happened.

Three states that all look like “fencing did not work”

They have different causes and different fixes, and the first command tells them apart:

# pcs stonith history show

reboot of node3 pending: client=pacemaker-controld.2891, origin=node1
reboot of node3 failed: delegate=node1, client=pacemaker-controld.2891,
  origin=node1, completed='Tue Aug 11 06:02:19 2026'
reboot of node2 successful: delegate=node1, client=pacemaker-controld.2891,
  origin=node1, completed='Tue Aug 11 05:58:41 2026'

Pair it with the summary view, which is where a failed fence becomes visible in the status everyone is already staring at:

# pcs status --full | sed -n '/Failed Fencing Actions/,/^$/p'

Failed Fencing Actions:
* reboot of node3 failed: delegate=node1, client=pacemaker-controld,
  origin=node1, last-failed='Tue Aug 11 06:02:19 2026'

Why a pending fence stops everything

This is the part that surprises operators. Pacemaker will not start a resource elsewhere until it can prove the previous owner has stopped running it. A fence that has not returned is not proof, so the cluster waits — indefinitely, and quietly.

The consequence is counter-intuitive and worth stating plainly: one unreachable fence device can hold an entire quorate cluster in a stopped state, with nothing in the default status output saying so.

Test the device rather than assuming the cluster is at fault. A status call proves credentials and reachability without rebooting anything:

# Substitute your own values before running:
FENCE_DEV=fence-node3

pcs stonith status
pcs stonith config "$FENCE_DEV"
stonith_admin --list-registered

# Substitute your own values before running:
NODE=node3

# Which devices does the cluster believe can fence this node,
# and can they be contacted right now? This is a query, not a fence.
stonith_admin --list "$NODE"

Then check the causes in the order they actually occur:

1. BMC credentials. Rotated by a security process that did not know the cluster held a copy, or an account locked out after failed attempts.
2. Management network reachability. The surviving node has to reach the victim BMC, and the management VLAN is often the one that changed.
3. Wrong port or host in the device configuration. Common after hardware replacement, when the BMC address was reused or reassigned.
4. Fence agent package upgraded and parameter names changed. The device configuration still validates but the agent rejects the call.
5. The BMC itself hung. BMCs are small computers that also crash, and they crash more often than the hosts they manage.

# Substitute your own values before running:
NODE=node3

# Only after you have PHYSICALLY confirmed the node is powered off.
# This tells Pacemaker the node is down; it does not check.
stonith_admin --confirm "$NODE"

The fence loop

A fence loop is two nodes fencing each other in turn, or one node being fenced repeatedly on every rejoin. The history is the diagnosis — alternating names, minutes apart, all successful:

# pcs stonith history show | head

reboot of node1 successful: delegate=node2, completed='Tue Aug 11 06:24:03 2026'
reboot of node2 successful: delegate=node1, completed='Tue Aug 11 06:18:55 2026'
reboot of node1 successful: delegate=node2, completed='Tue Aug 11 06:13:40 2026'
reboot of node2 successful: delegate=node1, completed='Tue Aug 11 06:08:12 2026'

Two causes account for nearly all of them:

• A two-node cluster in a partition. Both halves stay quorate, both decide the other has failed, both fence. Whichever node boots first fences the other, which boots and fences back. The period of the loop is the boot time of the hardware.
• A node that rejoins into the same fault. The node is fenced, reboots, starts the cluster automatically, fails its monitor or its cluster link again immediately, and is fenced again. The underlying fault — a bad NIC, a failing disk, a corrupt configuration — is never given a chance to be looked at, because the node is never up for long enough.

Break the loop before you diagnose it. The tool is fence delay, which makes the race deterministic instead of symmetric:

# Substitute your own values before running:
FENCE_DEV=fence-node2

# Give one node a head start so both cannot fence simultaneously.
# The node whose device carries the delay is the one that loses.
pcs stonith update "$FENCE_DEV" pcmk_delay_base=10s

# Cluster-wide alternative: the node running more resources wins
# the race, and the other waits this long before fencing.
pcs property set priority-fencing-delay=15s

For the rejoin case, the fix is to stop the node rejoining automatically, so that a rebooted node comes up as a plain host you can log into and investigate:

# Run on the node that keeps being fenced, while it is up
sudo systemctl disable corosync pacemaker
sudo systemctl disable pcsd 2>/dev/null || true

Then, once you understand the fault and have fixed it, bring it back deliberately:

# Substitute your own values before running:
NODE=node2

# From a healthy node: start the cluster on the repaired node
sudo pcs cluster start "$NODE"
sudo pcs status nodes

What a fenced node means for the timeline

A fence is a power event, not a shutdown. This changes what evidence exists, and the change catches people out repeatedly.

• The victim’s logs stop mid-sentence. There is no shutdown sequence, no “stopping unit” messages, no final flush. Anything written to the journal but not yet on disk is gone. The last line you can read is not the last thing that happened.
• The reason for the fence is on the peers, not the victim. The decision was made by the node that requested the fence. journalctl -u pacemaker on the delegate names the resource action that failed or the membership event that triggered it.
• journalctl -b -1 on the victim is the boot that ended. That is where the pre-fence state lives. Do not read -b (the current boot) and conclude the node was fine.

# Substitute your own values before running:
NODE=node2

# On the fenced node, AFTER it comes back: the boot that was cut short
journalctl -b -1 --utc -o short-iso-precise -n 200

# Confirm the boot ended abruptly - a clean shutdown leaves a record,
# a fence leaves nothing
journalctl --list-boots | tail -3

# On the delegate node: why the fence was decided
journalctl -u pacemaker --utc -o short-iso-precise --since '-2 hours' \
  | grep -iE 'stonith|fenc|unclean|lost'

Knowledge check