Break/Fix: the resources will not fail over and the node is UNCLEAN

A cluster cannot fail resources over; diagnose whether the cause is fencing, constraints, or shared storage, and fix it. The discipline is to apply the methodology to a cluster failure.

What “cannot failover” actually looks like

Failover has a prerequisite. Before Pacemaker starts a resource somewhere else, it must know the old owner has stopped. With shared storage, “probably stopped” is not good enough, so the cluster proves it: it fences the node, and only then recovers the resources.

That gives the real failure mode a precise shape. Fencing is armed, stonith-enabled is true, and the fence device cannot do its job — the BMC is unreachable, the credentials are stale, the device cannot map the target node. Pacemaker requests the fence, the fence fails, and the cluster deliberately refuses to start the resource anywhere. It would rather run nothing than run two copies.

That refusal is the symptom you are reproducing here.

Before you start

This drill deliberately leaves fencing armed. While the fence device is broken it cannot power-cycle anything, which is what keeps the exercise survivable — but the moment you repair the device the cluster fires the fence it has been retrying. Order matters, and it is covered in Task 3.

Every one of these preconditions is required:

• A disposable three-node lab cluster. Nodes you can rebuild, carrying no real workload, on a network you own. Never a shared cluster, never production.
• Never a two-node cluster. Two nodes with two_node: 1 both claim quorum when the ring breaks, so both fire the other node’s fence device. The node-disappears drill covers that fence race in full; the short version is that three nodes remove the problem instead of managing it.
• Fencing left armed. Do not disarm it. The broken fence path is the fault under test, and disarming STONITH would inject a different and far more dangerous failure.
• A fence device you can safely misconfigure, with its current settings recorded first, and an address on the BMC network that you know carries no host.
• Out-of-band console access to the target node, independent of the cluster network, so a wider-than-intended firewall rule does not lock you out.

Tasks

Task 1: Inject the failure

Blast radius: the disposable three-node lab cluster, and within it only the node you nominate as the target — node2 below. The injection is one CIB change and one firewall rule. Task 3 and Task 4 undo both.

First, break the fence path. Record what the device looks like now, then point it at an address where no BMC answers:

# On a majority node: save the current device configuration
pcs stonith config node2_ipmi > ~/node2_ipmi.before

# Break the path: an address on the BMC network with nothing on it
sudo pcs stonith update node2_ipmi ipaddr=10.0.0.254

The cluster accepts this without complaint. Agent validation checks that the options are well formed, not that a BMC answers at the address. That is the whole lesson in one command: a typo in a fence address is invisible until the day the cluster needs it. Newer fence agents name the parameter ip and keep ipaddr as a deprecated alias, so match whichever name your existing device already uses.

Now make the survivors believe node2 is gone, so they try to fence it. Drop inbound Corosync on the target node only:

# On node2 only
sudo iptables -A INPUT -p udp --dport 5405 -j DROP

One rule, inbound only. Knet holds a link up only while a ping and its pong both arrive, so blocking what comes in collapses the peers’ view of the link as well. A second rule on OUTPUT adds no isolation and doubles the rollback.

Check the port your ring actually uses before assuming 5405. It is mcastport in corosync.conf, and a second link uses a second port:

sudo corosync-cmapctl | grep -E 'mcastport|linknumber'

Within seconds the survivors declare node2 lost, schedule a fence, and fail to deliver it. The resources that were running on node2 do not move. They stay Stopped, and they stay that way as long as the fence keeps failing. node2 itself, now inquorate on its own, stops its own resources as well, so the service is down everywhere.

Task 2: Diagnose

Start from the resource, not from the node. The complaint is that something did not move; the cluster will tell you why it would not move it.

# Resources that should have failed over are Stopped
pcs status
crm_mon -1

# Which fence devices exist, where they run, and whether they are healthy
pcs stonith status
pcs stonith config

# What the cluster has already attempted
sudo pcs stonith history show

# The decision and the failure, in order
sudo journalctl -u pacemaker --since '15 min ago'
sudo journalctl -t pacemaker-fenced --since '15 min ago'

Use pcs stonith status and pcs stonith config, not the old pcs stonith show, which pcs 0.11 removed. The same applies to pcs resource show and pcs constraint show, replaced by pcs resource config and pcs constraint config.

The journal reads as a sequence, and the sequence is the diagnosis. The exact wording varies between releases, but the shape does not:

pacemaker-schedulerd  warning: Cluster node node2 will be fenced: peer is no longer part of the cluster
pacemaker-controld    notice:  Requesting fencing (reboot) of node node2
pacemaker-fenced      notice:  Requesting that node1 perform 'reboot' action targeting node2
pacemaker-fenced      error:   Operation 'reboot' targeting node2 by node1 failed: No route to host
pacemaker-controld    notice:  Peer node2 was not terminated (reboot) by node1: Error

crm_mon -1 shows the same thing as a Failed Fencing Actions block, and pcs stonith status shows node2_ipmi Stopped or FAILED, because its recurring monitor cannot reach the BMC either.

Read the three causes apart:

• Fencing. The resource is Stopped on every node, the fence history records failed actions, and the fence device itself is unhealthy. The cluster is not stuck; it is refusing. This is the case you injected.
• Constraints. No fencing was requested at all. The peers are Online, the resource is simply not allowed where you expect it — a location rule scoring -INFINITY, or a colocation tying it to something that is not running. Check with pcs constraint config --full and crm_simulate -sL, which prints the score the cluster gave each node.
• Shared storage. The resource does start on the target node and then fails there within seconds. The Filesystem or LVM agent logs a mount or activation error. Nodes and fencing are fine; the storage path is not. Check multipath, the iSCSI session, and whether the volume group is visible on that node.

Task 3: Fix

Depending on the cause:

• Fencing: repair the device — address, credentials, host mapping — and prove its monitor passes.
• Constraints: relax or correct the constraint that is scoring the target node out.
• Shared storage: restore access on the target node and re-test the resource there.

Step one, restore the ring:

# On node2: remove the one rule you added
sudo iptables -D INPUT -p udp --dport 5405 -j DROP

# Prove nothing is left behind
sudo iptables -S INPUT | grep 5405 || echo "no residual Corosync rules"

Step two, confirm the node is genuinely back before going near the fence configuration. Poll rather than sleeping; rejoin takes seconds, decided by the token timeout:

# On a majority node
pcs status
pcs quorum status
sudo corosync-cfgtool -s

Step three, and only once node2 reads Online, repair the fence device:

# Restore the real BMC address for node2
sudo pcs stonith update node2_ipmi ipaddr=10.0.0.101

# Compare against what you saved in Task 1
pcs stonith config node2_ipmi
diff <(pcs stonith config node2_ipmi) ~/node2_ipmi.before

Now prove the repair without resetting anything. pcs stonith fence is a real reboot and has no dry run, so it belongs in a scheduled fence test, not in a fix. The non-destructive evidence is the agent’s own status action and a passing monitor:

# Read-only: asks the BMC for power state, changes nothing
sudo fence_ipmilan --ip=10.0.0.101 --username=admin \
    --password-script=/etc/cluster/bmc-pw.sh --lanplus --action=status

# The device resource Started means its recurring monitor is passing
pcs stonith status

Use --password-script rather than --password. A password typed on the command line lands in your shell history and in the process table.

Task 4: Cleanup and verification

Return the cluster to its pre-drill state and prove it, rather than assuming it.

# 1. Clear the failed actions the drill created
sudo pcs resource cleanup
sudo pcs stonith history cleanup

# 2. Confirm fencing was never disarmed
pcs property config stonith-enabled

# 3. Confirm every fence device is Started, not just configured
pcs stonith status

# 4. Confirm all three nodes are Online and quorate
pcs status
pcs quorum status

# 5. Confirm the resources are running where they belong
pcs resource status

# 6. Remove the working copy of the configuration
rm -f ~/node2_ipmi.before

The lab is clean when all three nodes are Online, stonith-enabled is true, every fence device is Started, pcs stonith history show is empty, and pcs status lists no failed actions. A configured fence device whose monitor is failing is worse than no device at all, because the cluster trusts it.

Task 5: Document

TROUBLESHOOTING REPORT: Cluster Cannot Failover
Date: 2026-08-09
Symptom: node2 left the cluster; its resources stayed Stopped
         and did not start on node1 or node3
Root cause: the IPMI fence device for node2 pointed at an
         unreachable BMC address. With stonith-enabled=true,
         Pacemaker correctly refused to recover resources onto
         a peer it could not prove was dead
Fix: Restored the Corosync path, corrected node2_ipmi ipaddr,
     confirmed the device monitor passes, cleared failed actions
Prevention: Monitor fence device resource state, not only node state
         Alert if any stonith resource is Stopped
         Alert if stonith-enabled is ever set to false
         Quarterly fence test on every node

Knowledge check