Pacemaker fencing integration - tying STONITH to resources

STONITH is configured per device but the cluster decides when to use it. This lesson covers the relationship between fencing and resources.

The relationship

When a node is suspected to be failed:

1. Pacemaker fences the node (STONITH).
2. After fencing confirms the node is down, Pacemaker considers the node unavailable.
3. Resources previously on the failed node are migrated to surviving nodes (per constraints).

Fencing is the prerequisite for resource migration. Without it, Pacemaker does not know if the failed node is still accessing shared resources.

Configure STONITH

# Configure STONITH for each node. password_script= reads the
# BMC password from a root-only file at fence time. passwd=
# would store it in the CIB in clear text, where pcs stonith
# config and every CIB backup would carry it.
pcs stonith create node1_ipmi fence_ipmilan \
    pcmk_host_list="node1" \
    ip="10.0.0.100" \
    username="admin" \
    password_script="/etc/pacemaker/bmc-pw.sh" \
    lanplus=1

# Verify
pcs stonith status

/etc/pacemaker/bmc-pw.sh is the credential reader created in the STONITH lesson. It must exist on every node, because any node may be the one that executes the fence.

Each node must have a STONITH device, otherwise the cluster refuses to start resources on it (when stonith-enabled=true).

Fencing topology

“Topology” in Pacemaker means one specific thing: the ordered list of fencing devices to try for a given node. It is not a way to describe racks, and it has nothing to do with where resources run. There is no pcs topology subcommand - the command is pcs stonith level.

Use it when a node has more than one way to be killed. Level 1 is tried first; if it fails, level 2 is tried:

# node1: try the BMC first, fall back to the PDU
pcs stonith level add 1 node1 node1_ipmi
pcs stonith level add 2 node1 node1_pdu

pcs stonith level config

This matters because a BMC often shares power with the host it controls. If the PSU dies, the IPMI fence times out and the node is never confirmed dead, so resources stay stranded. A switched PDU on a separate circuit is the fallback that finishes the job.

Levels can also require two devices to succeed together, which is how you fence a node with redundant power supplies:

# BOTH outlets must be cut for the fence to count as successful
pcs stonith level add 1 node2 node2_pdu_a,node2_pdu_b

Rack awareness

Failure-domain placement is a separate mechanism: node attributes plus a rule-based location constraint. Tag each node with the rack it lives in, then write rules against the attribute:

# Tag the nodes
pcs node attribute node1 rack=rack1
pcs node attribute node2 rack=rack1
pcs node attribute node3 rack=rack2
pcs node attribute node4 rack=rack2
pcs node attribute            # verify

# Prefer rack1, but do not forbid rack2
pcs constraint location web rule score=100 rack eq rack1

A location constraint targets nodes, not labels. Writing pcs constraint location web prefers rack1 fails or, worse, creates a constraint against a non-existent node. And issuing two prefers lines back to back does not express “fallback”: both carry INFINITY, they conflict, and placement becomes unstable rather than ordered. Preference is a score, so express it as a score.

To keep the replicas of a clone in different racks, colocate the clone against itself with a negative score on the attribute:

pcs constraint colocation add web-clone with web-clone \
    -INFINITY node-attribute=rack

Test fencing and resources together

For every cluster, test:

1. STONITH for every node (verify the node actually stops).
2. Resource failover when a node is fenced.
3. Service recovery when the node returns.

# Test fencing on a node
pcs stonith fence node1
# Verify node1 is down
# Verify resources moved to node2
pcs status

If the test does not work, fix the STONITH configuration before relying on the cluster.

Quorum and fencing

Without quorum, Pacemaker cannot make decisions. Without fencing, Pacemaker does not know if the failed node is still running. Together:

• Quorum decides who can act.
• Fencing ensures the failed group stops acting.
• Resource migration follows.

The discipline is the combination.

Knowledge check