STONITH and data integrity - the shoot-the-other-node-in-the-head pattern

STONITH (Shoot The Other Node In The Head) is the Pacemaker name for fencing. This lesson covers what it guarantees and how to configure it.

What STONITH guarantees

STONITH is the discipline of forcibly stopping a node before its resources are reassigned. It guarantees:

• The node is no longer accessing shared resources.
• The node’s disk is no longer being written to.
• The node is not running the workload.

After STONITH, the storage layer can safely give the resource to another node. Without STONITH, the old node might still be writing while the new node also writes - data corruption.

Configure STONITH in Pacemaker

In Pacemaker, STONITH is configured per device. First put the BMC password somewhere the CIB cannot reach. passwd= stores it in the cluster configuration in clear text, so pcs stonith config, every CIB backup, and every support bundle carries it. password_script= keeps it in a root-only file:

# On every node - any node may execute the fence
sudo install -d -m 0700 /etc/pacemaker
sudo install -m 0600 /dev/null /etc/pacemaker/bmc-pw
read -rs -p 'BMC password: ' BMC_PW; echo
printf '%s' "$BMC_PW" | sudo tee /etc/pacemaker/bmc-pw >/dev/null
unset BMC_PW

sudo tee /etc/pacemaker/bmc-pw.sh >/dev/null <<'EOF'
#!/bin/sh
printf '%s' "$(cat /etc/pacemaker/bmc-pw)"
EOF
sudo chmod 0700 /etc/pacemaker/bmc-pw.sh

# IPMI STONITH
pcs stonith create node1_ipmi fence_ipmilan \
    pcmk_host_list="node1" \
    ip="10.0.0.100" \
    username="admin" \
    password_script="/etc/pacemaker/bmc-pw.sh" \
    lanplus=1 \
    pcmk_off_action="off"

# iLO STONITH
pcs stonith create node1_ilo fence_ilo \
    pcmk_host_list="node1" \
    ip="10.0.0.101" \
    username="admin" \
    password_script="/etc/pacemaker/bmc-pw.sh"

# Verify - this must print nothing
pcs stonith config | grep 'passwd='

For each node, configure a STONITH device. Test it:

# Test STONITH on a node (it WILL be powered off)
pcs stonith fence node1

Verify the node is actually powered off. If the test does not work, the STONITH configuration is wrong.

Stonith-enabled property

Pacemaker has a property stonith-enabled that controls whether STONITH is required:

pcs property set stonith-enabled=true

When stonith-enabled=true, the cluster refuses to start resources on a node if it cannot fence that node. This is the production default; do not disable it.

STONITH levels

Pacemaker has different STONITH levels (which devices to try first, second, etc.):

# Primary STONITH (IPMI)
pcs stonith create node1_ipmi fence_ipmilan ...

# Backup STONITH (PDU)
pcs stonith create node1_pdu fence_apc ...

# Levels - the TARGET NODE comes first, then the device id
pcs stonith level add 1 node1 node1_ipmi
pcs stonith level add 2 node1 node1_pdu

# Verify what was registered
pcs stonith level config

The documented signature is:

pcs stonith level add <level> <target> <stonith id> [<stonith id>]...

The target is the node to be fenced; the ids after it are the devices to use at that level. Writing pcs stonith level add 1 node1_ipmi node1 reverses the two: pcs either rejects it or registers a topology for a target called node1_ipmi that does not exist. Either way no escalation is configured for node1, and nothing tells you - the create commands above still succeeded, so pcs stonith config looks healthy.

That failure surfaces at the worst moment. The reason a PDU backup exists is that the BMC is unreachable; if the topology was never registered, the cluster has no second method, fencing fails permanently, and the resources stay stopped. The usual next move - someone disabling stonith-enabled to unstick the cluster - is how the outage becomes corruption.

Always read the topology back, then break level 1 deliberately and confirm level 2 fires:

pcs stonith level config
# Expect, per node:
#   Target: node1
#     Level 1 - node1_ipmi
#     Level 2 - node1_pdu

# Break level 1 on purpose (block the BMC), then fence
pcs stonith fence node1
# Watch the escalation
journalctl -u pacemaker -f | grep -Ei 'fence|stonith'

Repeat the two level add commands for every node. Pacemaker tries level 1 first; if that fails, level 2.

Production discipline

For every cluster:

1. Configure STONITH for every node.
2. Test STONITH on every node.
3. Set stonith-enabled=true.
4. Test STONITH under load.
5. Schedule quarterly STONITH tests.

A cluster without STONITH is a cluster waiting to corrupt data.

Knowledge check