Lab: Fencing scenario - simulate, verify, and document

This lab configures a STONITH device, triggers a fencing event, and verifies the cluster handles it correctly.

Objective

By the end of this lab, you can:

• Configure an IPMI STONITH device with the credential outside the CIB and outside argv.
• Prove the credential path works before an incident depends on it.
• Trigger a fence and confirm from the cluster’s own records that it completed, not merely that a host stopped answering ping.
• Confirm resources recovered onto surviving nodes and bring the fenced node back.

Architecture

A three-node test cluster. Each node has a BMC on a management network; fence_ipmilan talks to the BMC of the node it is configured to fence, which is why a wrong ip= powers off the wrong host.

  node1 ──┐                    ┌── BMC node1 (10.0.0.100)
  node2 ──┼── corosync ring    ├── BMC node2 (10.0.0.101)
  node3 ──┘                    └── BMC node3 (10.0.0.102)
                │
                └── Pacemaker: stonith device node1_ipmi
                    targets pcmk_host_list=node1 via 10.0.0.100

The credential lives in a root-only file that a small script reads and prints; Pacemaker calls the script through password_script=. That keeps the password out of the CIB - which is world-readable to anyone with cluster access and replicated to every node - and out of the process table.

Requirements

• A disposable three-node cluster. Task 4 powers a node off for real. Nested VMs or simulation mode; never production, and never a BMC that also fronts a production host.
• Quorum with a node down. Two nodes plus a qdevice works; a plain 2-node cluster does not demonstrate recovery honestly.
• pcs, fence-agents-ipmilan and ipmitool on every node.
• Working BMC credentials for the target node, and console access to that node in case it does not come back.
• Somewhere to record the test that is not your shell history.

sudo pcs status
sudo pcs quorum status
sudo pcs stonith list | grep fence_ipmilan
command -v ipmitool

Scenario

The cluster has had STONITH configured since it was built, and it has never fired. Last month a node hung - not down, hung, with its filesystem still mounted - and the cluster did not recover; the on-call engineer eventually power-cycled it by hand.

The postmortem question is the one nobody could answer: does the fence device work? A STONITH device that has never been tested is a configuration file, not a safety mechanism. Your job is to make it fire on purpose, in daylight, on a cluster you can afford to lose.

Tasks

Task 1: Store the BMC credential

The fence agent needs the BMC password. It must not go on a command line and it must not go into the CIB.

passwd= writes the password into the cluster configuration in clear text, where pcs stonith config, every CIB backup, and every support bundle will carry it. password_script= keeps it in a root-only file instead, and the agent reads it from the script’s standard output.

Run this on every node - any node may be the one that executes the fence:

sudo install -d -m 0700 /etc/pacemaker
sudo install -m 0600 /dev/null /etc/pacemaker/bmc-pw

# read -rs does not echo, and the value never enters argv
# or shell history.
read -rs -p 'BMC password: ' BMC_PW; echo
printf '%s' "$BMC_PW" | sudo tee /etc/pacemaker/bmc-pw >/dev/null
unset BMC_PW

sudo tee /etc/pacemaker/bmc-pw.sh >/dev/null <<'EOF'
#!/bin/sh
# Command substitution strips the trailing newline, so a stray
# newline in the file cannot corrupt the password.
printf '%s' "$(cat /etc/pacemaker/bmc-pw)"
EOF
sudo chmod 0700 /etc/pacemaker/bmc-pw.sh

Task 2: Configure a STONITH device

# Configure IPMI STONITH
sudo pcs stonith create node1_ipmi fence_ipmilan \
    pcmk_host_list="node1" \
    ip="10.0.0.100" \
    username="admin" \
    password_script="/etc/pacemaker/bmc-pw.sh" \
    lanplus=1

ip= and username= are the current parameter names; ipaddr= and login= are deprecated aliases.

Task 3: Verify the STONITH device

sudo pcs stonith status
sudo pcs stonith config node1_ipmi

Read the output and confirm two things. ip= must be the BMC of node1 and nothing else - a wrong address powers off a healthy node during a real event. And the configuration must show password_script=, with no passwd= anywhere:

sudo pcs stonith config node1_ipmi | grep -c 'passwd='
# Must print 0

Now prove the credential path works before you rely on it, reading the password from the file rather than the command line:

# sudo, because the credential file is root-only
sudo ipmitool -I lanplus -H 10.0.0.100 -U admin \
    -f /etc/pacemaker/bmc-pw chassis status | grep -i 'system power'

Task 4: Test the STONITH device

# Manual fencing test - node1 is powered off for real
sudo pcs stonith fence node1

The node should be powered off. Verify:

# From a working node
ping -c 3 node1
# Should fail after a few seconds

Verify the node is actually down (not just unreachable on the network).

Task 5: Verify the cluster continues

# Check cluster status
sudo pcs status

# Check resources
sudo pcs resource status

Resources should be migrated to surviving nodes.

Task 6: Bring the node back

# Power on the node - password read from the root-only file,
# not from the command line
sudo ipmitool -I lanplus -H 10.0.0.100 -U admin \
    -f /etc/pacemaker/bmc-pw chassis power on

# Wait for the node to rejoin
sleep 60
sudo pcs status

The node rejoins; resources may be re-balanced.

Task 7: Document the test

STONITH TEST REPORT
Date: 2026-08-09
Cluster: 3-node test cluster
Device: IPMI (10.0.0.100)

Steps:
1. Configured fence_ipmilan
2. Triggered STONITH on node1
3. Verified node1 is powered off (IPMI confirmed)
4. Verified cluster continues (node2 and node3)
5. Resources migrated to node2/node3
6. Powered on node1
7. Verified node1 rejoins cluster

Findings:
- STONITH worked as expected
- Node was actually powered off
- Resources migrated successfully
- Recovery was automatic

Improvements:
- Test with different node to verify all STONITH devices
- Test under load to verify STONITH at scale

Do not paste the BMC password into the report. Record the account name and the address only.

Validation

A fence is only proven by the cluster’s own record of it. Every one of these must hold.

# 1. The credential never entered the CIB
sudo pcs stonith config node1_ipmi | grep -c 'passwd='
# Must print 0

# 2. The cluster recorded a SUCCESSFUL fence, with a timestamp
sudo pcs stonith history show node1
# Must show a completed 'off' or 'reboot' action, not 'pending'

# 3. The power state agrees, read from the BMC and not from ping
sudo ipmitool -I lanplus -H 10.0.0.100 -U admin \
    -f /etc/pacemaker/bmc-pw chassis power status
# Must report 'Chassis Power is off'

# 4. Resources actually moved, and nothing is stuck
sudo pcs status --full | sed -n '/Failed Resource Actions/,/^$/p'
# Must be empty

# 5. The surviving partition is quorate
sudo pcs quorum status

# 6. The service is still serving
curl -sf http://vip/health

Expected outcome

Recovery should be automatic. If you had to start a resource by hand, the fence or the constraints are wrong - record that as the finding, because it is the one that would have mattered.

Troubleshooting

Cleanup

Return the test cluster to its starting state:

# 1. Every node must be up and in the cluster
sudo pcs status | grep -E 'Online|OFFLINE'

# 2. Clear the fencing history and any failed actions
sudo pcs stonith history cleanup node1
sudo pcs resource cleanup

# 3. Remove the lab fence device
sudo pcs stonith delete node1_ipmi

# 4. Remove the credential file and its reader, on every node
sudo rm -f /etc/pacemaker/bmc-pw /etc/pacemaker/bmc-pw.sh

Keep the device and the credential file if this cluster continues into the next lab. Remove them if the cluster is being torn down: an orphaned mode 0600 password file on a recycled host is a credential leak.

Finally, confirm nothing leaked into your history:

grep -n -- '-P ' ~/.bash_history | grep ipmitool
# Must print nothing

If it does print something, the password is compromised. Rotate it on the BMC, then clear the entry.

What you learned

• A STONITH device that has never fired is a configuration file, not a safety mechanism. The only way to know it works is to make it work on a cluster you can lose.
• The BMC credential belongs in a root-only file behind password_script=, not in passwd=. The CIB is replicated to every node and readable by anyone with cluster access.
• A fence is verified by two independent sources: the BMC’s power state and pcs stonith history. An unreachable node is not a fenced node, and acting as though it were is how shared storage gets double-mounted.
• The dangerous configuration error is a wrong ip=: it does not fail, it fences a healthy host. Check every device against a written BMC inventory before you test.