Pacemaker HA cluster lab - build a 3-node cluster

This lab builds a 3-node Pacemaker cluster from scratch and tests failover. By the end you will have a working cluster.

The addressing plan

Write the addresses down before you configure anything. Two separate networks, and every address in the plan is unique:

The cluster VIP is 10.0.0.100. It lives on the service network and belongs to no node permanently.

Two rules follow from the table, and the rest of the lab depends on both:

• Each node has its own BMC address. A BMC is a physical chip on one motherboard. It can only power one machine.
• The VIP is on the service network, never the management network. Fencing must keep working while the VIP is up.

Tasks

Task 1: Install Pacemaker

On each of 3 nodes:

This lab is written for Ubuntu/Debian throughout. Pick one distribution and stay on it: the package names, the PostgreSQL data directory and the Apache config path all differ between the Debian and RHEL families, and a lab that mixes them produces resources that fail to start with a path error rather than anything to do with clustering.

sudo apt install pacemaker pcs fence-agents resource-agents
sudo systemctl enable --now pcsd

pacemaker and pcs alone are not enough. fence-agents provides fence_ipmilan, which Task 3 needs, and resource-agents provides the ocf:heartbeat:* agents Task 4 needs. Without them those tasks fail with “agent not found”, which reads like a cluster problem and is not one.

Authentication is next, and it needs a credential that does not exist yet. pcs talks to pcsd on each node as the local hacluster user, which the package creates with a locked password. Set the same password on all three nodes, then authenticate once:

# On every node:
sudo passwd hacluster

# Once, from any node:
sudo pcs host auth node1 node2 node3 -u hacluster

pcs host auth with no -u prompts for a username, and a bare pcs host auth node1 node2 node3 in a script simply hangs. If authentication fails, check /etc/hosts or DNS first: every node must resolve the other two by the exact names used here.

Task 2: Configure the cluster

sudo pcs cluster setup mycluster node1 node2 node3 \
    --start --enable

Task 3: Configure STONITH

First store the BMC password out of the CIB. passwd= writes the password into the cluster configuration in clear text, where pcs stonith config, every CIB backup and every support bundle will carry it. password_script= keeps it in a root-only file instead.

Run this on every node - any node may be the one that executes the fence:

sudo install -d -m 0700 /etc/pacemaker

# Read the password without echoing it, and without putting it
# in shell history or in the process list.
read -rs -p 'BMC password: ' BMC_PW; echo
printf '%s' "$BMC_PW" | sudo tee /etc/pacemaker/bmc-pw >/dev/null
unset BMC_PW
sudo chmod 0600 /etc/pacemaker/bmc-pw

sudo tee /etc/pacemaker/bmc-pw.sh >/dev/null <<'EOF'
#!/bin/sh
# The agent uses stdout verbatim. Command substitution strips the
# trailing newline, so an editor cannot break the password.
printf '%s' "$(cat /etc/pacemaker/bmc-pw)"
EOF
sudo chmod 0700 /etc/pacemaker/bmc-pw.sh

Now create one fence device per node. Each one points at that node’s own BMC address from the addressing plan:

sudo pcs stonith create node1_ipmi fence_ipmilan \
    pcmk_host_list="node1" \
    ip="10.0.99.11" \
    username="stonith" \
    password_script="/etc/pacemaker/bmc-pw.sh" \
    lanplus=1 \
    pcmk_delay_max=15s

sudo pcs stonith create node2_ipmi fence_ipmilan \
    pcmk_host_list="node2" \
    ip="10.0.99.12" \
    username="stonith" \
    password_script="/etc/pacemaker/bmc-pw.sh" \
    lanplus=1 \
    pcmk_delay_max=15s

sudo pcs stonith create node3_ipmi fence_ipmilan \
    pcmk_host_list="node3" \
    ip="10.0.99.13" \
    username="stonith" \
    password_script="/etc/pacemaker/bmc-pw.sh" \
    lanplus=1 \
    pcmk_delay_max=15s

Three commands, written out in full. Do not shorten this to one command and a “repeat for node2 and node3” comment: that is how all three devices end up on one address. ip= is the current parameter name for fence_ipmilan; ipaddr= is a deprecated alias for the same thing. pcmk_delay_max adds a random delay before fencing so two nodes cannot shoot each other simultaneously in a partition.

Stop each node from fencing itself. A node that is hung or already powered off cannot be trusted to run its own fence agent:

sudo pcs constraint location node1_ipmi avoids node1
sudo pcs constraint location node2_ipmi avoids node2
sudo pcs constraint location node3_ipmi avoids node3

Now verify the addresses before you trust the cluster:

for n in 1 2 3; do
    sudo pcs stonith config node${n}_ipmi | grep -E 'ip=|pcmk_host_list='
done

Read the output against three conditions. All three must hold:

• Three distinct ip= values, one per device.
• Each ip= matches the BMC row of the addressing plan for the node named in its pcmk_host_list.
• No ip= equals the VIP 10.0.0.100, or any other address on the service network.

If two devices share an address, delete and recreate them now. Do not continue: a wrong fence address is only discovered during an outage, when it powers off a healthy node.

Matching addresses in the CIB only proves the configuration is self-consistent. Prove the mapping physically as well. Ask each node for its own serial number, then ask each BMC which chassis it manages, and check the pairs:

# On node1, node2 and node3 in turn:
sudo dmidecode -s system-serial-number

# From an admin host, one query per BMC:
for ip in 10.0.99.11 10.0.99.12 10.0.99.13; do
    printf '%s ' "$ip"
    sudo ipmitool -I lanplus -H "$ip" -U stonith \
        -f /etc/pacemaker/bmc-pw fru print 0 | grep -i 'Product Serial'
done

Three different serials, each matching the node whose fence device carries that BMC address. Two BMCs reporting the same serial means two fence devices are aimed at one machine.

Task 4: Configure resources

The VIP is a service-network address. Confirm it is free before you hand it to the cluster - IPaddr2 will claim it and ARP for it regardless of who holds it today:

# Both checks must show the address is unused.
# Replace eth0 with the service-network interface on this node.
ping -c 2 -W 1 10.0.0.100                  # must time out
sudo arping -D -I eth0 -c 2 10.0.0.100     # must report no reply

If either answers, something already owns 10.0.0.100. Find it and pick a different VIP. Never assign a VIP that resolves to a BMC, a switch, or a router.

# Virtual IP - service network, distinct from every BMC address
sudo pcs resource create vip ocf:heartbeat:IPaddr2 \
    ip=10.0.0.100 \
    cidr_netmask=24 \
    op monitor interval=30s

# PostgreSQL. Debian/Ubuntu paths - pg_ctlcluster layout, version in the path.
# Confirm yours: pg_lsclusters
sudo pcs resource create db ocf:heartbeat:pgsql \
    pgctl=/usr/lib/postgresql/16/bin/pg_ctl \
    psql=/usr/bin/psql \
    pgdata=/var/lib/postgresql/16/main \
    op monitor interval=30s

# Apache. Debian/Ubuntu path; on RHEL this is /etc/httpd/conf/httpd.conf.
sudo pcs resource create web ocf:heartbeat:apache \
    configfile=/etc/apache2/apache2.conf \
    op monitor interval=30s

Run pg_lsclusters before you paste that pgdata= — the major version in the path is whatever your release ships, and a wrong path gives you a resource stuck in Stopped with the reason buried in pcs status --full.

Both services must also be disabled in systemd on every node:

sudo systemctl disable --now postgresql apache2

The cluster starts and stops these services. If systemd also starts them at boot, two copies run, both bind the same port, and the cluster’s monitor operation reports a healthy resource it does not actually control.

Task 5: Configure constraints

The naive set of constraints for “these three run together” is a trap, and it is worth seeing the trap before the fix.

Colocating web with db and web with vip looks complete — web is pinned to both. But nothing pins db to vip. The cluster is free to place db on node1 and vip on node2, and at that point web has two INFINITY colocations pointing at different nodes and no node satisfies both. Web does not start, and pcs status reports it as unable to run anywhere with no obvious cause. There is no order constraint for vip either, so even in the placements that do work Apache can start before the address it is supposed to bind exists.

Use a resource group. A group enforces colocation and ordering in one object, in the order listed, and cannot express the unsatisfiable set:

sudo pcs resource group add appstack vip db web

That reads: all three on the same node, started vip → db → web, stopped in reverse.

If you need the resources to stay independent — different monitor intervals, or one of them cloned — write the constraints explicitly, and make sure every pair is constrained:

sudo pcs constraint colocation add db with vip INFINITY
sudo pcs constraint colocation add web with db INFINITY
sudo pcs constraint order vip then db
sudo pcs constraint order db then web

Either way, verify rather than assume:

sudo pcs constraint config --full
sudo pcs status resources     # all three on ONE node

Task 6: Verify

sudo pcs status

# Where is the VIP, and is it really on that node?
sudo pcs status resources
sudo pcs stonith config | grep -E 'ip=|pcmk_host_list='

Verify:

• All 3 nodes online.
• All 3 resources started.
• Constraints applied.
• The VIP 10.0.0.100 answers, and each BMC on 10.0.99.0/24 still answers ipmitool ... chassis status. If bringing the VIP up broke BMC access, the two networks are not actually separate - stop and fix the addressing before you go near Task 8.

Task 7: Test failover

# Stop Pacemaker on node1
sudo pcs cluster stop node1

# Verify resources moved
sudo pcs status

Resources should be on node2 and node3.

Task 8: Test STONITH

This powers off node2. Run it only on the disposable lab nodes.

# Fence one node, by name
sudo pcs stonith fence node2

# The named node must be down...
ping -c 3 node2    # should fail

# ...and the other two must still be up
ping -c 3 node1    # should succeed
ping -c 3 node3    # should succeed

Check the last two pings, not just the first. A fence device pointed at the wrong BMC still reports success - it powered off a machine, just not the one you named. The only proof the mapping is right is that node2 went down and nothing else did.

Once Task 9 has brought the cluster back to three healthy nodes, repeat this test for node1 and node3 so every fence device is exercised at least once. A device that is never tested is a device that fails during the first real outage.

Task 9: Test recovery

# Power node2 back on through its own BMC
sudo ipmitool -I lanplus -H 10.0.99.12 -U stonith \
    -f /etc/pacemaker/bmc-pw chassis power on

# Bring node1 back into the cluster
sudo pcs cluster start node1

# Wait for it to rejoin
sleep 60
sudo pcs status

The node should rejoin and resources may be re-balanced.

Task 10: Document

CLUSTER SETUP REPORT
Date: 2026-08-09
Cluster: mycluster
Nodes: node1, node2, node3

Components:
- Corosync: membership and quorum
- Pacemaker: resource management
- STONITH: IPMI per node

Networks:
- Service:    10.0.0.0/24  (hosts .11 .12 .13, VIP .100)
- Management: 10.0.99.0/24 (BMCs .11 .12 .13)

Fence devices (verified distinct, none on the service net):
- node1_ipmi -> pcmk_host_list=node1, ip=10.0.99.11
- node2_ipmi -> pcmk_host_list=node2, ip=10.0.99.12
- node3_ipmi -> pcmk_host_list=node3, ip=10.0.99.13

Resources:
- vip: 10.0.0.100/24 (service network)
- db: PostgreSQL
- web: Apache

Constraints:
- web with db: INFINITY (colocation)
- db then web: order
- web with vip: INFINITY (colocation)
- nodeN_ipmi avoids nodeN (no self-fencing)

Tests:
- Failover: PASS
- STONITH node1: PASS (only node1 lost power)
- STONITH node2: PASS (only node2 lost power)
- STONITH node3: PASS (only node3 lost power)
- BMC reachable while VIP up: PASS
- Recovery: PASS

Findings:
- Cluster handles single-node failure
- STONITH works on every node, and fences only that node
- Resources migrate correctly

Validation

Commands and the exact result each must produce. A lab report that says “the cluster looked healthy” is not a validation.

# 1. Every node is a full member. Three "Online" entries, no
#    node listed as OFFLINE, standby, maintenance or pending.
sudo pcs status --full

# 2. Every STONITH device is Started, and none is started on the
#    node it fences - a device co-located with its target cannot
#    fence it.
sudo pcs stonith status
sudo pcs constraint location config --full | grep -i ipmi

# 3. Zero failcounts. A resource that recovered still leaves a
#    record, and a stale failcount changes future placement.
sudo crm_mon -1 -f
sudo pcs resource failcount show          # must report no failcounts

# 4. Quorum is held by the whole membership, not a subset.
sudo pcs quorum status
# Expected votes: 3, Total votes: 3, Quorum: 2, Flags: Quorate

# 5. The VIP is on exactly one node.
for h in node1 node2 node3; do
    echo -n "$h: "; ssh "$h" ip -br a show dev eth0 | grep -c 10.0.0.100
done
# Exactly one host prints 1; the other two print 0.

# 6. The fence test from Task 8 is recorded in the log, and it
#    fenced only its target.
sudo pcs stonith history show
sudo journalctl -u pacemaker --since '2 hours ago' | grep -i 'fenc'

If any of the six fails, the cluster is not validated — fix it before you write the report. In particular, a cluster that has never been fence-tested (step 6 empty) has not been validated at all: fencing that has not been exercised is fencing that does not work, and you will discover that during an incident.

Cleanup

Step 1. Make sure no node is still powered off from Task 8. Fencing leaves the node down, and a down node will not receive the destroy:

for ip in 10.0.99.11 10.0.99.12 10.0.99.13; do
    sudo ipmitool -I lanplus -H "$ip" -U stonith \
        -f /etc/pacemaker/bmc-pw chassis power on
done

sudo pcs status    # all 3 nodes online before continuing

Step 2. Stop the resources so the VIP is released cleanly rather than disappearing with the node:

sudo pcs resource disable web db vip
sudo pcs status resources    # all three Stopped

Step 3. Confirm the VIP is gone from the segment. It should no longer answer:

ping -c 2 -W 1 10.0.0.100    # must time out

Step 4. Tear the cluster down:

sudo pcs cluster stop --all
sudo pcs cluster destroy --all

Step 5. Remove the BMC credentials from every node, then rotate the BMC password. A password that has sat in a lab file is no longer a secret:

sudo rm -f /etc/pacemaker/bmc-pw /etc/pacemaker/bmc-pw.sh

The BMCs themselves keep their addresses. They are hardware management interfaces, not cluster state, and nothing in this lab should have changed them.