Pacemaker troubleshooting - the systematic approach

Pacemaker troubleshooting is the discipline of finding the root cause of a cluster issue. This lesson covers the systematic approach and the common failure modes.

The loop

1. Symptom: what is the user seeing?
2. Verify: is the symptom real?
3. Logs: what do the cluster logs say?
4. Hypothesis: what could cause this?
5. Test: is the hypothesis correct?
6. Fix: apply the fix.
7. Verify: is the symptom resolved?
8. Document: capture for next time.

Each step is a check. Do not skip steps.

Common failure modes

Symptom: resources are not starting

sudo pcs status
sudo journalctl -u pacemaker -n 50

Common causes:

• STONITH device failure.
• Constraint violation.
• Resource agent error.
• Quorum loss.

Symptom: nodes are offline

sudo pcs status
sudo corosync-cmapctl | grep members

Common causes:

• Network partition.
• Corosync not running.
• Firewall blocking a cluster port. 5405/UDP is the one everyone remembers, and it is not the whole list — see below.

The cluster ports, all of them

Almost every “cluster firewall” guide names 5405/UDP and stops. That is enough to keep membership alive and not enough to operate the cluster, which produces a maddening class of half-working failure: pcs status is fine on every node, but pcs cluster start node2 from node1 times out.

Restrict every one of these to the cluster’s own subnet. They are unauthenticated or weakly authenticated by design and must never face a general network:

# nftables, cluster ring subnet only
sudo nft add rule inet filter input ip saddr 10.0.0.0/24 udp dport 5404-5412 accept
sudo nft add rule inet filter input ip saddr 10.0.0.0/24 tcp dport 2224 accept
sudo nft add rule inet filter input ip saddr 10.0.0.0/24 tcp dport 3121 accept
sudo nft add rule inet filter input ip saddr 10.0.0.0/24 tcp dport 21064 accept
sudo nft add rule inet filter input ip saddr 10.0.0.0/24 ip protocol 112 accept

# firewalld has a service bundle for the common set
sudo firewall-cmd --permanent --add-service=high-availability
sudo firewall-cmd --reload

Confirm what is actually bound before you trust the ruleset, and confirm reachability from a peer, never from the node itself:

sudo ss -lntup | grep -E 'corosync|pcsd|pacemaker'
corosync-cfgtool -s          # per-link status; 'connected' on every link
pcs cluster status           # run this from another node

Check the port your rings really use rather than assuming 5405 — corosync.conf can set any base port, and a second link runs on the next one up:

sudo corosync-cmapctl | grep -E 'totem.interface.*mcastport|nodelist'

Symptom: resources are flapping

sudo journalctl -u pacemaker | grep -E 'monitor|failed'

Common causes:

• Resource agent monitor failure (false positive).
• STONITH device timeout.
• Network blip causing monitor failure.

Symptom: STONITH not working

sudo pcs stonith status                          # is the device resource Started?
sudo pcs stonith config                          # what is it configured to talk to?
sudo pcs stonith history show                    # what has it already attempted, and why did it fail?
sudo stonith_admin --list-registered             # which agents this node can actually use
sudo fence_ipmilan --ip=<bmc-ip> --username=<user> \
     --password-script=/etc/cluster/bmc-pw.sh --lanplus --action=status

Every command above is read-only. The last one asks the BMC for the node’s power state through the same agent the cluster uses, so it proves the credential, the network path and the host mapping without touching the power button. A fence device whose resource is Started has a passing recurring monitor; that is the non-destructive evidence you want.

pcs stonith show was removed in pcs 0.11, along with pcs resource show and pcs constraint show. The replacements are pcs stonith status for state, pcs stonith config for configuration.

Common causes:

• Wrong credentials.
• Network issue to BMC.
• Wrong host list.

Tools

# Cluster status
sudo pcs status

# Detailed
sudo pcs status --full

# Corosync
sudo corosync-cmapctl
sudo corosync-quorumtool

# Logs
sudo journalctl -u pacemaker
sudo journalctl -u corosync

# STONITH health - read-only. `pcs stonith fence` is a real power-off
# and is deliberately not in this list; see the callout above.
sudo pcs stonith status
sudo pcs stonith history show

# Resource history - assembled, not a single command
sudo pcs resource failcount show
sudo crm_mon --one-shot --failcounts
sudo journalctl -u pacemaker | grep -E 'Transition|Recover|Move|fence'

Common fixes

Resources not starting due to STONITH

# Check STONITH status - all read-only
sudo pcs stonith status
sudo pcs stonith config
sudo pcs stonith history show

# Fix STONITH configuration
sudo pcs stonith update node1_ipmi ...

# Confirm the repair without resetting anything
sudo pcs stonith status                # device Started == monitor passing

Restore the network path before repairing the fence device. A pending fence request does not expire while the node still looks lost, so a device repaired first delivers the fence for real the moment it starts working.

Quorum loss

sudo pcs quorum status
sudo corosync-quorumtool -s

Identify the partitioned node and fix the network.

Resource agent error

sudo pcs resource debug-start web
sudo journalctl -u pacemaker -n 50

The debug-start runs the resource agent with debug output.

Prevention

• Test every change in staging.
• Document the runbook for common issues.
• Schedule quarterly cluster tests.
• Monitor cluster state (Pacemaker can be scraped via pcs status —full).

Knowledge check