critical riskcluster affecting~40 min

Runbook: Quorum loss

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Establish how many nodes the cluster expects and how many are visible from each partition
· Establish whether fencing is configured, armed and healthy - this decides what is safe to do next
· Confirm out-of-band console access to every node before touching membership
· Confirm whether resources are currently running anywhere, and whether shared storage is involved
· Record the current CIB and corosync configuration before any change

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Read quorum state from every reachable node, not just one
2Determine the partition topology: who can see whom
3Establish whether the cause is node failure, network partition, or a quorum-device failure
4Restore the underlying cause - network path, node, or qdevice - in preference to overriding quorum
5If nodes are genuinely down and cannot return, only then consider an explicit expected-votes change
6Verify quorum returns and resources are running in exactly one place
7Restore expected_votes to its configured value once the cluster is whole
8Record the timeline and the cause

4 · Verification

Confirm the procedure actually fixed the problem.

✓pcs quorum status shows Quorate: Yes with the expected total vote count
✓corosync-quorumtool -s agrees from every node
✓Every resource is running on exactly one node (pcs status)
✓No manual expected_votes override remains in effect
✓Fencing is armed and every fence device reports Started

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Restore the saved CIB if a configuration change made things worse: pcs cluster cib-push
↶Return expected_votes to the configured value with corosync-quorumtool -e <n>
↶Re-arm fencing if it was disabled during recovery - a cluster left with stonith-enabled=false is a broken cluster that looks healthy

6 · Escalation

When the runbook isn't enough, contact:

· Both partitions are running the same resource against shared storage: this is an active split brain. Escalate immediately and stop the cluster on one side before anything else.
· Quorum cannot be restored without forcing it and the data is shared: escalate to the data owner before forcing anything
· The quorum device or witness is unreachable and cannot be restored: escalate to the network or platform team
· Fencing is not functional: escalate; recovery without working fencing is not safe to complete alone

Quorum is the cluster’s rule for deciding which partition is allowed to act. Losing it is the cluster working correctly: it has stopped rather than risked two nodes writing to the same data. Every dangerous action in this runbook is one that overrides that decision.

Step 1: Read quorum from every reachable node

One node’s view is a partition’s view, not the truth.

Read-only / Safequorum status— Reads membership and vote state. No side effects.

pcs quorum status
sudo corosync-quorumtool -s
sudo corosync-cfgtool -s
pcs status --full

Read four numbers: expected votes, total votes, quorum, and Quorate: Yes/No. Then run the same commands on every other node you can reach and write the answers side by side. The shape of the disagreement is the diagnosis.

All reachable nodes report the same small member list, and the missing nodes are unreachable from everywhere → probably a genuine node failure.
Two groups each report themselves as members and each reports the other as missing → network partition.
Vote count is short by exactly one and no node is missing → the quorum device is down.

Step 2: Establish the topology

Read-only / Safering and membership— Reads corosync ring state, listeners and traffic. No changes.

# Per-link ring status: FAULTY tells you which link broke
sudo corosync-cfgtool -s

# Configured members and the ports actually in use
sudo corosync-cmapctl | grep -E 'members|mcastport|linknumber|nodelist'

# Is corosync bound and listening
sudo ss -lunp | grep corosync

# Is anything arriving from the peers
sudo timeout 10 tcpdump -ni any udp port 5405 -c 5

# Quorum device, if configured
sudo corosync-qdevice-tool -s
sudo pcs quorum device status

Do not use nc -zuv <node> 5405 as a reachability test. For UDP, nc -z reports success unless an ICMP port-unreachable comes back; corosync never replies to a stray datagram, and a firewall DROP suppresses the ICMP. It reports “succeeded” against a blocked port and sends you down the wrong branch while the cluster is degraded.

Step 3: Fix the cause, not the symptom

In descending order of preference:

Restore the network path. A partition caused by a switch, a firewall rule or a bonded link is fixed on the network, and quorum returns on its own within a token timeout.
Bring the failed node back. If a node is genuinely down, power it on and let it rejoin.
Restore the quorum device. A missing qdevice vote is a service problem on the witness host, not a cluster problem.

Service impact possiblerestore membership— Restarts cluster services on one node. Affects that node's membership only.

# On the recovered node, after the network path is proven
sudo corosync-cfgtool -s
sudo pcs cluster start
sudo pcs status

# Quorum device service on the witness
sudo systemctl status corosync-qnetd
sudo systemctl start corosync-qnetd

Only when the underlying cause is fixed does quorum come back safely, because only then are the other nodes really there.

Step 4: When nodes are genuinely gone

If nodes are confirmed down — you have looked at the console or the BMC, they are powered off, and they cannot be returned within the outage tolerance — the cluster can be told to expect fewer votes.

Cluster-wide riskexpected_votes— Overrides the vote count the cluster requires. Runtime only; resets on restart.

# CONFIRM FIRST, from the BMC, that the missing nodes are powered off.
# Then lower the expectation on a surviving node.
sudo corosync-quorumtool -e 2

sudo corosync-quorumtool -s
pcs status

Cluster-wide riskpcs quorum unblock— Forces quorum during startup while nodes are unavailable. Fences the missing nodes.

# Only during a cluster start where some nodes cannot be contacted.
# This will attempt to fence the nodes it cannot see.
sudo pcs quorum unblock

pcs quorum unblock is safer than a bare vote override precisely because it fences what it cannot see — but that means it will power-cycle those nodes. Never run it on the assumption that fencing is disabled.

Step 5: Two-node clusters

Read-only / Safetwo-node config— Reads the votequorum configuration.

sudo corosync-cmapctl | grep -E 'two_node|wait_for_all|expected_votes|last_man_standing'
pcs property config | grep -i 'priority-fencing-delay\|stonith'
pcs stonith config | grep -i pcmk_delay

Step 6: Verify, and undo the overrides

Read-only / Safeverify quorum— Confirms quorum, membership and single-instance resources.

pcs quorum status
sudo corosync-quorumtool -s
pcs status --full

# Every resource must appear exactly once
pcs status resources

# Fencing must be armed and every device Started
pcs property config stonith-enabled
pcs stonith status

Cluster-wide riskrestore expected_votes— Returns the vote expectation to the configured value.

# Once every node is back and Online
sudo corosync-quorumtool -e 3
sudo corosync-quorumtool -s

The incident is not closed while an override is in place. A cluster running with a lowered vote expectation will accept the next partition silently.

Common patterns

Symptom	Likely cause	Resolution
Two groups each quorate	Network partition with quorum forced or two_node	Stop one side immediately; fix the network
Short by exactly one vote, all nodes Online	Quorum device down	Restart `corosync-qnetd` on the witness
Quorum lost after a reboot of one node in three	Expected; returns when the node rejoins	Wait for rejoin; investigate why it rebooted
Cluster will not start after a full outage	`wait_for_all` waiting for every node	Bring all nodes up, or `pcs quorum unblock` knowingly
Resources stopped everywhere, cluster inquorate	Working as designed	Restore membership; do not force quorum first
Both nodes fenced each other	Two-node fence race	Add `pcmk_delay_max` or a third vote
Quorum returns but resources do not start	Leftover constraint or failed action	`pcs constraint config --full`, `pcs resource cleanup`