Cluster IR: quorum loss - the arithmetic and the decision record

Quorum loss is the incident where the cluster stops doing anything and the operator has to resist the one command that makes it start again. linux-failure-cluster-and-dependencies establishes the rule — never force quorum on a partition you have not proven is alone. This lesson is about the work either side of that rule: reading the arithmetic precisely enough to know what is missing, and writing the record that makes a manual override a decision rather than a reflex.

The arithmetic, field by field

Everything you need is in one output. Read it as a sum, not as a status:

# corosync-quorumtool -s

Quorum information
------------------
Date:             Tue Aug 11 04:41:02 2026
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          1
Ring ID:          1.31a
Quorate:          No

Votequorum information
----------------------
Expected votes:   5
Highest expected: 5
Total votes:      2
Quorum:           3
Flags:

Membership information
----------------------
  Nodeid      Votes Name
       1          1 node1 (local)
       2          1 node2

In the capture above: five expected, two present, three needed. Three votes are missing. The membership list has two entries, so the missing votes are three cluster nodes — unless one of the five votes was never a node at all, which is the next section.

When the missing vote is not a node

A quorum device contributes votes without being a cluster node, so a healthy fleet can lose quorum because a witness went away. This is the quorum-loss incident that confuses people most: every node is up, pcs status shows all nodes online, and the cluster is frozen.

# corosync-quorumtool -s

Quorum information
------------------
Date:             Tue Aug 11 04:58:11 2026
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          1
Ring ID:          1.3c8
Quorate:          No

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2  Activity blocked
Flags:            Qdevice

Membership information
----------------------
  Nodeid      Votes    Qdevice Name
       1          1   NR       node1 (local)
       2          1   NR       node2
       0          1            Qdevice (votes 1)

The Qdevice column carries the state per node. NR is not registered — this node is not currently being polled by the quorum device. A registered, voting device shows A,V,NMW (alive, vote, not-master-wins) or similar. The device line at node ID 0 shows the votes the device is configured to contribute, which is not the same as votes it is contributing.

Check the device from both ends:

# On a cluster node: is the local qdevice client running and connected?
systemctl status corosync-qdevice --no-pager
pcs quorum device status

# On the arbitrator host: is qnetd running and does it see the cluster?
systemctl status corosync-qnetd --no-pager
corosync-qnetd-tool -l

# Substitute your own values before running:
QNETD_HOST=192.0.2.60

# The qnetd port must be reachable from every node - a firewall
# change on the arbitrator is a common cause
ss -lntp | grep 5403
timeout 5 bash -c "cat < /dev/null > /dev/tcp/$QNETD_HOST/5403" \
  && echo "qnetd port reachable" || echo "qnetd port NOT reachable"

The maintenance-window trap

Expected votes can be lowered deliberately. During a planned reduction — one node out for a motherboard swap, a site down for a power test — an operator drops the expected count so the remainder stays quorate:

# Substitute your own values before running:
NEW_EXPECTED=2

# Legitimate during a planned reduction. Runtime only: it does
# not survive a corosync restart on the node where it is run.
corosync-quorumtool -e "$NEW_EXPECTED"

The trap is that this is easy to do and easy to forget. If it is never restored, the cluster now has a permanently lowered bar, and a later single failure gives you a quorate partition that should not have been quorate.

Highest expected is the tell. When it exceeds Expected votes, the value has been reduced and not put back:

# corosync-quorumtool -s | grep -E 'Expected|Quorum:'

Expected votes:   2
Highest expected: 3
Quorum:           2

Compare it against the configured value, which is where the cluster returns to on restart:

# The persistent configuration, as opposed to the runtime value
grep -E 'expected_votes|two_node|wait_for_all' /etc/corosync/corosync.conf
corosync-cmapctl | grep -E 'quorum.expected_votes|runtime.votequorum'

A runtime value that disagrees with corosync.conf is a finding worth a ticket even when nothing is currently broken. It means the cluster behaves one way today and a different way after the next reboot.

The manual override, and its record

Sometimes the peers really are gone and are not coming back inside the incident window — a rack lost power, a site is out, the hardware is dead. Relaxing quorum on the survivors is then a legitimate act. It is also an act that removes the protection preventing split brain, so it needs to look like a decision.

1. Establish out of band, per node, that each missing node is powered off or unable to run resources. Record how you established it.
2. Where you cannot establish it, power the node off at the BMC and record that instead.
3. Name an owner for the decision. This is not the on-call engineer acting alone unless your escalation policy says it is.
4. Relax quorum on the surviving partition, with the smallest change that restores service.
5. Write the decision record before you move on, while the reasoning is still in your head.
6. Restore the original quorum configuration as soon as the peers return, and verify Expected votes matches Highest expected again.

The record is four lines and it is the difference between an override and an accident:

QUORUM OVERRIDE
===============
Incident:   INC-4820
Time (UTC): 2026-08-11T05:14Z
Actor:      e.brandi        Authorised by: s.okonkwo (on-call lead)
Missing:    node3, node4, node5 (rack B)
Evidence:   BMC chassis power status = off for all three, taken from
            jump01 on the OOB network at 05:09Z. Console screenshots
            attached to INC-4820.
Action:     corosync-quorumtool -e 2 on node1 (runtime only; corosync.conf
            unchanged, so the original bar returns on restart)
Risk:       Split brain if any of node3-5 regains power and rejoins.
            Mitigated: all three left powered off at the BMC.
Revert by:  restore expected_votes to 5 once rack B is back; verify
            Expected votes == Highest expected on every node

The Risk and Revert by lines are the ones people skip and the ones that matter. Without Revert by, the temporary override becomes the permanent configuration, and the next incident starts from a weakened cluster.

What quorum loss looks like from the service side

The cluster freezing is the safety mechanism, but “frozen” means different things depending on no-quorum-policy:

Check what yours is set to before you interpret anything:

pcs property config no-quorum-policy

A cluster on freeze with a still-answering VIP is the case that gets misread most often. The service looks alive, so the incident is under-declared, and the first additional failure takes it out with no cluster left to recover it.

Knowledge check