Quorum concepts - the math of agreement

A quorum is the minimum number of nodes that must agree for the cluster to make decisions. This lesson covers the math and the failure modes.

Why quorum

In a distributed system, nodes can fail and networks can partition. Without quorum, two groups of nodes might both think they are the cluster, leading to split-brain (data corruption). Quorum ensures only one group can make decisions.

The rule: a cluster makes decisions only if a majority of nodes agree.

Calculate quorum

For N nodes, the quorum is (N/2) + 1:

Even N gives the same surviving failure count as N-1. So always use odd N for the cluster.

Split brain

A split brain happens when two groups of nodes each think they are the cluster:

• Network partition: nodes 1, 2 can talk to each other but not to 3, 4, 5.
• Without fencing: both groups can promote themselves to primary, leading to data corruption.

Quorum prevents split brain: only the group with majority (3 of 5) can make decisions. The minority (2 of 5) cannot.

The third-party witness

A witness is a node that participates in quorum but does not run the workload. It is used to break ties when the cluster is split geographically:

• Site A: 3 nodes.
• Site B: 3 nodes.
• Witness: 1 node in a third location.

If Site A and Site B are partitioned, the witness votes with the side that can reach it. This breaks the tie.

Fencing

Quorum is not enough. The minority group must be prevented from running the workload. Fencing is the discipline:

• Power fencing (STONITH) — cut the node’s power via IPMI/BMC/iLO or a switched PDU, or have the node fence itself via an SBD watchdog. STONITH stands for Shoot The Other Node In The Head, and in Pacemaker it names this class specifically.
• Storage fencing — leave the node running but revoke its access to the data: SCSI-3 persistent reservations (fence_scsi, fence_mpath) or SAN zoning.
• Hypervisor fencing — stop the VM through the platform API (fence_vmware_soap, fence_virsh).

The classes are not interchangeable. Power fencing is the only one that also stops a node from doing anything else — a storage-fenced node keeps running, keeps holding its cluster IP, and keeps answering clients.

Without fencing, a “stopped” node may still be writing to the disk, causing data corruption.

Fencing across sites

Power fencing works for a single dead node in a single site. It does not work across a partitioned link, and the witness design above is exactly where people assume it does.

Inter-site link DOWN
  Site A (has quorum) wants to fence the Site B nodes
  Fence agent: IPMI to their BMCs
  Those BMCs are at Site B
  The route to Site B is the link that is down
  => the fence request times out

The fence can never succeed. The failure that triggers it is the same failure that removes the path to the fence device. With stonith-enabled=true the surviving site will not start resources it cannot prove are stopped, so the service stays down at both sites. The predictable next move under pressure - disabling STONITH to get the service up - is how a split brain becomes data corruption.

Two designs solve it, and one must be chosen deliberately:

• SBD watchdog self-fencing (one stretch cluster). Each node runs a hardware watchdog. A node that loses quorum stops petting it and is reset after stonith-watchdog-timeout. The survivor waits out the timeout and needs to reach nothing. Use a real hardware or hypervisor watchdog, never softdog.
• booth tickets (two independent clusters). A ticket granted by a booth arbitrator in the third location decides which site may run the service. Resources are constrained on the ticket, and the loss policy is enforced locally - which always works, because the fence target is on the same side of the failed link.

Production choices

• 3 nodes: minimum viable, 1 failure tolerated.
• 5 nodes: better, 2 failures tolerated.
• 7 nodes: large clusters.
• Always odd: even N gives no benefit.
• Geographic distribution: separate sites, separate failure domains.
• Witness: for site failure scenarios.
• Fence path per failure mode: IPMI intra-site; SBD watchdog self-fencing or booth tickets across sites.

Knowledge check