Witness and quorum devices - breaking geographic ties · Linux

A witness (or quorum device) contributes votes to quorum without being a cluster node and without running any workload. It is the solution to geographic splits with even node counts. This lesson covers it.

The geographic problem

A cluster split across two data centres:

Site A: 3 nodes.
Site B: 3 nodes.
Partition: sites cannot reach each other.

Each site has 3 nodes, which is quorum. Both groups think they are the cluster. Split brain.

The math: 3 + 3 = 6 nodes, quorum = 4. Either site has 3 nodes, neither has quorum. Both are minority.

Without intervention, the cluster is broken.

The witness

A witness contributes votes to quorum but is not a cluster member and runs no workload. It is typically hosted in a third location (cloud, separate network).

Site A: 3 nodes            -> 3 votes
Site B: 3 nodes            -> 3 votes
Witness (qdevice):         -> 1 vote   (third location)
Total: 6 nodes, 7 votes, quorum 4

On partition:

Site A (3 votes) + witness (1) = 4. Quorum.
Site B (3 votes), witness unreachable = 3. Minority.
Site A continues; Site B is fenced.

The witness breaks the tie.

Configure qdevice for Corosync

Corosync supports qdevice via the corosync-qnetd daemon. The work happens in two places, and the two commands are easy to confuse because both contain the word qdevice.

On the witness host - this creates the qnetd service and its certificate database:

sudo apt install corosync-qnetd
sudo pcs qdevice setup model net --enable --start
sudo pcs qdevice status net

On one cluster node - this registers the cluster against the device:

# Authenticate the cluster to the witness host first
sudo pcs host auth witness.example.com

sudo pcs quorum device add model net \
    host=witness.example.com algorithm=ffsplit

Verify from a cluster node:

sudo pcs quorum status
# Expect a Qdevice line, Quorate: Yes,
# Total votes 3 for a 2-node cluster (2 nodes + 1 qdevice), Quorum 2

sudo corosync-quorumtool -s

Choosing the algorithm

The algorithm decides how many votes the device casts and which partition it favours.

ffsplit (fifty-fifty split): the device holds 1 vote and gives it to the partition with more than half the nodes, or on an exact tie to the partition containing the lowest node ID. This is the usual choice for an even split such as 3+3 or a two-node cluster.
lms (last man standing): the device holds n-1 votes, where n is the number of nodes. Any single node that can still reach the qnetd host keeps quorum. This is what you want when surviving on one remaining node matters more than symmetry.

The choice changes which partition survives, so it belongs in the design document, not in the person who typed the command. With lms a single node plus the witness outvotes every other node combined - powerful, and exactly wrong if your real risk is a site link failure rather than node loss.

Trade-offs

A witness is a single point of failure (for quorum). If the witness is unreachable, the cluster falls back to site quorum (which may not work for 3+3).
The witness must be on a different network failure domain than the cluster.
The witness should be small (no workload, just voting).

When to use a witness

2-site cluster with even node counts.
Network between sites can be partitioned.
Cost of split brain is high.

For single-site clusters, no witness is needed (sites do not partition internally in a well-designed network).