Quorum — why majority is the only safe rule

Why this matters in production

Quorum is a rule about who is allowed to make decisions, and it exists to guarantee one property: two disjoint groups can never both hold quorum at the same time.

The proof is arithmetic. Two groups that are both strict majorities of the same set would together contain more than the whole set, which is impossible. That single fact is what prevents split-brain, and everything else about quorum design follows from it.

The arithmetic

members  quorum needed  failures tolerated
   1           1                0
   2           2                0
   3           2                1
   4           3                1
   5           3                2
   6           4                2
   7           4                3

Two observations follow immediately:

• Even counts add nothing. Four tolerates one failure, exactly as three does. Six tolerates two, exactly as five does. The extra member adds cost and the possibility of an even split without adding tolerance.
• Two members are worse than one. With two, quorum is two, so either failure stops the system — twice the failure surface, no tolerance.

Why not more monitors

Since three tolerates one failure and five tolerates two, more sounds strictly better. It is not free:

• Every map change requires a majority to agree, so commit latency rises with member count.
• Every monitor stores the full monitor database, so more monitors means more replication of that store.
• Elections take longer with more participants.

Three is right for most clusters. Five is appropriate for large clusters, for clusters spanning more than one failure domain where two domains must each be able to lose a monitor, and for clusters where monitor maintenance happens often enough that running at reduced tolerance is common. Seven is rare and usually indicates a topology problem being solved with monitors.

Checking it

ceph quorum_status --format json-pretty
ceph mon stat
ceph mon dump

quorum_status reports which monitors are in quorum, which is the leader, and the monmap epoch. Read it before and after every monitor change, and read it first during any incident involving monitors — before restarting anything.

Quiz

Production discipline

Keep monitor counts odd, and read ceph quorum_status before and after every monitor change rather than reasoning from the configured count — available tolerance is what matters, and a monitor already down consumes it. Never remove a failed monitor from the monmap as tidying-up on a three-member set, since that produces a two-member set with no tolerance at all. Add monitors for topology reasons, not for a vague sense of safety: five costs commit latency and election time, and seven usually means a topology problem being solved in the wrong place.

Cross-course references

• Ceph: Part IX (Monitor Quorum) for the operational scenarios.
• Ceph: Part VIII (Monitors) for what the monitors actually maintain.
• Ceph: Part CXVII (Lost Monitor Quorum) for recovery.