CephCXVII · Lost Monitor QuorumLost Monitor Quorum
Five monitors and the placement that makes them count
What you'll learn
- Compare failure tolerance across monitor counts
- Evaluate monitor placement against failure domains
- State what a larger monitor set costs
- Choose an election strategy deliberately
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
Five monitors is the standard answer for a large cluster, and the number alone decides nothing. A five-member set concentrated in two rooms tolerates fewer real-world failures than a three-member set spread across three, and the monmap will not tell you which one you have.
The arithmetic across sizes
| Members | Majority | Failures tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
| 6 | 4 | 2 |
| 7 | 4 | 3 |
Even counts buy no additional tolerance over the odd number below them, cost an extra host, and add the case where an even split leaves neither half able to act. Use odd counts.
Placement is the variable that matters
ceph mon dump
ceph node ls mon
ceph orch ps --daemon-type mon| Layout | Host failures tolerated | Domain failures tolerated |
|---|---|---|
| 3 monitors, 3 rooms | 1 | 1 |
| 5 monitors, 2 rooms as 3+2 | 2 | 0 |
| 5 monitors, 3 rooms as 2+2+1 | 2 | 1 |
| 5 monitors, 5 racks | 2 | 2 |
| 5 monitors, 2 hypervisor hosts | 2 | 0 |
The 3+2 layout is the common trap: it passes every count-based check and loses quorum entirely when the larger room goes. So does a five-member set whose VMs land on two hypervisors, which no Ceph command will reveal.
What five costs
Every Paxos commit requires acknowledgement from a majority, so with five members the third-fastest acknowledgement gates every map change — the median monitor sets the pace, and a single slow store or saturated link raises latency for the whole cluster. There are also more stores to keep below the disk thresholds and more sync traffic after any restart.
ceph daemon mon.ceph-mon-a perf dump | head -30Election strategy
ceph mon dump | grep election_strategy
ceph mon set election_strategy connectivity| Strategy | Behaviour |
|---|---|
classic | lowest-ranked reachable monitor wins; the default |
disallow | as classic, but named monitors may never lead |
connectivity | scores monitors by observed peer connectivity and prefers the best-connected leader |
ceph mon add disallowed_leader ceph-mon-e
ceph mon rm disallowed_leader ceph-mon-econnectivity is the strategy intended for members separated by links of
unequal quality; disallow is the lighter tool for keeping leadership off
one known-poor member.
Adding monitors is not incident response
A monmap change is itself a Paxos commit, so it requires the quorum you have already lost. Monitor count is a design decision to make while the cluster is healthy, and the moment you need a fourth member is the moment you cannot add one.
Quiz
Knowledge check · 4 questions
Q1. What does moving from three monitors to five change about commit latency?
Q2. Five monitors split three and two across two rooms tolerate fewer room failures than three monitors spread across three rooms.
Q3. Review a proposed monitor expansion.
A team plans to grow from three monitors to five to improve resilience. All five will be virtual machines, and the virtualisation team will place them wherever capacity allows.
Q4. Why can adding a monitor never be part of the response to lost quorum?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Audit monitor placement against real failure domains — rack, room, power feed, and hypervisor — rather than against the member count, and record the resulting tolerance next to the monmap export. Keep the count odd, and treat any monitor expansion as healthy-cluster work: the moment you want another member is the moment you can no longer add one.
Cross-course references
- Kubernetes: control-plane replicas without anti-affinity are one node away from a single point of failure
- Linux: quorum devices exist because member counts alone never describe a topology