Three monitors and what the arithmetic actually buys

Advanced⏱ ~18 minceph

What you'll learn

Compute the majority a monmap requires
Read monitor state when the normal tooling cannot answer
Identify the two ways three monitors become zero margin
State what stops the moment quorum is lost

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Losing quorum takes every other recovery tool away at the same moment, including the command you were about to run to diagnose it. Knowing exactly what the majority is computed from, and how to read monitor state without a working cluster, is the difference between a twenty-minute incident and an afternoon.

The arithmetic, and what it counts

The majority is derived from the number of monitors in the monmap, not from the number of monitor daemons currently running. Three members means two votes are required and one failure is survivable.

ceph mon stat
ceph quorum_status -f json-pretty
ceph mon dump

epoch 7
fsid 4d2b8c1e-9f0a-4c7d-b2e1-8a3f5c6d7e90
last_changed 2026-06-02T11:04:18.229915+0000
election_strategy: 1
0: [v2:10.20.4.10:3300/0,v1:10.20.4.10:6789/0] mon.ceph-mon-a
1: [v2:10.20.4.11:3300/0,v1:10.20.4.11:6789/0] mon.ceph-mon-b
2: [v2:10.20.4.12:3300/0,v1:10.20.4.12:6789/0] mon.ceph-mon-c

Reading monitor state without assuming quorum

ceph -s needs a quorum to answer. The admin socket does not — it is served by the local daemon regardless of what the rest of the cluster is doing, which makes it the first command of every quorum incident.

ceph daemon mon.ceph-mon-a mon_status
ceph daemon /var/run/ceph/$FSID/ceph-mon.ceph-mon-a.asok mon_status
cephadm enter --name mon.ceph-mon-a -- ceph daemon mon.ceph-mon-a mon_status

State	Meaning
`probing`	looking for the other members named in its monmap
`synchronizing`	pulling store contents from the quorum; does not vote
`electing`	an election is in progress
`leader`	holds quorum and drives Paxos proposals
`peon`	in quorum, following the leader

The two ways three becomes zero margin

A monitor decommissioned in practice but never removed from the monmap is still in the denominator. The cluster looks like it has one spare and has none.

A monitor that restarts enters synchronizing and does not vote until it has caught up. Two daemons running is therefore not two votes, and on a cluster whose store has grown during a long unhealthy period the catch-up is not instantaneous.

ceph mon dump | grep -c '^[0-9]*:'
ceph node ls mon

What stops when quorum stops

Capability	Without quorum
`ceph -s`, `ceph health`, `ceph osd ...`	block, then time out
`ceph orch` and the dashboard	unavailable — the MGR needs the monitors
Marking OSDs up, down, in, or out	nothing commits
PG peering and new acting sets	cannot complete
New client sessions and mounts	fail to obtain a map
cephx ticket renewal	fails once `auth_service_ticket_ttl` expires
In-flight I/O to already-active PGs	continues briefly, then blocks

Quiz

Knowledge check · 4 questions

Q1. Two of three monitor daemons are running after a restart, yet the cluster reports no quorum. What is the most likely reason?

The monmap was rewritten during the restart, so the two survivors disagree about who the members are The restarted monitor is in `synchronizing` state and does not vote until it has caught up A three-member monmap requires all three to be reachable, since Paxos proposals need unanimous acknowledgement The MGR has not reconnected, and quorum is only reported once a manager is active to publish it
Q2. A monitor host that has been powered off permanently no longer counts toward the majority once it stops running.

True False
Q3. Respond to a monitor failure on a three-monitor cluster.

mon.ceph-mon-c has been down for six hours after a disk failure. A scheduled kernel patch window for ceph-mon-b starts in one hour, and the change ticket is already approved.
Q4. Why is the admin socket the first command of a quorum incident rather than `ceph -s`?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Never start a monitor maintenance window while another monitor is down — the remaining pair is the majority, not spare capacity. Remove decommissioned monitors from the monmap the day they leave service, and make ceph daemon mon.<id> mon_status the first line of the quorum runbook, because everything else needs the quorum you are trying to diagnose.

Cross-course references

Kubernetes: an etcd member removed from the cluster must also be removed from the member list
Linux: a cluster manager that counts configured nodes will not forgive a decommission you never recorded