Quorum failure exercises — two scenarios worth rehearsing

Why this matters in production

Both scenarios here are common, both are entirely preventable at design time, and both are much easier to handle if you have thought about them before rather than during.

Scenario A — one host, two monitors

A cluster grew from three hosts to twelve. During a rebuild, two monitors ended up on the same physical host. Nobody noticed, because ceph mon dump shows monitor names and addresses, not physical hosts.

The host is rebooted for patching. Two of three monitors disappear simultaneously. The cluster stops.

Diagnosis. ceph orch ps --daemon-type mon shows the placement, and comparing hostnames reveals two on one host. In the moment, the symptom is that two monitors went away from one action.

Immediate action. Bring the host back. Quorum returns and the cluster resumes. There is nothing to repair.

The design fix.

ceph orch apply mon --placement="ceph-01 ceph-05 ceph-09"
ceph orch ps --daemon-type mon

with the hosts chosen from different racks. Then add a check:

ceph orch ps --daemon-type mon --format json | \
  jq -r '.[].hostname' | sort | uniq -d

Any output means two monitors share a host.

Scenario B — a network split isolating one monitor

Five monitors: three in room A, two in room B. The inter-room link fails.

Room A holds three of five — quorum. It marks room B’s OSDs down and then out, and begins recovery. Room B has two monitors, no quorum, and its OSDs stop serving despite being healthy.

Diagnosis. From room A, ceph quorum_status shows three in quorum. From room B, commands hang. The split is symmetric, so the two sides agree about who is where — unlike an asymmetric fault.

Immediate action. Repair the link. Room B’s monitors rejoin, its OSDs peer, and PGs recover the changes they missed. If the link will be down for a long time, ceph osd set noout from room A prevents re-replication of room B’s data — worth doing if room B is coming back soon, since re-replicating it and then re-replicating back is wasted work.

The design question. Room A decides every partition. That should be deliberate: room A should be where the more important workload lives, or the monitor split should be reconsidered. A three-two split across two rooms is not a stretch cluster; it is a cluster with a designated survivor.

What both scenarios share

Neither was a Ceph fault. Both were topology decisions that nobody wrote down: monitors on the same host, and monitors split so that one room always wins. In both cases the cluster behaved exactly as designed — the design was simply not the one anyone intended.

Quiz

Production discipline

Verify monitor placement against physical hosts and racks on a schedule — Ceph knows names and addresses, not shared hardware, so drift is silent and only you can detect it. Treat a two-site three-two monitor split as a designated-survivor design and make the choice of survivor deliberate, or deploy stretch mode with a tiebreaker if either site must genuinely survive. And when a known outage has a known return time, noout avoids re-replicating data twice — provided you reassess rather than forget the flag.

Cross-course references

• Ceph: Part CXIX (Disaster Recovery Architecture) for stretch mode.
• Ceph: Part IV (Failure Domains) for placing monitors correctly.
• Ceph: Part XCV (Maintenance Flags) for noout during known outages.