Triaging why quorum is gone

Why this matters in production

Every cause of quorum loss presents as the same silence from ceph -s, and the recoveries range from restarting a service to irreversible surgery. The triage is the whole skill; the fix is usually short once you know which one you have.

What you can still collect

cephadm ls --no-detail | grep mon
systemctl status ceph-$FSID@mon.ceph-mon-b.service
journalctl -u ceph-$FSID@mon.ceph-mon-b.service -n 200 --no-pager
ceph daemon mon.ceph-mon-b mon_status
df -h /var/lib/ceph/$FSID/mon.ceph-mon-b
chronyc tracking
ss -ltn | grep -E ':3300|:6789'

Run all of it on every monitor host before forming a hypothesis. Two monitors stuck in probing and one refusing to start are three different answers depending on which you look at first.

Signatures and their recoveries

Clock skew

chronyc tracking
chronyc sources -v
ceph config get mon mon_clock_drift_allowed

The warning threshold is fifty milliseconds, which is nowhere near where quorum breaks. Seconds of offset is what matters, and it arrives when an NTP source fails, when a virtual machine is live-migrated, or when one host was never configured in the first place.

The election loop

2026-08-18T09:14:04 mon.ceph-mon-a calling monitor election
2026-08-18T09:14:09 mon.ceph-mon-a calling monitor election
2026-08-18T09:14:14 mon.ceph-mon-a calling monitor election

Quorum that forms and dissolves every few seconds is its own signature. Its causes are clock skew, packet loss between members, a monitor starved of CPU or I/O, or a store so large that synchronisation never finishes before the next election. ceph -s answering once and then timing out is this, not an intermittent network to your workstation.

Deciding where you are

ceph log last 500 info cluster | grep -iE 'election|quorum'

If any monitor can be restored to service, you are in the restoration case. Only when every remaining member is confirmed unrecoverable does the monmap surgery become correct — and that decision needs the same evidence you have just gathered, written down.

Quiz

Production discipline

Collect the same evidence set from every monitor host before forming a hypothesis, because the first host you log into will otherwise decide your diagnosis. When a majority fails simultaneously, look for the shared cause or the recent change first, and fix time before anything else — every diagnosis made under skew is contaminated by it.

Cross-course references

• Kubernetes: simultaneous control-plane failures almost always trace to one rollout or one shared dependency
• Linux: lease-based protocols surface clock problems as flapping membership, never as clock errors