The monitor quorum alert and its response

Why this matters in production

Quorum loss is the only Ceph failure where the cluster stops entirely, and the recovery involves commands most operators have never run. Preparing the response is the work; running it is straightforward.

The first five minutes

# 1. what is the actual state?
ceph -s --connect-timeout 5
ceph quorum_status --connect-timeout 5

If those hang, quorum is already lost and the CLI cannot reach a leader. Go directly to the daemons:

# 2. per-monitor state, from each monitor host
ceph daemon mon.$(hostname -s) mon_status | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("state:", d["state"])
print("quorum:", d.get("quorum"))
print("monmap:", [m["name"] for m in d["monmap"]["mons"]])'

# 3. which monitors are running?
for h in mon-01 mon-02 mon-03; do
  printf '%-8s ' "$h"
  ssh "$h" 'systemctl is-active ceph-*@mon.*' 2>/dev/null || echo unreachable
done

# The cluster fsid names the systemd unit. With quorum lost `ceph fsid`
# hangs, so read it from /etc/ceph/ceph.conf and substitute it here:
FSID=3e0b2c14-9f3a-4d21-8a77-1c9f0e2b5d64

# 4. why did they stop?
journalctl -u "ceph-$FSID@mon.$(hostname -s)" --since '30 min ago' | tail -50
df -h /var/lib/ceph

Four commands establish whether this is a stopped daemon, a full filesystem, a network partition, or a host failure.

The common causes and their fixes

Preparing

The artefacts this response needs, prepared in advance:
  - the monitor hostnames and their addresses
  - the cluster fsid
  - a copy of the monmap, exported regularly
  - the procedure for removing a dead monitor from the monmap
  - the procedure for rebuilding a monitor from a surviving one
  - who has access to the monitor hosts

# export the monmap periodically
ceph mon getmap -o /backup/monmap-$(date +%F)
monmaptool --print /backup/monmap-$(date +%F)

Rehearsing

# capture the fsid while quorum still holds
FSID=$(ceph fsid)

# in a test cluster: stop a monitor, verify quorum holds
systemctl stop "ceph-$FSID@mon.b"
ceph -s

# stop a second on a three-monitor cluster: quorum lost
systemctl stop "ceph-$FSID@mon.c"
ceph -s --connect-timeout 5      # hangs

# restore
systemctl start "ceph-$FSID@mon.c"
ceph quorum_status

Running this once in a test environment is what makes the production response calm rather than exploratory.

Quiz

Production discipline

Keep the monitor addresses, cluster fsid, and an exported monmap somewhere reachable without the cluster — every artefact the quorum response needs is otherwise inside the thing that is down. Rehearse the full loss and recovery once in a test cluster; the commands are unfamiliar and the incident is the wrong time to learn them.

Cross-course references

• Kubernetes: etcd quorum recovery needs the same prepared artefacts and rehearsal
• Linux: any consensus system’s recovery procedure must live outside the system