Getting one monitor back is the whole job

Why this matters in production

When quorum is lost on a three-member cluster, restoring any one failed monitor to service ends the incident. Every more dramatic option — store restores, monmap surgery — exists for the case where that is impossible, and reaching for one first is how a recoverable outage becomes a destructive one.

Establish the real state first

# on each monitor host, in turn
systemctl status ceph-$FSID@mon.ceph-mon-b.service
cephadm ls --no-detail | grep mon
ceph daemon mon.ceph-mon-b mon_status
journalctl -u ceph-$FSID@mon.ceph-mon-b.service -n 100 --no-pager

Two monitors that report probing are a different incident from one that will not start at all. Collect all three before deciding anything.

Preference order

The fourth row is the one people skip. A monitor whose RocksDB store will not open is not a data loss event as long as one monitor is alive: remove the member and add it back using the ordinary mechanics, and it rebuilds its store by synchronising from the quorum.

The store partition filling

df -h /var/lib/ceph/$FSID/mon.ceph-mon-b
du -sh /var/lib/ceph/$FSID/mon.ceph-mon-b/store.db
ceph config get mon mon_data_avail_crit
ceph config get mon mon_data_avail_warn

A monitor stops when free space falls below mon_data_avail_crit, five per cent by default, and warns from mon_data_avail_warn at thirty. The health checks are MON_DISK_LOW, MON_DISK_CRIT, and MON_DISK_BIG, and they usually fired for days before anyone looked.

ceph daemon mon.ceph-mon-a compact
ceph config set mon mon_compact_on_start true

Compaction reclaims space, but only space that is already trimmable — the monitors retain OSDMap history until every OSD has caught up, so a cluster that has been unhealthy for a week has a store that cannot shrink until it is clean again. Free space on the partition first; compaction is not a substitute for capacity.

Verify quorum formed, and that it stayed

ceph daemon mon.ceph-mon-a mon_status
ceph quorum_status -f json-pretty
ceph mon stat
ceph log last 200 info cluster | grep -i election

mon: 3 daemons, quorum ceph-mon-a,ceph-mon-b,ceph-mon-c (age 94s)

A monitor in peon or leader is in quorum. One that oscillates between electing and peon has not recovered — a quorum that re-forms every few seconds answers ceph -s intermittently and is easily mistaken for success.

Quiz

Production discipline

Work the preference order strictly: start what can be started, free space where space is the fault, and remove and re-add a monitor whose store is broken — all of which are safe while any monitor remains quorate. Never restore a monitor store into a cluster with no quorum, and never declare recovery on a quorum that is re-electing every few seconds.

Cross-course references

• Kubernetes: an etcd member with a broken data directory is replaced, not repaired
• Linux: a service that exits at start after logging a disk threshold is telling you the answer