How a monitor rejoins quorum

Why this matters in production

A monitor whose systemd unit is active can sit outside quorum indefinitely, and the difference between a slow rejoin and a stuck one decides whether you wait or intervene.

The sequence

probing -> synchronizing -> electing -> peon

Reading mon_status

ceph tell mon.ceph-b mon_status

# the monitor is unreachable through the cluster, so go to the host
sudo cephadm enter --name mon.ceph-b -- \
  ceph daemon mon.ceph-b mon_status | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("state:    ", d["state"])
print("rank:     ", d["rank"])
print("quorum:   ", d.get("quorum"))
print("outside:  ", d.get("outside_quorum"))
print("probing:  ", d.get("extra_probe_peers"))
print("epoch:    ", d["monmap"]["epoch"])'

What blocks a rejoin

ceph health detail | grep -A3 -E 'MON_DOWN|MON_CLOCK_SKEW|MON_DISK'
ceph config get mon mon_clock_drift_allowed
chronyc sources -v | head -8

# is it moving, or is it stuck
FSID=$(ceph fsid)
for i in 1 2 3; do
  sudo du -sm /var/lib/ceph/$FSID/mon.ceph-b/store.db
  sleep 30
done

Confirming from the cluster

ceph quorum_status --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
mons = [m["name"] for m in d["monmap"]["mons"]]
missing = [m for m in mons if m not in d["quorum_names"]]
print("map:     ", ",".join(mons))
print("quorum:  ", ",".join(d["quorum_names"]))
print("missing: ", ",".join(missing) or "none")'

ceph mon stat
ceph health detail

Quiz

Production discipline

Judge a rejoining monitor by whether its store is growing, not by how long it has been synchronising — restarting a slow one puts it back at the beginning. Treat a monitor that joins and leaves in a cycle as a clock problem until chronyc says otherwise, and never widen mon_clock_drift_allowed to silence it.

Cross-course references

• Kubernetes: a Ready probe passing is not the same as a member having joined a quorum
• Linux: cyclic membership loss with no process restart is almost always time synchronisation