Replacing a lost monitor

Why this matters in production

A three-monitor cluster with one monitor dead tolerates no further failure, and the obvious repair — add the replacement, then tidy up — takes quorum down before the replacement can help.

Establish what you actually have

ceph -s
ceph mon stat
ceph orch ps --daemon-type mon

mon: 3 daemons, quorum ceph-a,ceph-c (age 41m), out of quorum: ceph-b

Remove first, then add

# a healthy monitor you are removing deliberately
ceph mon ok-to-stop ceph-b

# cephadm: drop the daemon, then confirm the monmap really lost it
ceph orch daemon rm mon.ceph-b --force
ceph mon dump | grep -E '^[0-9]+:'
ceph mon remove ceph-b        # only if it is still listed

# the host is gone entirely
ceph orch host rm ceph-b --offline --force

# deploy the replacement by naming the full placement
ceph orch apply mon --placement="ceph-a,ceph-c,ceph-d"

Rebuilding a monitor store by hand

Use this where there is no orchestrator, or where cephadm cannot reach the new host yet. The store is built empty and filled by synchronising from the quorum.

ceph mon getmap -o /tmp/monmap
ceph auth get mon. -o /tmp/mon.keyring

sudo -u ceph mkdir -p /var/lib/ceph/mon/ceph-ceph-d
sudo -u ceph ceph-mon --mkfs -i ceph-d \
  --monmap /tmp/monmap --keyring /tmp/mon.keyring
sudo systemctl enable --now ceph-mon@ceph-d

ceph mon add ceph-d 10.0.1.14:6789

The exported monmap does not contain the new monitor. It is the map of
the cluster the monitor must find, not a description of the monitor.

Verifying

ceph quorum_status --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
mons = [m["name"] for m in d["monmap"]["mons"]]
print("in map:  ", ",".join(mons))
print("in quorum:", ",".join(d["quorum_names"]))
print("tolerates", len(mons) - (len(mons)//2 + 1), "further failures")'

ceph health detail
sudo du -sh /var/lib/ceph/*/mon.ceph-d/store.db

Quiz

Production discipline

Remove the failed monitor from the map before adding its replacement, every time — growing the map raises the quorum requirement while the new monitor is still synchronising. Build the replacement store with ceph-mon --mkfs; copying a working monitor store produces a daemon with a borrowed identity and a Paxos position that is not its own.

Cross-course references

• Kubernetes: etcd member removal precedes member addition for exactly the same quorum arithmetic
• Linux: cluster membership changes are safest when they shrink before they grow