Forcing quorum by editing the monmap

Advanced⏱ ~18 minceph-monmonmaptoolcephadm

What you'll learn

State the preconditions that must hold before editing a monmap
Extract, edit, and inject a monmap on a stopped monitor
Run the procedure under cephadm where the tools live in a container
Verify what survived and what did not

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Removing monitors from the monmap makes a single survivor a quorum of one and brings a dead cluster back. It also makes that survivor’s store the undisputed truth, discards whatever it never received, and cannot be undone. It is correct in exactly one situation and a catastrophe in every other.

Preconditions

Every one of these must hold before you touch anything.

Precondition	How to establish it
The other monitors are unrecoverable	hardware confirmed dead, not merely unreachable
No other side is running its own monitors	confirm the partition is not the actual fault
The survivor’s store opens	the daemon starts and reaches `probing`
You have the fsid	`/etc/ceph/ceph.conf`, or the daemon directory name
You have the admin keyring	on this host, or from your off-cluster copy

ceph fsid needs a quorum and will not answer. Read it from the configuration file or the cephadm directory layout under /var/lib/ceph/.

The procedure

A monitor must be stopped before its monmap is extracted or injected; running the tools against a live store is how a recoverable outage becomes an unrecoverable one.

systemctl stop ceph-$FSID@mon.ceph-mon-a.service
ceph-mon -i ceph-mon-a --extract-monmap /tmp/monmap
monmaptool --print /tmp/monmap
monmaptool /tmp/monmap --rm ceph-mon-b --rm ceph-mon-c
monmaptool --print /tmp/monmap
ceph-mon -i ceph-mon-a --inject-monmap /tmp/monmap
systemctl start ceph-$FSID@mon.ceph-mon-a.service

monmaptool: monmap file /tmp/monmap
epoch 7
fsid 4d2b8c1e-9f0a-4c7d-b2e1-8a3f5c6d7e90
election_strategy: 1
0: [v2:10.20.4.10:3300/0,v1:10.20.4.10:6789/0] mon.ceph-mon-a

Print before and after: the map you inject is the entire membership the cluster will have, and a typo in a name silently removes the wrong member.

Under cephadm

ceph orch is unavailable — the MGR needs the monitors — so everything here is host-local. A temporary file inside cephadm shell does not outlive the invocation, so do the whole edit in one call.

cephadm unit --fsid $FSID --name mon.ceph-mon-a stop
cephadm shell --fsid $FSID --name mon.ceph-mon-a -- bash -c '
  ceph-mon -i ceph-mon-a --extract-monmap /tmp/monmap &&
  monmaptool --print /tmp/monmap &&
  monmaptool /tmp/monmap --rm ceph-mon-b --rm ceph-mon-c &&
  ceph-mon -i ceph-mon-a --inject-monmap /tmp/monmap'
cephadm unit --fsid $FSID --name mon.ceph-mon-a start

What you have agreed to lose

Category	Effect
Object data on OSDs	untouched — it lives on the OSDs, not the monitors
Recent OSDMap epochs	any the survivor had not received are gone
Pools created just before the failure	may not exist in the restored map
Recent cephx key changes	may revert to their earlier values
Recent CRUSH edits	may be undone, changing where data is expected
The removed monitors	permanently out; their hosts must be wiped

Verify each deliberately rather than assuming the cluster came back correct.

ceph -s
ceph mon stat
ceph osd dump | head -3
ceph daemon osd.0 status
ceph osd pool ls detail
ceph auth ls | head -20

Compare newest_map from the OSD against the epoch the monitors now serve, then re-add monitors with the ordinary mechanics to return to an odd count.

Quiz

Knowledge check · 4 questions

Q1. What must be true of a monitor before you extract or inject its monmap?

It must be the current Paxos leader, so the edit is authoritative It must hold quorum with at least one other surviving member It must be stopped — the tools operate on the store directly It must have a manager co-located, which serves the map to tools
Q2. Removing monitors from the monmap can discard recently committed cluster state even though no object data is touched.

True False
Q3. Restore a cluster whose monitor majority is physically destroyed.

A rack fire destroyed the hosts running mon.ceph-mon-b and mon.ceph-mon-c. mon.ceph-mon-a is intact, starts, and sits in `probing`. Around 400 TiB of data is on OSDs in other racks and is untouched.
Q4. Why must the hosts of removed monitors be wiped rather than restarted later?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Treat monmap surgery as irreversible and take a filesystem copy of the survivor’s store directory first — it is the only rollback you will have. Print the map before and after the edit, verify pools, auth, and CRUSH against expectation before declaring recovery, and wipe the removed hosts rather than ever starting them again.

Cross-course references

Kubernetes: etcd --force-new-cluster carries the same one-way, one-survivor-wins semantics
Linux: any tool that edits a live daemon’s on-disk state expects the daemon to be stopped