The artefacts and the drill that make the recovery routine

Intermediate⏱ ~17 mincephmonmaptoolcrushtool

What you'll learn

Export the artefacts a quorum recovery depends on
Verify each exported artefact is readable offline
Run the quorum drill that is safe in production
Define what the drill must measure

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Every artefact a quorum recovery needs is produced by a command that requires quorum. The exports must therefore already exist when the incident starts, and the only way to know they do — and that they are readable — is to have used them recently.

The artefacts

Artefact	Command	Why it cannot wait
fsid	`ceph fsid`	every offline tool and unit name needs it
Monitor membership	`ceph mon dump`	you cannot ask a cluster without quorum
Binary monmap	`ceph mon getmap -o`	readable offline with `monmaptool`
Admin keyring	`ceph auth get client.admin`	issued by the monitors you have lost
Configuration	`ceph config dump`	the config database lives in the monitors
CRUSH map	`ceph osd getcrushmap -o`	a lost CRUSH edit makes data unaddressable
Pool definitions	`ceph osd pool ls detail`	the verification target after any surgery

OUT=/secure/ceph-state/$(date +%F)
mkdir -p $OUT
ceph fsid > $OUT/fsid
ceph mon dump > $OUT/mon-dump.txt
ceph mon getmap -o $OUT/monmap.bin
ceph config dump > $OUT/config.txt
ceph osd getcrushmap -o $OUT/crushmap.bin
ceph osd pool ls detail > $OUT/pools.txt
ceph auth get client.admin > $OUT/client.admin.keyring

The keyring is a credential and needs the protection of one. The rest is topology, and worth nothing on a filesystem inside the cluster it describes.

Proving they are readable

monmaptool --print $OUT/monmap.bin
crushtool -d $OUT/crushmap.bin -o /tmp/crush.txt

An export never read back is a hope. Both checks are offline and take seconds, so they belong in the job that produces the files.

The drill that is safe in production

Stop one monitor of three, confirm quorum holds, and time how long the restarted member takes to return to peon.

systemctl stop ceph-$FSID@mon.ceph-mon-c.service
ceph mon stat
systemctl start ceph-$FSID@mon.ceph-mon-c.service
ceph daemon mon.ceph-mon-c mon_status

That interval is the real length of a monitor maintenance window, and it grows quietly as the store does. Measuring it quarterly is what stops someone assuming a restart is instantaneous during an incident.

The drill that must run in staging

Full quorum loss and monmap surgery — stopping two of three, confirming everything stops, then extracting, editing, and injecting a monmap on the survivor. It is irreversible and belongs nowhere near production. Run it on a staging cluster with real data volume, because store synchronisation time is what the exercise is actually measuring.

What the drill measures

Measure	Why it matters
Time to detect	alerting must fire on `MON_DOWN` and on `ceph` timing out
Time to a correct diagnosis	the triage table is the lever
Time for a restarted monitor to reach `peon`	bounds every maintenance window
Whether the runbook works without quorum	the classic and fatal defect
Whether the exports are current and readable	otherwise discovered mid-incident

Why the paper record outlives the monitor store backup

A monitor store backup restored into a live quorum is discarded and resynchronised, and restored into a dead cluster it becomes authoritative at whatever epoch it was taken — so its usefulness decays with every hour since the export, and the situation where you most want it is the one where it does the most harm. The text exports behave in the opposite way. The fsid never changes; monitor addresses change rarely and visibly; a CRUSH map and a pool list are human-readable and can be compared against a recovered cluster or re-applied deliberately. That asymmetry is why the export schedule matters more than the backup schedule, and why the pool, CRUSH, and auth records are the artefacts that turn post-surgery verification into a diff rather than a guess.

Quiz

Knowledge check · 4 questions

Q1. Why must the client.admin keyring be stored outside the cluster?

It is regenerated on every monitor restart, so any copy held on a monitor host is already stale by the time it is needed It is too large to keep on a monitor host once the auth database has grown beyond a few thousand entities The dashboard requires an external copy in order to authenticate to the cluster during a quorum outage It is issued by the monitors, so the failure that takes quorum away also takes away the command that would produce it
Q2. Text exports such as the CRUSH map and pool list are more durably useful during a quorum recovery than a monitor store backup.

True False
Q3. Design a quorum-loss drill programme.

A team has a documented quorum recovery runbook that has never been executed. Their production cluster has three monitors; a staging cluster exists with a tenth of the data.
Q4. What does timing a single monitor restart in production actually tell you?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Schedule the exports and the offline readback together — an export nobody has read is a hope, not an artefact — and keep them off the cluster they describe. Rewrite any quorum runbook so its first commands are host-local, and measure a single monitor restart quarterly so the maintenance window you assume matches the one you have.

Cross-course references

Kubernetes: an etcd snapshot never restored is not a backup, and restore time is the number that matters
Linux: recovery media stored on the system it recovers has failed before it is needed