Backing up and restoring the monitor store

Advanced⏱ ~18 mincephceph-monstore-toolmonmaptool

What you'll learn

Take a consistent copy of a monitor store
Verify a backup is readable before you need it
Explain what a stale store does when restored
Decide when a restore is the wrong recovery

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A monitor store backup is stale within seconds of being taken, and the one case it genuinely rescues is the case where every monitor is gone at once.

What the store is and where it lives

FSID=$(ceph fsid)
sudo du -sh /var/lib/ceph/$FSID/mon.$(hostname -s)/store.db
sudo ls /var/lib/ceph/$FSID/mon.$(hostname -s)/store.db | head

000412.sst  000415.sst  000418.log  CURRENT  IDENTITY  LOCK  MANIFEST-000410

Content	Change rate
osdmap history back to the oldest epoch any OSD needs	continuous
Current monmap, crushmap, mdsmap, mgrmap	on change
The `auth` database — every cephx key	rare
The config-key store and the central config database	rare
Paxos state and pending proposals	continuous

Taking a consistent copy

RocksDB rewrites and deletes files during compaction, so a copy taken while the daemon runs can reference an SST that no longer exists. Stop the monitor.

ceph mon ok-to-stop ceph-b
ceph orch daemon stop mon.ceph-b

FSID=$(ceph fsid)
SRC=/var/lib/ceph/$FSID/mon.ceph-b
sudo tar -C "$SRC" -czf /secure/mon-ceph-b-$(date +%F).tar.gz store.db keyring

# or a compacted, self-consistent copy through RocksDB itself
sudo ceph-monstore-tool "$SRC/store.db" store-copy /secure/mon-store-copy

ceph orch daemon start mon.ceph-b
ceph quorum_status --format json | python3 -c '
import sys,json; print("quorum:", ",".join(json.load(sys.stdin)["quorum_names"]))'

Verifying the backup is readable

A backup nobody has opened is a guess. Check it on a spare host, not on a monitor.

sudo ceph-monstore-tool /secure/mon-store-copy get-monmap -- --out /tmp/m
monmaptool --print /tmp/m

sudo ceph-monstore-tool /secure/mon-store-copy dump-keys | wc -l
sudo ceph-monstore-tool /secure/mon-store-copy get-osdmap -- --out /tmp/osdmap
osdmaptool --print /tmp/osdmap | head -5

When restoring is the wrong answer

Situation	Correct action
One monitor lost, quorum intact	rebuild empty and let it sync — never restore
One store corrupt, quorum intact	remove the monitor, `--mkfs`, re-add
All monitors lost, OSDs intact	rebuild the store from the OSDs
All monitors lost, no OSDs	the backup is all you have, and it is stale

Restoring a week-old store into a cluster that still has quorum
accomplishes nothing: the monitor discards it and synchronises the
current state from its peers.

# keep the parts that do not go stale, separately and often
ceph auth export        > /secure/ceph-auth-$(date +%F).keyring
ceph config-key dump    > /secure/ceph-configkey-$(date +%F).json
ceph mon getmap -o        /secure/monmap-$(date +%F)
ceph osd getcrushmap -o   /secure/crushmap-$(date +%F)

Quiz

Knowledge check · 4 questions

Q1. What part of a monitor store is genuinely worth backing up on a schedule?

The osdmap history, because it is the largest component and the most expensive to rebuild from the OSDs The Paxos state, because it records the proposals that were accepted but not yet committed when the daemon stopped The RocksDB MANIFEST, because it names every SST file and can rebuild the rest of a damaged store The auth database, config-key store, and central config, because they change only when an operator changes them
Q2. A monitor store copied with rsync while the daemon is running is a valid backup.

True False
Q3. Decide how to recover a monitor with a corrupt store.

`mon.ceph-b` crashes on start with a RocksDB corruption error. The other two monitors hold quorum and the cluster is serving normally. A four-day-old store backup exists.
Q4. Why does restoring a stale monitor store into a cluster that still has quorum accomplish nothing?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Back up ceph auth export, ceph config-key dump, ceph config dump, and the monmap on a schedule — they are small, they do not go stale, and they are what a total monitor loss actually needs. Rebuild rather than restore whenever quorum survives; the restored store is discarded and resynchronised regardless.

Cross-course references

Kubernetes: an etcd snapshot restored into a live quorum is likewise overwritten, not honoured
Linux: lazy validation in embedded databases moves the failure from backup time to restore time