The monitor store — RocksDB, growth, and backups

Advanced⏱ ~16 mincephceph-monstore-tool

What you'll learn

Describe what the monitor store contains and how it is organised
Explain why the store grows and when it can be trimmed
Back up and restore a monitor store
Respond to monitor disk pressure without endangering the cluster

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Every byte of object data can be intact while a cluster is unusable because its monitors are gone. The monitor store is the cluster’s identity and topology, it is small, and it is the single most worthwhile thing to back up.

What it contains

/var/lib/ceph/<fsid>/mon.<id>/store.db/

A RocksDB database holding:

the monmap and its history,
osdmap epochs (full and incremental),
the CRUSH map inside those osdmaps,
mdsmap and fsmap state,
cephx authentication keys and capabilities,
MGR module configuration and the config store (ceph config),
Paxos state.

The cephx keys are worth noting. Losing every monitor means losing every key, which means every client and daemon must be re-keyed even if the maps could somehow be reconstructed.

Why it grows

Monitors retain osdmap history so lagging participants can catch up incrementally. Trimming is bounded by two things:

mon_min_osdmap_epochs (500 by default) — always keep this many.
Whether the cluster is in a state where older epochs are safe to discard, which requires PGs to be clean.

The second is the operationally important one:

Monitor stores grow during long incidents, exactly when you least want it

While PGs are non-clean, monitors cannot trim old osdmaps, because a PG that is not clean may still need an older epoch to establish authoritative history.

So an incident that lasts many hours grows the monitor store steadily. On a cluster with a small monitor partition, the result is MON_DISK_LOW or a full filesystem in the middle of an already difficult situation — and a monitor that cannot write is a monitor that drops out of quorum.

The response is to finish recovery, not to delete anything. Never remove files from store.db by hand: it is a RocksDB database, and partial deletion corrupts it. If space is genuinely critical, extend the volume, or stop one monitor to halt its growth provided the remaining two hold quorum.

Backing it up

# stop the monitor first for a consistent copy
# Cluster FSID from `ceph fsid`; substitute your own:
FSID=3e0b2c14-9f3a-4d21-8a77-1c9f0e2b5d64

ceph orch daemon stop mon.ceph-01
tar czf "/backup/mon-ceph-01-$(date +%F).tar.gz" \
    "/var/lib/ceph/$FSID/mon.ceph-01/"
ceph orch daemon start mon.ceph-01
ceph quorum_status

Stopping one monitor of three is safe — the other two hold quorum. Do it on a schedule, keep the archives off the cluster, and record the fsid alongside them.

For an online copy, ceph-monstore-tool can dump the store without stopping the daemon, though a stopped-daemon copy is simpler and more obviously consistent.

Rebuilding

If every monitor is lost but OSDs survive, the store can be reconstructed from the OSDs:

# on each OSD host
# OSD_ID is the numeric id from `ceph osd ls`; substitute your own:
OSD_ID=12

ceph-objectstore-tool --data-path "/var/lib/ceph/osd/ceph-$OSD_ID" \
    --op update-mon-db --mon-store-path /tmp/mon-store
# then merge, rebuild auth, and start a monitor
ceph-monstore-tool /tmp/mon-store rebuild

This works and it is slow, error-prone, and requires reconstructing cephx keys separately. It is a genuine last resort, and its existence is not a reason to skip backups.

Why the config store lives in the monitors and what that implies

Since the Mimic release, ceph config set writes to a configuration database inside the monitor store rather than to ceph.conf files on each host. This is a substantial improvement: settings apply cluster- wide, take effect at runtime, and are visible with ceph config dump.

The implication for backups is that the monitor store now holds operational configuration as well as maps and keys. A cluster restored from an old monitor backup comes back with old settings — recovery throttles, memory targets, and module configuration as they were at backup time.

That is usually acceptable and occasionally surprising. Exporting ceph config dump alongside the store backup costs nothing and makes the configuration state explicit and diffable, which is worth doing regardless.

The routine

Back up one monitor store weekly, stopped-daemon, archived off cluster.
Export ceph config dump and ceph osd getcrushmap at the same time.
Record the fsid in the same location.
Verify quorum after every backup run.

Quiz

Knowledge check · 4 questions

Q1. Why does a monitor store grow during a long incident with many non-clean PGs?

Because each degraded PG writes state into the monitor store Because old osdmap epochs cannot be trimmed while PGs are non-clean Because monitors buffer client requests that cannot be served Because Paxos retains failed proposals until the cluster is healthy
Q2. Losing every monitor while all OSDs survive means the cluster data is lost.

True False
Q3. MON_DISK_LOW appears during hour nine of a recovery. One monitor partition is at 92% full. Decide what to do.

Three monitors, each with a 30 GB partition for /var/lib/ceph. Recovery from a two-host failure is still running with roughly 600 PGs non-clean. mon.a is at 92%, mon.b at 89%, mon.c at 90%. Recovery is progressing but slowly, with an estimated four hours remaining. No monitor has dropped out of quorum yet.
Q4. List what a monitor store contains and explain why a backup should include ceph config dump as well.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Back up one monitor store on a schedule with the daemon stopped — two of three hold quorum, so the operation is safe — and archive it off the cluster alongside ceph config dump, the CRUSH map, and the fsid. During a long incident, expect the store to grow and plan for it rather than react: finishing recovery is what allows trimming, and manual deletion inside store.db turns a space problem into a rebuild. Size monitor partitions for the bad day, not the good one.

Cross-course references

Ceph: Part CX (Monitor Recovery) for the rebuild procedure.
Ceph: Part CV (Backup Strategy) for where this fits in the wider plan.
Ceph: Part XXXI (Ceph Authentication) for the cephx keys held here.