Cluster map and cluster identity — FSID, monmap, osdmap

Intermediate⏱ ~15 mincephcrushtool

What you'll learn

Describe the contents and change rate of each map component
Explain the role of the FSID and the consequences of duplication
Read map epochs and use them as a diagnostic signal
Export and inspect maps for troubleshooting

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The cluster map is the shared truth. Reading it directly — rather than reading a summary of it — is how many diagnoses get made, and knowing which component changes often and which almost never turns epoch numbers into useful signal.

The components

Map	Contains	Changes when	Typical rate
`monmap`	monitor members, addresses, ranks	monitors added or removed	very rarely
`osdmap`	OSDs, states, weights, pools, CRUSH map	OSD state, pool config, CRUSH	frequently
`pgmap`	PG states and statistics	constantly	continuously
`mdsmap`	CephFS ranks, MDS states	MDS failover, rank changes	rarely
`fsid`	cluster identity	never	never

Inspecting them:

ceph mon dump              # monmap
ceph osd dump              # osdmap
ceph pg dump               # pgmap
ceph fs dump               # mdsmap
ceph fsid                  # cluster identity

Epochs as a diagnostic signal

Every map has an epoch that increments on each change:

ceph osd stat
# 96 osds: 96 up (since 3d), 96 in (since 3d); epoch: e41207

A stable cluster’s osdmap epoch increases slowly — a few per hour at most. Rapid growth means something is repeatedly changing: OSDs flapping, a balancer working hard, or pool settings being churned by automation. Each change forces every OSD and client to update, which costs latency for reasons that no disk metric explains.

Watch it over time rather than as a snapshot. The rate is the signal; the absolute number is meaningless.

The osdmap contains CRUSH

The osdmap is where the CRUSH map lives, which is why CRUSH changes increment the osdmap epoch and propagate to every participant:

ceph osd getcrushmap -o /tmp/crush.bin
crushtool -d /tmp/crush.bin -o /tmp/crush.txt
# edit
crushtool -c /tmp/crush.txt -o /tmp/crush.new
crushtool -i /tmp/crush.new --test --show-mappings --rule 0 --num-rep 3
ceph osd setcrushmap -i /tmp/crush.new

crushtool --test is the step people skip. It simulates placement against the proposed map without applying it, which is the only way to find out that a rule cannot be satisfied before the cluster tells you by leaving PGs undersized.

Why old osdmaps are kept, and why that can fill a monitor

Monitors retain a history of osdmap epochs so that a daemon or client that has been away can catch up incrementally rather than needing a full map. The retention is bounded by mon_min_osdmap_epochs (500 by default) and by whether all OSDs have acknowledged the newer maps.

The failure mode appears on clusters with PGs stuck non-clean for a long period: monitors cannot trim old maps because the cluster is not in a state where trimming is safe, so the monitor store grows. On a long incident this produces a monitor disk filling up in the middle of an already bad situation.

The signal is MON_DISK_LOW or a growing /var/lib/ceph/mon directory, and the resolution is to get the PGs clean rather than to delete anything by hand. It is a good reason to give monitor hosts generous, fast local storage.

Backing it up

The monmap and the monitor store are the cluster’s identity and topology. Losing all monitors with no backup means a cluster whose data is entirely intact and which cannot be assembled again without a lengthy reconstruction from OSDs. Back up the monitor store, and record the FSID somewhere outside the cluster.

Quiz

Knowledge check · 4 questions

Q1. The osdmap epoch on a 96-OSD cluster has risen from 41,000 to 44,000 in six hours. Health is OK. What does this indicate?

Normal operation — the osdmap epoch advances as placement groups report their statistics Something is repeatedly changing OSD or pool state, forcing map updates cluster-wide The monitors are struggling to commit and are re-publishing the same map repeatedly The balancer is working normally, and this is the epoch rate a healthy balancer produces
Q2. Running crushtool --test against a proposed CRUSH map will reveal a rule that cannot be satisfied before the map is applied.

True False
Q3. During a long incident with many PGs stuck non-clean, monitors begin reporting MON_DISK_LOW. Explain and respond.

Cluster has been in a degraded state for eleven hours after a host loss. Roughly 400 PGs remain non-clean while backfill proceeds slowly. Monitor hosts have 50 GB partitions for /var/lib/ceph. MON_DISK_LOW appeared two hours ago and the monitor store directory has grown steadily. Recovery is still progressing but slowly.
Q4. Explain what the FSID is, why duplicating it is unrecoverable, and how to prevent it.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Track the osdmap epoch rate as a standing signal — a cluster committing thousands of map changes in hours is telling you something no health check reports. Always run crushtool --test before applying a CRUSH map, so an unsatisfiable rule is a desk finding rather than a production one. Give monitor hosts generous fast local storage, since map history cannot be trimmed while PGs are non-clean and long incidents grow the store exactly when you can least afford it. And record the FSID outside the cluster alongside a monitor store backup.

Cross-course references

Ceph: Part VIII (Monitors) for how maps are agreed and served.
Ceph: Part XV (CRUSH Maps and Rules) for editing the CRUSH map safely.
Ceph: Part CX (Monitor Recovery) for rebuilding from a monitor store.