Cluster maps in detail — what each one holds and who reads it

Intermediate⏱ ~15 mincephcrushtool

What you'll learn

Enumerate the contents of each map component
Identify which daemons and clients consume each map
Extract and inspect maps with the appropriate commands
Use map contents to answer operational questions

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Dashboards summarise. Maps state. When a summary and reality disagree — or when the summary does not contain the field you need — going to the map is the answer.

monmap

ceph mon dump

Contains: monitor names, ranks, addresses (v1 and v2), the fsid, and the monmap epoch. Consumed by everything, because every participant must know how to reach a monitor.

Changes only when monitors are added, removed, or readdressed — perhaps a handful of times in a cluster’s life. A monmap epoch that changes unexpectedly is worth investigating.

osdmap

ceph osd dump
ceph osd tree
ceph osd getcrushmap -o /tmp/cm.bin

The largest and busiest map. Contains: every OSD with its state (up/ down, in/out), CRUSH weight, reweight, and addresses; every pool with its full configuration; the CRUSH map; pool snapshots; and cluster flags such as noout.

Consumed by every OSD and every client. Changes on any OSD state change, any pool change, and any CRUSH change — which is why its epoch is the useful churn indicator.

pgmap

ceph pg dump
ceph pg dump_stuck
ceph pg stat

Contains per-PG state, acting and up sets, object counts, and statistics. Changes constantly. Consumed primarily by the manager for reporting; OSDs know their own PGs without needing the global view.

ceph pg dump_stuck is the practical entry point — it filters to PGs that are inactive, unclean, or stale rather than dumping everything.

mdsmap

ceph fs dump
ceph fs status

Contains: filesystems, their ranks, which MDS holds each rank, standby daemons, and per-filesystem settings. Consumed by CephFS clients and MDS daemons. Changes on MDS failover and rank changes.

Read the map when the summary looks wrong

ceph status reports “96 osds: 96 up, 96 in”. ceph osd dump tells you their weights, their reweights, their addresses, when each was last marked up, and which flags are set.

The commands to reach for when a summary is not enough:

ceph osd dump | grep '^osd\.'      # per-OSD state and weights
ceph osd dump | grep '^pool'        # every pool setting
ceph osd dump | grep flags          # cluster-wide flags
ceph pg dump_stuck                  # PGs needing attention

A cluster flag such as noout left set months ago shows in ceph status as a warning, but the full flag list only appears in the osdmap.

Why incremental maps exist and when a full map is sent

Sending a full osdmap to every participant on every change would be prohibitive on a large cluster — the map can be megabytes and the recipients number in thousands.

Monitors therefore compute incremental maps: the delta between consecutive epochs. A daemon or client a few epochs behind receives only the deltas and applies them in order.

If a participant is too far behind — beyond what the monitors have retained, bounded by mon_min_osdmap_epochs — it receives a full map instead. This is one reason an OSD that has been down for a long time is expensive to bring back: it needs a full map, then full peering, then likely backfill rather than recovery.

It is also why monitors retain map history at all, and why that history cannot be trimmed while PGs are non-clean: a PG that is not clean may still need an older epoch to establish authoritative history.

Extracting maps for offline work

ceph osd getmap -o /tmp/osdmap.bin
osdmaptool /tmp/osdmap.bin --print
osdmaptool /tmp/osdmap.bin --test-map-pgs-dump --pool 7

osdmaptool can simulate placement against a saved map, which is how you answer “what would happen if we changed this” without touching the cluster. Together with crushtool --test, it is the offline analysis toolkit.

Quiz

Knowledge check · 4 questions

Q1. Which map contains the CRUSH map, and why does that placement matter?

The monmap, so the monitors can compute placement authoritatively before publishing it The pgmap, since placement groups are what CRUSH maps onto OSDs and the pgmap tracks them A separate crushmap with its own epoch, distributed independently of the other maps The osdmap, so CRUSH changes increment its epoch and propagate to every participant
Q2. An OSD that has been down long enough falls outside the retained epoch history and must be sent a full osdmap rather than a set of deltas.

True False
Q3. A cluster shows unexpectedly slow recovery. ceph status shows no obvious problem beyond the degraded PGs. Use the maps to investigate.

A single OSD failed six hours ago on a 96-OSD cluster. Backfill is progressing at a fraction of the expected rate. ceph status shows the degraded PG count falling very slowly and reports HEALTH_WARN for the degraded PGs only. Devices show low utilisation and the network is not saturated.
Q4. Name the offline tools for analysing maps and explain what question each answers.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Go to the map when the summary is insufficient — ceph osd dump holds per-OSD weights, complete pool settings, and the full cluster flag list that ceph status only hints at. Use ceph pg dump_stuck rather than a full PG dump when hunting problems. And validate every topology change offline first with crushtool --test and osdmaptool --test-map-pgs-dump, since both answer “what would happen” without touching the cluster.

Cross-course references

Ceph: Part LIV (ceph status and health detail) for reading summaries well.
Ceph: Part XV (CRUSH Maps and Rules) for editing with crushtool.
Ceph: Part XXII (PG Investigation) for pg dump in anger.