Distributed state — where the truth lives

Foundation⏱ ~14 min

What you'll learn

Explain why distributed state requires a designated source of truth
Describe what the Ceph cluster map contains and who maintains it
Distinguish authoritative state from cached state in Ceph
Predict client behaviour when its cached map is stale

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A single server has one copy of the truth: what is in its memory and on its disk. A distributed system has many copies, and they will disagree — not because of bugs, but because messages take time and nodes fail at inconvenient moments.

Every distributed system therefore has to designate where the authoritative answer lives. Ceph’s answer is the cluster map, maintained by the monitors, and almost everything that surprises new Ceph operators follows from that one decision.

What the cluster map is

The cluster map is a small set of versioned documents:

monmap    which monitors exist and how to reach them
osdmap    which OSDs exist, their state, weights, and the CRUSH map
pgmap     placement group states and statistics
mdsmap    CephFS metadata server ranks and state
fsid      the cluster's unique identity

Each has an epoch — a monotonically increasing version number. Epoch is how every participant knows whether its copy is current.

flowchart LR
  M[Monitors: authoritative map] -->|"map epoch 8412"| O[OSDs]
  M -->|"map epoch 8412"| C[Clients]
  O -->|"epoch check on every op"| O2[Peer OSDs]
  C -->|"computes placement locally"| O

Why clients compute rather than ask

A client that wanted to write an object could ask a central service where it goes. That service would then be a bottleneck and a single point of failure — the design Ceph exists to avoid.

Instead the client fetches the map once, then computes placement itself with CRUSH. No lookup, no metadata server in the data path, and the computation is deterministic: every participant with the same map epoch computes the same answer.

The consequence is that a client’s map is a cache, and caches go stale.

Authoritative versus cached

State	Authority	Cached by
Cluster map	monitors	OSDs, clients, MDS, RGW
Object data	primary OSD of the PG	client page cache, RBD cache
PG membership	computed from map	everyone, identically
Object existence	the acting set	nothing

Only monitors are authoritative about the map. Only the acting set is authoritative about the data. Everything else is a copy that may be behind.

Why the map must be small

Every OSD holds the full osdmap, and every client fetches it. On a 1,000-OSD cluster with 10,000 clients, distributing a map is a broadcast, so map size directly affects how expensive a topology change is.

This is why the map contains structure rather than content. It records which OSDs exist and how CRUSH should place data — not where any individual object lives. Object placement is computed from the map on demand, so the map stays roughly constant in size as data grows from terabytes to petabytes.

It is also why monitors ship incremental map updates rather than full maps wherever possible: a participant a few epochs behind receives only the deltas. A participant far behind gets a full map, which is one reason an OSD that has been down for a long time is expensive to bring back.

What this means operationally

Monitors are the most important daemons in the cluster, because without them there is no authoritative map and nothing can proceed.
Clients holding stale maps recover automatically; you do not need to restart them after topology changes.
A rapidly climbing map epoch is a symptom worth investigating even when health is OK.
The map is small enough to back up and worth backing up, because losing every monitor means losing the cluster’s identity and topology even though every byte of data survives.

Quiz

Knowledge check · 4 questions

Q1. Why do Ceph clients compute object placement themselves rather than asking a metadata service?

Because a central lookup service would be a bottleneck and a single point of failure Because a local CRUSH computation is faster than a round trip to any metadata service Because clients cache object locations and consult the monitors only when a lookup misses Because CRUSH returns a different placement per client, which spreads read load across replicas
Q2. A client copy of the cluster map is a cache, and an operation sent with a stale epoch is rejected rather than serviced against the wrong OSD.

True False
Q3. Clients report intermittent slowness. Disk and network metrics are normal, but ceph osd stat shows the map epoch climbing by hundreds per hour. Investigate.

40-OSD cluster, HEALTH_OK at the moment of checking, though the health history shows repeated brief OSD_DOWN warnings. Client latency spikes correlate with no visible disk or network saturation. ceph osd stat epoch increased from 41,200 to 41,900 in the last hour. Normal rate on this cluster is a few epochs per hour.
Q4. Explain why the Ceph cluster map stays roughly constant in size as stored data grows from terabytes to petabytes.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Treat the monitors as the most critical daemons in the cluster, since without an authoritative map nothing else can proceed regardless of how healthy the OSDs are. Back the monitor store up: every byte of data can survive while the cluster becomes unusable because its identity and topology are gone. And watch the map epoch rate as a first-class signal — a cluster committing hundreds of map changes an hour is telling you something no health check will.

Cross-course references

Ceph: Part VIII (Monitors) for how the map is maintained and served.
Ceph: Part XIII (CRUSH Fundamentals) for the placement computation itself.
Ceph: Part CX (Monitor Recovery) for what losing the map store involves.