What the monitors actually do

Foundation⏱ ~15 minceph

What you'll learn

State the monitor responsibilities precisely
Explain why monitors are not in the data path
Describe what a monitor does when an OSD failure is reported
Size monitor hardware appropriately

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Monitors are the smallest daemons in a Ceph cluster by resource consumption and the most consequential by failure impact. Knowing exactly what they do — and what they conspicuously do not do — makes both sizing and incident response straightforward.

The responsibilities

Maintain the cluster map. monmap, osdmap (including CRUSH), pgmap summary, mdsmap, and the fsid.
Reach consensus on changes. Any map change is committed through Paxos and requires a majority.
Serve maps. To clients on connect, to daemons continuously, and incrementally where possible.
Authenticate. cephx keys are held in the monitor store, and monitors issue the tickets clients use.
Adjudicate failure reports. OSDs report peers; monitors decide.

That last point is worth emphasising. An OSD cannot mark another OSD down. It can only report, and the monitors apply mon_osd_min_down_reporters and decide.

What monitors do not do

They do not store object data. Not one byte.
They are not in the read or write path. A client contacts monitors once to get a map and then talks only to OSDs.
They do not compute placement. Clients do that.
They do not perform recovery. OSDs do.

flowchart LR
  C[Client] -->|"1. get map (once)"| M[Monitors]
  C -->|"2. all I/O"| O[OSDs]
  O -->|"failure reports, map updates"| M
  M -->|"new map epochs"| O

Handling a failure report

osd.14 → mon: "osd.7 failed"
osd.22 → mon: "osd.7 failed"
mon: reporters >= mon_osd_min_down_reporters (2)?  yes
mon: propose osdmap change marking osd.7 down
mon: Paxos commit (majority agrees)
mon: publish osdmap epoch N+1

The monitors do not probe osd.7 themselves. They act on reports, weighted by the requirement for distinct reporters, which is what distinguishes “osd.7 is down” from “osd.14 cannot reach anything”.

Why monitor load rises with cluster size but not with data volume

A monitor’s work is proportional to the rate of map changes and the number of participants that must receive them — not to how many bytes the cluster holds.

A petabyte archive with 100 OSDs and a stable topology generates very little monitor load. A 500-OSD cluster with frequent OSD restarts, active balancing, and thousands of clients generates a great deal, because every change must be committed and distributed.

This is why monitor sizing follows OSD count and client count rather than capacity, and why the same three monitors that comfortably serve a 100 TB cluster may need to become five with faster disks on a 1,000-OSD cluster. It is also why an incident that causes OSD flapping loads the monitors hardest at the moment they are most needed.

Sizing

Cluster	Monitors	Per-monitor
under 100 OSDs	3	4 cores, 8-16 GB RAM, SSD
100-500 OSDs	3-5	8 cores, 16-32 GB RAM, NVMe
over 500 OSDs	5	8+ cores, 32 GB+ RAM, NVMe

Colocating monitors with OSDs is common and acceptable on small clusters, provided the monitor store is on a separate fast device. On larger clusters, dedicated monitor hosts avoid the case where a busy OSD host slows the cluster’s decision-making.

Checking monitor health

# MON_ID: substitute a monitor name from the ceph mon stat output below
MON_ID=ceph-mon-01
FSID=$(ceph fsid)

ceph mon stat
ceph quorum_status --format json-pretty
ceph daemon "mon.$MON_ID" perf dump | grep -A5 rocksdb
ceph health detail
du -sh "/var/lib/ceph/$FSID/mon.$MON_ID/"

Quiz

Knowledge check · 4 questions

Q1. An OSD detects that a peer is not responding to heartbeats. What happens next?

It marks the peer down in its own copy of the osdmap and informs the monitors afterwards It asks the monitors to probe the suspect peer directly and waits for their verdict It reports the peer to the monitors, which decide based on how many distinct OSDs report it It begins recovery for the affected placement groups immediately, before any map change
Q2. Because a monitor store is typically only a few gigabytes, monitor storage performance is not an important sizing consideration.

True False
Q3. A 400-OSD cluster has three monitors colocated on OSD hosts, with their stores on the same spinning disks as OSD data. Map changes are slow and failure detection lags. Advise.

400 HDD OSDs across 20 hosts. Three monitors on three of those hosts, monitor stores on partitions of the same spinning disks used for OSD data. Symptoms: ceph status takes several seconds, OSD down events take noticeably longer than the configured grace to appear, and the monitor perf dump shows high RocksDB commit latency. The cluster is otherwise healthy.
Q4. Explain why monitor load scales with OSD and client count rather than with stored capacity.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Put every monitor store on a dedicated SSD or NVMe, never on a spindle and never on a device shared with OSD data — monitor commit latency is cluster decision latency. Size the partition for the bad day, 50 to 100 GB, because map history cannot be trimmed while PGs are non-clean. Size the monitor count and hardware against OSD and client counts rather than capacity, and move to dedicated monitor hosts once OSD host load can plausibly interfere with consensus.

Cross-course references

Ceph: Part IX (Monitor Quorum) for the availability model.
Ceph: Part VI (Ceph Architecture) for where monitors sit in the write path.
Ceph: Part CX (Monitor Recovery) for restoring a lost monitor.