The daemons — mon, mgr, osd, mds, rgw and what each one owns

Foundation⏱ ~15 mincephcephadm

What you'll learn

State the responsibility of each Ceph daemon type
Determine how many of each daemon a cluster needs
Predict cluster behaviour when each daemon type fails
Locate and read the logs for each daemon under cephadm

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Every Ceph incident begins with identifying which daemon is misbehaving. Knowing what each one owns turns that from a search into a deduction.

The five types

MON — monitors. Maintain the cluster map and reach consensus on changes. Three or five, always odd. Essential: without quorum nothing works.

MGR — managers. Provide metrics, the dashboard, the balancer, and the orchestrator. Active/standby, two is normal. Not in the data path — the cluster serves I/O with no MGR at all, but you lose visibility and orchestration.

OSD — object storage daemons. One per storage device (or several per fast NVMe). Hold the data, participate in acting sets, perform recovery and scrubbing. The bulk of the cluster.

MDS — metadata servers. Only for CephFS. Serve directory structure, file metadata, and capabilities to clients. Active plus standby per filesystem rank.

RGW — RADOS gateways. Only for object storage. Stateless HTTP daemons presenting S3 and Swift. Scale by adding more behind a load balancer.

Failure consequences

Daemon	One fails	Enough fail
MON	quorum continues on the rest	no quorum: cluster halts
MGR	standby takes over in seconds	no metrics, dashboard, or orchestration; I/O unaffected
OSD	PGs degrade, recovery starts	PGs below min_size: writes block
MDS	standby takes over; brief pause	CephFS unavailable; RBD and RGW unaffected
RGW	load balancer routes elsewhere	S3 unavailable; RBD and CephFS unaffected

The pattern worth internalising: MON failures are cluster-wide, OSD failures are pool-scoped, and MDS and RGW failures are service-scoped.

Under cephadm

All daemons run as containers managed by systemd, named by cluster FSID and daemon id:

FSID=$(ceph fsid)

ceph orch ps
ceph orch ps --daemon-type osd
systemctl status "ceph-$FSID@osd.12.service"
cephadm logs --name osd.12
ceph orch daemon restart mgr.host1.abcdef

ceph orch ps is the inventory: what is running, where, which version, and how long it has been up. It is the first command when a daemon is suspected of being absent rather than misbehaving.

Why OSDs are the only stateful daemon that cannot simply be replaced

Monitors hold state, but that state is the cluster map, which is small and replicated across every monitor — a lost monitor is rebuilt by adding a new one and letting it sync.

MGR, MDS, and RGW hold no durable state of their own. MDS metadata lives in a RADOS pool, RGW state lives in RADOS pools, and MGR state is derived. Any of them can be destroyed and recreated with no data consequence.

OSDs are different: an OSD owns the only copy of its share of the data on its device. Destroying one means the cluster must reconstruct that share from replicas, which takes time and consumes I/O. This asymmetry is why OSD operations are always the careful ones — ceph osd out before ceph osd purge, one at a time, waiting for active+clean — while restarting a MGR or RGW is routine.

Sizing

MON: 3 for most clusters, 5 for large or multi-domain ones. Odd.
MGR: 2. One active, one standby.
OSD: one per device, or 2-4 per high-end NVMe.
MDS: one active per rank plus at least one standby, per filesystem.
RGW: at least 2 behind a load balancer, then scale with request rate.

Quiz

Knowledge check · 4 questions

Q1. The active MGR fails and there is no standby. What is the immediate effect on client I/O?

None — client I/O is unaffected, but metrics, dashboard, and orchestration are lost All client I/O stops, because OSDs obtain their placement group statistics from the MGR Writes continue, but reads fall back to stale data because no MGR is refreshing the pgmap The cluster drops to read-only as a protective measure until a manager is available again
Q2. MDS, MGR, and RGW daemons can be destroyed and recreated with no data consequence, while OSDs cannot.

True False
Q3. ceph status has been unavailable for twenty minutes and the dashboard is down, but a monitoring alert says VM disk I/O is normal. Determine what has failed and what to do.

Cluster with 3 MONs, 1 MGR, 96 OSDs, RGW and CephFS both in use. ceph status from a client host hangs or returns very sparse output. The dashboard URL does not respond. Independent monitoring shows VM disk latency normal and S3 requests succeeding. The single MGR host was rebooted for patching twenty minutes ago and has not come back.
Q4. Classify the blast radius of a failure in each Ceph daemon type.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Run two MGRs so the failure that blinds your monitoring is a seconds-long failover, and alert on the absence of an active MGR because that failure removes the alerting path itself. Keep monitor counts odd and sized to the topology. Treat OSD operations as the careful ones — out, wait for active+clean, then purge, one at a time — since OSDs are the only daemon that owns data, while MGR, MDS, and RGW can be destroyed and recreated freely.

Cross-course references

Ceph: Part X (Manager Daemons) for MGR modules and failover.
Ceph: Part XLI (Metadata Servers) for MDS ranks and standbys.
Ceph: Part XLIX (cephadm) for how the daemons are deployed and managed.