Ceph architecture: RADOS, MON, MGR, OSD, MDS

Why this matters in production

Ceph is the most operationally complex piece of a Proxmox cluster. Misunderstanding the primitives produces clusters that look healthy but degrade under failure. This lesson gives you the vocabulary for the rest of the Ceph lessons.

RADOS — the reliable autonomic distributed object store

Ceph’s lowest layer is RADOS. Everything else — RBD, CephFS, RGW — is built on RADOS.

RADOS provides:

• Objects stored in placement groups distributed across OSDs.
• Replication (or erasure coding) of each object.
• Self-healing when an OSD fails.
• CRUSH placement algorithm that decides where each object lives.

flowchart TB
  A[Client: rbd / cephfs] --> R[RADOS]
  R --> PG[Placement Groups]
  PG --> O[OSDs]
  O --> D[Physical disks]

The daemons

MON — the cluster brain

MONs maintain a master copy of the cluster map: which OSDs are up, what the cluster topology looks like, which pools exist. A cluster of MONs uses Paxos to maintain consensus. Quorum requires a majority of MONs.

flowchart LR
  M1[MON 1] --> M2[MON 2]
  M2 --> M3[MON 3]
  M3 --> M1

Losing 1 MON: cluster survives (2 of 3). Losing 2: cluster blocks (no quorum).

MGR — the orchestrator

The MGR exposes cluster metrics, handles PG autoscaling, runs ceph-volume management, and hosts pluggable modules. MGRs use an active-standby model: one is active, others are warm standbys. Losing all active MGRs blocks some operations (especially PG autoscaling and device health monitoring).

OSD — the storage

Each OSD is a daemon that manages a single disk (or partition). OSDs:

• Store objects in placement groups.
• Replicate objects to peer OSDs on writes.
• Detect failures of peer OSDs.
• Participate in recovery and rebalancing.

OSD resources:

MDS — the metadata server

Required only for CephFS. Tracks filesystem metadata (directory structure, file attributes). Multiple MDS daemons can run; typically one is active and others are standby.

Placement groups (PGs)

Each pool is split into PGs. Each PG is replicated across multiple OSDs (usually 3).

flowchart LR
  P[Pool 'vm-storage'] --> PG1[PG 0]
  P --> PG2[PG 1]
  P --> PG3[PG 2]
  PG1 --> O1[OSD 1]
  PG1 --> O2[OSD 2]
  PG1 --> O3[OSD 3]
  PG2 --> O1
  PG2 --> O2
  PG2 --> O3

PGs make data distribution tractable:

• Each object maps to a PG via a hash.
• The PG is placed on OSDs by CRUSH.
• CRUSH recomputes when the cluster changes.

CRUSH

CRUSH (Controlled Replication Under Scalable Hashing) decides which OSDs host each PG. It uses a map of buckets (hosts, racks, rooms) and rules (e.g. “replicate across hosts”).

flowchart TB
  root[root] --> dc1[datacentre]
  dc1 --> rack1[rack 1]
  dc1 --> rack2[rack 2]
  rack1 --> h1[host 1]
  rack1 --> h2[host 2]
  rack2 --> h3[host 3]
  rack2 --> h4[host 4]
  h1 --> osd1[OSD]
  h1 --> osd2[OSD]
  h2 --> osd3[OSD]
  h2 --> osd4[OSD]

CRUSH rules can require:

• Replicas on different hosts (default).
• Replicas on different racks.
• Replicas on specific device classes (e.g. all-NVMe).
• Erasure coding profiles.

The failure domain is one word in the rule, and everything depends on it

A CRUSH rule contains a chooseleaf step naming a bucket type. That one word decides what “three copies” protects you against.

step chooseleaf firstn 0 type host   # three copies on three hosts
step chooseleaf firstn 0 type osd    # three copies on any three OSDs

With type host, CRUSH will not place two copies of a placement group on the same node, so losing a node costs each PG exactly one copy. With type osd, nothing stops two or three copies landing on the same node — and losing that node takes those PGs to one copy or to zero.

ceph osd crush rule dump | grep -E '"rule_name"|"type"'
ceph osd pool ls detail | grep -E 'pool|crush_rule'
ceph osd tree

size, min_size and the failure domain are one decision

The three settings only make sense read together, and the Ceph pools lesson works through the combinations. The summary that matters here:

A cluster with all three is one where losing a host is a health warning. A cluster missing any one of them is a cluster where losing a host is an incident, and which one is missing determines whether that incident is an outage or a silent loss of redundancy.

Replication vs erasure coding

For VM storage, replication is the standard. Erasure coding is more common for archive/cold storage.

HEALTH_OK is necessary, not sufficient

A cluster can report HEALTH_OK and still have problems:

• Replication factor fine, but PG count too low (recovery is slow).
• Capacity fine, but nearfull_ratio reached on individual OSDs.
• Performance fine, but network saturated.
• Storage fine, but CRUSH rule leaves replicas on the same host.
• Replication fine, but min_size is 1, so a degraded pool will accept writes onto a single copy without saying so.

CLI walkthrough

ceph status

ceph osd tree

ceph osd pool stats && ceph df

ceph pg stat && ceph pg dump_stuck unclean

Production considerations

Common mistakes

• Running with only 2 MONs. Loses quorum if 1 MON dies.
• Setting min_size=1 on a replicated pool.
• Leaving a CRUSH rule at type osd after the cluster grew past one node. The replication arithmetic silently stops meaning what it says.
• Trusting HEALTH_OK without checking capacity and PG distribution.
• Co-locating all MONs on one host.

Key takeaways

• RADOS is the storage layer; MON/MGR/OSD/MDS are daemons.
• CRUSH decides object placement, and the chooseleaf bucket type in the rule decides what a copy is protected against.
• size, min_size and the failure domain are one decision. Three copies on a rule that permits them to share a host is not three-node protection.
• PGs are how data is distributed; tune them carefully.
• HEALTH_OK is necessary but not sufficient.

Knowledge check