Ceph architecture overview — the path a write takes

Why this matters in production

Almost every Ceph troubleshooting question reduces to “which step in the write path is slow or broken?” Learning the path once makes the rest of the course — and most incidents — a matter of localisation rather than guesswork.

The path

flowchart TD
  A[Application: fsync on a file] --> B[Guest filesystem]
  B --> C[librbd: image → RADOS objects]
  C --> D[librados: get cluster map from MON]
  D --> E[CRUSH: object name → PG → acting set]
  E --> F[Primary OSD]
  F --> G[Replica OSDs]
  F --> H[BlueStore: RocksDB metadata + raw block data]
  G --> I[BlueStore on each replica]
  I --> F
  H --> F
  F --> C
  C --> A

Step by step:

1. Application writes and calls fsync. The guest filesystem converts that into block writes at specific offsets.
2. librbd maps the image to objects. An RBD image is a sequence of RADOS objects, 4 MiB by default. A write at offset 9 MiB lands in object rbd_data.<id>.0000000000000002.
3. librados needs a cluster map. It fetches it from a monitor once and caches it, refreshing when an OSD tells it the epoch is stale.
4. CRUSH computes placement. The object name is hashed to a PG id; CRUSH maps the PG to an ordered list of OSDs. First in the list is the primary.
5. The client sends to the primary only. Never to replicas.
6. The primary replicates and waits. It forwards to the other OSDs in the acting set and waits for them to commit.
7. Each OSD commits through BlueStore, writing metadata to RocksDB and data to the raw block device.
8. The primary acknowledges once the acting set has committed.

Which component is authoritative where

Reading a symptom against this table is usually enough to decide where to look next.

Using the path to localise

A worked example. Writes to one RBD image are slow; others are fine.

1. Which objects? rbd info gives the image prefix.
2. Which PGs and OSDs? ceph osd map <pool> <object> for several objects from the image.
3. Are those OSDs outliers? ceph osd perf.
4. Is one of them on a device with problems? smartctl, ceph device get-health-metrics.

Four commands, following the path downwards, each one narrowing the search. That method generalises to nearly every Ceph problem, and it is the reason this lesson comes before the troubleshooting parts rather than after them.

Quiz

Production discipline

Learn the path once and use it as the diagnostic spine: symptom, then which layer owns that behaviour, then the command that inspects that layer. ceph osd map is the entry point for any object-level question precisely because placement is computed rather than recorded. And remember that the primary waits for the entire acting set, so client latency is set by its slowest member — which makes outlier hunting, not aggregate throughput, the correct first measurement.

Cross-course references

• Ceph: Part VII (RADOS) for objects, pools, and placement in depth.
• Ceph: Part XII (BlueStore) for the bottom of the path.
• Ceph: Part LXVII (Performance Methodology) for the full localisation method.