The storage layer cake — RBD, CephFS, and RGW on one RADOS

Foundation⏱ ~15 min

What you'll learn

Describe what each service layer adds on top of RADOS
Explain which properties are set by RADOS and cannot be changed by a service
Choose the appropriate service layer for a given workload
Attribute a symptom to the service layer or to RADOS beneath it

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Ceph is one storage system with three faces. Understanding which face you are looking at — and which properties come from underneath and cannot be changed by the face — is what stops an operator tuning RGW for a problem that lives in RADOS.

The layers

┌───────────────────────────────────────────────┐
│ Applications: VMs, POSIX clients, S3 clients  │
├───────────────┬─────────────┬─────────────────┤
│ RBD           │ CephFS      │ RGW             │
│ block device  │ POSIX FS    │ S3/Swift object │
│ librbd        │ MDS + client│ radosgw daemon  │
├───────────────┴─────────────┴─────────────────┤
│ librados                                      │
├───────────────────────────────────────────────┤
│ RADOS: MON + MGR + OSD, objects, pools, PGs   │
├───────────────────────────────────────────────┤
│ BlueStore on raw block devices                │
└───────────────────────────────────────────────┘

RBD presents a block device. It maps an image onto a sequence of fixed-size RADOS objects and adds snapshots, clones, layering, and mirroring. It adds no metadata daemon — everything is computed from the image name and offset.

CephFS presents a POSIX filesystem. It needs a metadata daemon (MDS) because POSIX requires directories, permissions, and rename semantics that objects do not provide. File data goes to a data pool as objects; file metadata goes to a metadata pool.

RGW presents S3 and Swift APIs. It runs as a daemon that maps buckets and objects onto RADOS objects, and it maintains bucket indexes, user accounts, and multi-site replication state.

What RADOS sets and services cannot change

Property	Set by	A service can…
Durability (`size`, EC)	pool config	choose which pool, not change the guarantee
Placement	CRUSH rule	choose the rule per pool
Consistency model	RADOS	not weaken it (RGW multi-site is between zones)
Write latency floor	acting set commits	add caching above, not remove the floor
Capacity	cluster	not exceed it

This is the table to consult when someone asks whether a service-level setting can fix a problem. If the property lives in RADOS, the answer is no.

Attribute the symptom to a layer before tuning anything

A slow S3 PUT can be an RGW problem (bucket index contention, too few shards, RGW daemon CPU) or a RADOS problem (slow OSD, full cluster, degraded PGs). The commands differ completely.

The cheapest discriminator is to test RADOS directly, bypassing the service:

# DATA_POOL is the pool beneath the slow service; for RGW, its data pool:
DATA_POOL=default.rgw.buckets.data

rados -p "$DATA_POOL" bench 10 write --no-cleanup
rados -p "$DATA_POOL" bench 10 seq
rados -p "$DATA_POOL" cleanup

If rados bench is healthy and S3 is slow, the problem is in RGW. If rados bench is slow too, the problem is beneath it and RGW tuning will not help. Five minutes of this saves hours.

Choosing a layer

RBD for anything that wants a disk: VMs, containers with block PVCs, databases. One writer per image unless the filesystem above is cluster-aware.
CephFS for shared POSIX access from many clients: home directories, render farms, shared application data. Costs an MDS and its metadata pool.
RGW for applications that speak S3: backups, artefact stores, data lakes, anything with an existing S3 client.

The mistake worth naming is using CephFS where RBD would do, because a single VM needs a disk rather than shared access, and CephFS adds a metadata daemon, a metadata pool, and a whole class of failure modes for no benefit.

Diagnostic shortcut

For any symptom, ask: does it affect one service or all of them? One service points at that service’s daemon or metadata. All services point at RADOS — capacity, PG health, OSD latency, or the network.

Quiz

Knowledge check · 4 questions

Q1. S3 PUT latency is poor. What is the cheapest test to determine whether the problem is in RGW or beneath it?

Increase RGW daemon count and observe whether latency improves Check the RGW logs for slow request entries and correlate with bucket names Reshard the bucket index and re-measure Run rados bench against the RGW data pool to test RADOS directly
Q2. Because RBD, CephFS, and RGW are separate services, heavy load on one is naturally isolated from the others.

True False
Q3. A team plans to store VM disks on CephFS so that all hypervisors can see the same files. Assess the design.

Six hypervisors running 80 VMs. The proposal is a single CephFS mounted on every hypervisor, with each VM disk stored as a qcow2 file. The stated reason is that live migration needs shared storage. The cluster currently has no MDS deployed. The alternative under consideration is RBD with one image per VM.
Q4. Name three properties that RADOS sets and no service layer can change, and explain the diagnostic use of that fact.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Attribute every symptom to a layer before tuning: rados bench against the service’s data pool separates a gateway or client problem from a RADOS one in five minutes and routinely saves hours. Remember that the services share OSDs by default, so isolation between workloads is a CRUSH and device-class decision rather than something the service layers provide. And choose the simplest layer that meets the requirement — CephFS earns its metadata daemon only when many clients genuinely need simultaneous POSIX access.

Cross-course references

Ceph: Part XXXV (RBD Architecture) for the block layer in detail.
Ceph: Part XL (CephFS Architecture) for the file layer.
Ceph: Part XLV (RADOS Gateway) for the object layer.