Storage primitives — why the primitive is the contract

Why this matters in production

Almost every difficult storage conversation is really an argument about the primitive. Somebody wants the archive tier on RBD because “it is already there”. Somebody else wants the database on CephFS because “a filesystem is simpler”. Both are choosing a primitive for a reason that has nothing to do with what the workload needs, and both decisions are expensive to reverse once data is in them.

The primitive is a contract: it fixes what operations exist, what they cost, and who owns the consistency. Everything downstream — latency budget, failure behaviour, how you back it up, how many clients can write at once — follows from that contract. Ceph is unusual in serving all three primitives from one RADOS cluster, which makes the choice feel low-stakes. It is not. The pools, the CRUSH rules, the client libraries, and the recovery characteristics all differ.

The three primitives

flowchart TD
  A[Client workload] --> B{What does it need?}
  B -->|"Byte-range random access,<br/>owns its filesystem"| C["Block<br/>RBD"]
  B -->|"Shared POSIX namespace,<br/>many concurrent writers"| D["File<br/>CephFS"]
  B -->|"Whole immutable blobs,<br/>HTTP clients, huge scale"| E["Object<br/>RGW / S3"]
  C --> F[RADOS]
  D --> F
  E --> F

Block exposes a flat, fixed-size address space. You read and write byte ranges by offset. The storage knows nothing about files — the client puts a filesystem on top and owns every question about consistency, journalling, and crash recovery. One writer at a time.

File exposes a hierarchical namespace with POSIX semantics: directories, permissions, locks, rename() that is atomic. Many clients can mount it and write concurrently, and the storage is responsible for making that safe. That responsibility is why file storage needs a metadata service, and why it is the primitive with the most moving parts.

Object exposes a flat namespace of opaque blobs addressed by key, over HTTP. Objects are written whole and replaced whole; there is no byte-range write, no rename, no directory. In exchange you get an API that scales to billions of objects and clients that need nothing but a network connection and credentials.

What each contract actually costs you

Read the “extra daemons” row carefully, because it is the row operators forget. CephFS availability depends on an MDS being up and having replayed its journal. RGW availability depends on gateways being up behind a load balancer. RBD needs neither — a client with a keyring and a route to the OSDs can do I/O. That difference shows up in every outage.

Where the primitives are not interchangeable

The classic mistakes, each of which reaches production regularly:

• A database on object storage. PostgreSQL wants 8 KiB page writes at a bounded latency and its own WAL semantics. An object API gives whole-object PUT at tens of milliseconds. There is no tuning that closes that gap.
• A shared web-asset tree on RBD. RBD is single-writer. Mounting the same image read-write on two hosts corrupts the filesystem — not slowly and not subtly.
• Backups on CephFS because it is convenient. It works, and it puts your backups behind an MDS, in the same cluster, in the same failure domain as the thing they protect. Object storage in a different cluster is the honest answer.

Choosing, in one page

Ask the workload four questions:

1. Does one client own the data, or many? Many concurrent writers rules out block immediately.
2. Are writes partial or whole? Byte-range updates rule out object. Appending to a 4 GiB log file is a partial write.
3. What is the latency budget per operation? Under a few milliseconds means block, and probably NVMe-backed.
4. How many things will there be? Hundreds of millions of items means object; a filesystem with that many inodes is a metadata problem you will spend years on.

If the answers conflict — and for real workloads they often do — the resolution is usually to split the data, not to compromise on one primitive. A media platform puts its originals in RGW, its database on RBD, and its render scratch on CephFS. That is three pools with three CRUSH rules in one cluster, and it is the design Ceph exists to make possible.

Quiz

Production discipline

Decide the primitive from the access pattern before anything else, and write the decision down with its reasoning. It is the least reversible choice in a storage design: pools, CRUSH rules, client libraries, backup strategy, and failure behaviour all follow from it, and changing it later means migrating data and rewriting clients. When one workload’s answers conflict, split the data across primitives rather than forcing a compromise — one Ceph cluster serving RBD, CephFS, and RGW side by side is the design the system was built for.

Cross-course references

• Linux: Part XIII (Disks and Block Devices) and Part XIV (Filesystems) cover the block and POSIX layers this lesson sits above.
• Proxmox: Part V (Storage Fundamentals) and Part VIII (Ceph) for the hypervisor view of the block primitive.
• Kubernetes: Parts XLVIII-LV (storage and CSI) for how the primitives reach a Pod.