Object storage — the flat namespace primitive

Why this matters in production

Object storage is where the data that never fits anywhere else ends up: backups, media originals, build artefacts, logs, machine-learning training sets, anything measured in “how many billion”. It is also the primitive most often described badly — as “just files over HTTP” — which sets operators up to be surprised by the three things it does not do.

For a Ceph operator, RGW is the service that turns a RADOS cluster into an S3 endpoint. Understanding what the object contract promises tells you why RGW needs its own pools, why the bucket index is the component that fails first at scale, and why the answer to “can we just mount it?” is no.

The contract

An object is an opaque blob with a key, some metadata, and a body. The operations are essentially four:

PUT    bucket/key        write the whole object
GET    bucket/key        read the whole object (or a byte range)
DELETE bucket/key        remove it
LIST   bucket?prefix=    enumerate keys, lexicographically

That is close to the entire API surface, and the omissions are the interesting part:

• No partial write. You cannot change byte 4096 of a 10 GiB object. You PUT a new object. Multipart upload lets you assemble a large object from parts, but each part is still written whole and the object is immutable once complete.
• No rename. Renaming is a copy followed by a delete, and for a large object that is a real data movement, not a metadata update.
• No directories. logs/2026/08/app.log is not three nested containers; it is one key that happens to contain slashes. Listing “a directory” is a prefix scan.

Why flat scales and hierarchy does not

A filesystem directory is a real object with real state: it has a size, permissions, a link count, and a lock. Every create in a directory contends on it. Every rename has to be atomic across two directories. Every traversal walks the tree. This is exactly what a metadata server exists to coordinate, and it is why filesystems get harder as they get larger.

A flat namespace has none of that. A key is just a string. Two clients writing a/b/c and a/b/d share no state at all, so they do not contend. The system shards keys across the cluster by hashing them, and adding capacity is adding shards.

The price is that anything hierarchy gave you for free, you now do yourself: listing a prefix is a scan, moving a “folder” is copying every object under it, and there is no such thing as “size of this directory” without enumerating it.

What RGW actually is

RGW is a stateless HTTP daemon that speaks S3 (and Swift) and translates into librados calls. Stateless matters: you run several behind a load balancer and lose one without consequence.

flowchart LR
  C[S3 client] -->|HTTPS| LB[Load balancer]
  LB --> G1[radosgw]
  LB --> G2[radosgw]
  G1 --> P1[(.rgw.buckets.index)]
  G1 --> P2[(.rgw.buckets.data)]
  G1 --> P3[(.rgw.meta / .rgw.log)]
  G2 --> P1
  G2 --> P2

The pools matter and they have different needs. The data pool holds object payloads and is the natural home for erasure coding — large objects, sequential reads, capacity-dominated. The index pool holds bucket indexes, which are small, hot, and latency sensitive; it belongs on replicated NVMe. Putting the index on the same slow spinners as the data is one of the most common RGW performance mistakes.

Where object storage fits, and where it does not

Good fits, in the order you meet them:

• Backups and archives. Write once, read rarely, keep forever, measured in TiB. Erasure coding makes the capacity affordable.
• Media and static assets. Whole-file reads, HTTP-native clients, cacheable, often served through a CDN that speaks S3 already.
• Anything with an S3 client already. Velero, Restic, Loki, Prometheus long-term storage, Spark, and most data tooling assume S3 exists. Giving them RGW is often a configuration change.

Bad fits:

• Databases. Covered elsewhere and worth repeating: no partial writes, latency an order of magnitude too high.
• Anything expecting POSIX. If the application calls open(), seek(), and write(), an S3 gateway mounted with a FUSE shim will technically work and will disappoint everyone.
• Small objects in vast numbers where listing matters. Hundreds of millions of 4 KiB objects is a bucket-index problem long before it is a capacity problem.

Quiz

Production discipline

Treat the bucket index as the component that fails first. Put the index pool on replicated NVMe, size shard counts against expected object count before the bucket exists, and leave dynamic resharding enabled so growth is handled automatically rather than discovered as an incident. Erasure-code the data pool where objects are large and capacity dominates; never erasure-code the index. And push back on any design that needs partial writes, rename, or directory semantics from object storage — those requirements are a signal that the primitive is wrong, not that the API needs a wrapper.

Cross-course references

• Linux: Part XLVIII (Backup Tools) for the restic and Borg clients that consume S3 endpoints.
• Observability: Loki and Prometheus long-term storage both use S3-compatible object storage as their backing store.
• Kubernetes: Velero writes cluster backups to an S3 endpoint, which RGW can provide.