The RGW pools and what each requires

Why this matters in production

An RGW deployment creates six or seven pools automatically, and treating them uniformly is a mistake: one holds nearly all the bytes, one determines nearly all the latency, and one breaks large uploads if it is missing.

The pools

ceph osd pool ls | grep rgw
radosgw-admin zone get --rgw-zone=default | jq 'keys'

The index pool

The one that matters most for responsiveness:

ceph osd crush rule create-replicated rgw-index-nvme default host nvme
ceph osd pool set default.rgw.buckets.index crush_rule rgw-index-nvme
ceph osd pool set default.rgw.buckets.index size 3
ceph osd pool set default.rgw.buckets.index pg_num_min 128
ceph osd pool set default.rgw.buckets.index pg_autoscale_bias 4

It holds omap, so it cannot be erasure coded. It is small by capacity and touched by every PUT, DELETE, and LIST. Flash placement and a PG floor are the two settings worth making explicitly.

The data pool

ceph osd erasure-code-profile set rgw-ec k=8 m=3 crush-failure-domain=host
ceph osd pool create default.rgw.buckets.data erasure rgw-ec
ceph osd pool application enable default.rgw.buckets.data rgw
ceph osd pool set default.rgw.buckets.data bulk true

This is where the capacity is and where EC pays. Verify average object size against k × min_alloc_size before choosing a wide profile.

The non-ec pool

ceph osd pool ls | grep non-ec

Holds in-flight multipart upload state, which involves operations EC pools handle poorly. It must exist and be replicated. Its absence breaks large uploads specifically while ordinary PUTs continue working — a distinctive failure signature.

Per-pool symptoms

Quiz

Production discipline

Place the RGW index pool on the fastest device class available and plan its capacity from projected object count rather than data volume; both decisions are cheap and both are commonly got wrong by relying on the automatic pool creation. Verify the non-ec pool exists on any EC-backed deployment, since its absence breaks only large uploads.

Cross-course references

• Kubernetes: separating etcd onto fast storage follows the identical reasoning
• Linux: filesystem inode capacity versus data capacity is the same count-versus-volume distinction