Buckets: the unit of policy and the unit of scale

Intermediate⏱ ~17 minradosgw-adminaws

What you'll learn

Explain what a bucket is and what it controls
Describe the bucket index and its sharding
Predict the behaviour of very large buckets
Design a bucket structure for a workload

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A bucket looks like a folder and behaves like a sharded index. The design decision that matters most — how many buckets and how many objects in each — follows from the index behaviour, not from any organisational preference.

What a bucket provides

Property	Scope
Key namespace	keys are unique within a bucket
Access policy	bucket policies and ACLs apply per bucket
Versioning	enabled or disabled per bucket
Lifecycle rules	configured per bucket
Placement target	which pools back it
Quota	per-bucket limits

The index

Every bucket has an index — a set of RADOS objects in the index pool listing the bucket’s keys. LIST reads it; every PUT and DELETE updates it.

radosgw-admin bucket stats --bucket=data
radosgw-admin metadata get bucket:data

The index is the reason bucket size matters. A single index object serving millions of keys becomes a contention point for every write to the bucket.

Sharding

radosgw-admin bucket reshard --bucket=data --num-shards=64
radosgw-admin reshard status --bucket=data

ceph config get client.rgw rgw_max_objs_per_shard      # 100000 default
ceph config get client.rgw rgw_dynamic_resharding      # true by default

Dynamic resharding increases shard count automatically as a bucket grows, targeting roughly rgw_max_objs_per_shard keys per shard.

Resharding is not free: it rebuilds the index, and during the rebuild operations on the bucket are affected. On a very large bucket that is a noticeable event.

Designing bucket structure

Approach	Suits
One bucket per tenant	clear policy boundary, natural quota unit
One bucket per dataset	lifecycle rules differ per dataset
One enormous bucket	rarely right — index pressure, no policy granularity
Many small buckets	metadata overhead per bucket, listing across them is manual

The practical guidance: buckets in the tens or hundreds per tenant, with objects in the millions per bucket, and prefixes for organisation within them.

Listing cost

aws s3api list-objects-v2 --bucket data --prefix 2026/08/ --max-keys 1000

Listing reads the index across all shards and merges the results, so a prefix-scoped listing on a heavily sharded bucket still touches every shard. Applications that list frequently should list narrowly and cache results.

Quiz

Knowledge check · 4 questions

Q1. Why does adding index shards improve write throughput but not listing performance?

Listing is served from a cached copy of the index, so the shard count has no bearing on it Writes update one shard chosen by key hash, while listing must read and merge every shard to produce sorted order Listing reads only the first shard, which holds the sorted key range for the whole bucket Sharding applies only to the write path of versioned buckets, and listing bypasses it entirely
Q2. Rebuilding the index of a bucket holding hundreds of millions of objects degrades operations on that bucket throughout.

True False
Q3. Design bucket structure for a large ingest platform.

A platform will ingest 8 billion small objects over three years from 200 data sources. Access is predominantly writes with occasional retrieval by exact key, and rare listings scoped to a source and a date.
Q4. What does the bucket index cost on every write?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Set shard counts at bucket creation for buckets you expect to be large; dynamic resharding works but performs a disruptive rebuild at whatever moment the threshold is crossed. Design key prefixes so listings can be narrow, since listing is the operation sharding does not help.

Cross-course references

Kubernetes: label selectors across large object counts have the same list-versus-write asymmetry
Linux: directory size effects on filesystem operations follow similar scaling