RGW: translating S3 into RADOS

Why this matters in production

RGW performance problems are almost always in one of three places: the gateway process, the index pool, or the data pool. Knowing which stage of the request touches which makes the investigation short.

A PUT, stage by stage

graph TD
    A[S3 client PUT] --> B[RGW: authenticate signature]
    B --> C[RGW: resolve bucket and user - meta pool]
    C --> D[RGW: check policy and quota]
    D --> E[RGW: write data objects - data pool]
    E --> F[RGW: update bucket index - index pool]
    F --> G[RGW: 200 OK]

The last two are where latency lives, and they have different characteristics: data writes scale with object size, index updates are small and latency-sensitive.

Large objects

aws s3 cp large.bin s3://data/large.bin

RGW splits objects above rgw_max_chunk_size into a head object plus tail objects:

ceph config get client.rgw rgw_max_chunk_size       # 4 MiB
ceph config get client.rgw rgw_obj_stripe_size      # 4 MiB

rados -p default.rgw.buckets.data ls | head
# 8f3a2b1c.1_large.bin              ← head
# 8f3a2b1c.1__shadow_.abc123_1      ← tail
# 8f3a2b1c.1__shadow_.abc123_2

The head object carries the metadata and the first chunk; tails carry the rest. This is why a single S3 object appears as many RADOS objects.

Statelessness

A gateway holds no durable state. Everything — bucket metadata, user records, object data, indexes — is in RADOS.

Consequences:

• Any gateway can serve any request for any bucket
• Gateways can be added or removed without coordination
• A gateway crash loses only the requests in flight
• Load balancing needs no session affinity

ceph orch apply rgw default --placement="3 rgw1 rgw2 rgw3"
ceph orch ps --daemon-type rgw

Structuring an investigation

# is it the gateway process?
ceph orch ps --daemon-type rgw
journalctl -u ceph-radosgw@* --since '30 min ago' | grep -iE 'error|warn'

# is it the index pool?
ceph osd pool stats default.rgw.buckets.index

# is it the data pool?
ceph osd pool stats default.rgw.buckets.data

# per-request detail
ceph config set client.rgw debug_rgw 10      # bounded window

Slow small operations — HEAD, LIST, small PUT — point at the index pool. Slow large transfers point at the data pool or the network.

Quiz

Production discipline

Classify RGW performance complaints by operation size before investigating: small operations point at the index pool, large transfers at the data pool or network. Put the index pool on flash for the same reason as the CephFS metadata pool — it is small, and its latency is paid by every operation.

Cross-course references

• Kubernetes: stateless application replicas scale the same way behind a load balancer
• Linux: separating a metadata-intensive path onto fast storage is a recurring pattern