RGW identity: two separate systems

Why this matters in production

There are two authentication systems in play and they do not overlap. Confusing them produces both wasted effort — looking for S3 users in ceph auth ls — and real security errors, such as granting an S3 user cephx capabilities it has no way to use.

The two layers

graph LR
    A[S3 client] -->|access key + secret, AWS SigV4| B[radosgw daemon]
    B -->|cephx: client.rgw.node1| C[RADOS cluster]

S3/Swift users exist in the RGW user database, stored in RGW’s own pools. They authenticate with an access key and secret key using AWS signature v4. They have no cephx entity, appear nowhere in ceph auth ls, and cannot talk to RADOS directly.

The radosgw daemon has a cephx entity like any other Ceph daemon and is the only thing that touches RADOS.

Managing S3 users

radosgw-admin user create --uid=analytics --display-name="Analytics Team"
radosgw-admin user info --uid=analytics

# rotate a key
radosgw-admin key create --uid=analytics --key-type=s3 --gen-access-key --gen-secret
radosgw-admin key rm --uid=analytics --access-key=OLDACCESSKEY

# quotas
radosgw-admin quota set --uid=analytics --quota-scope=user --max-size=10T
radosgw-admin quota enable --uid=analytics --quota-scope=user

# suspend rather than delete
radosgw-admin user suspend --uid=analytics

Key rotation here is straightforward because a user may hold multiple access keys simultaneously — create the new one, migrate consumers, remove the old. That overlap is built in, unlike cephx.

Restricting what a user can do

Subusers for Swift-style scoping:

radosgw-admin subuser create --uid=analytics --subuser=analytics:readonly \
    --access=read

Bucket policies for S3-style fine-grained control:

aws --endpoint-url https://rgw.example.com s3api put-bucket-policy \
    --bucket reports --policy file://policy.json

Policies express per-bucket, per-action, per-principal rules and are the right mechanism for anything beyond coarse read/write separation.

The daemon’s own capabilities

ceph auth get client.rgw.node1
# caps: [mon] allow rw
# caps: [osd] allow rwx

The gateway needs broad access because it manages its own pools and acts on behalf of every user. Scope it to the RGW pools where your deployment allows:

ceph auth caps client.rgw.node1 \
    mon 'allow rw' \
    osd 'allow rwx pool=default.rgw.buckets.data, allow rwx pool=default.rgw.buckets.index, allow rwx pool=default.rgw.meta, allow rwx pool=default.rgw.log, allow rwx pool=default.rgw.control, allow rwx pool=default.rgw.buckets.non-ec'

Verify carefully after doing this — a missing pool breaks the gateway.

Quiz

Production discipline

Cover both identity databases in any access review, and record that they are separate so the next reviewer does not repeat the omission. Secure the gateway hosts to the standard implied by their access — the gateway process holds effective authority over every bucket, and tenant isolation depends entirely on it behaving correctly.

Cross-course references

• Kubernetes: application-level tenancy over shared infrastructure credentials has the same property
• Linux: a service that multiplexes users onto one system account carries the same concentration of trust