RGW multi-tenancy in operation

Why this matters in production

RGW tenancy is billed and audited more often than block or file tenancy, which makes its accounting the part that gets scrutinised.

Usage accounting

ceph config get client.rgw rgw_enable_usage_log
ceph config set client.rgw rgw_enable_usage_log true

Usage logging is off by default. Without it there is no per-user
operation record, and enabling it later does not backfill.

radosgw-admin usage show --uid='acme$acme-prod' --show-log-entries=false

{
    "summary": [{
        "user": "acme$acme-prod",
        "total": { "bytes_sent": 91827364, "bytes_received": 4736251,
                   "ops": 18422, "successful_ops": 18401 }
    }]
}

# per-user storage, which is separate from operation counts
radosgw-admin user stats --uid='acme$acme-prod' --sync-stats

radosgw-admin bucket stats --uid='acme$acme-prod' | python3 -c '
import sys,json
for b in json.load(sys.stdin):
    u = b.get("usage", {}).get("rgw.main", {})
    print("%-28s %10.2f GiB  %8d objects" %
          (b["bucket"], u.get("size_actual",0)/1024**3, u.get("num_objects",0)))'

--sync-stats recalculates from the bucket indices rather than reading
the cached figure, which is what makes a billing number defensible.

Key lifecycle

radosgw-admin key create --uid='acme$acme-prod' --key-type=s3 --gen-access-key
radosgw-admin user info --uid='acme$acme-prod' | python3 -c '
import sys,json
for k in json.load(sys.stdin).get("keys", []):
    print(k["access_key"])'

OLD_ACCESS_KEY=report.pdf
radosgw-admin key rm --uid='acme$acme-prod' --key-type=s3 \
  --access-key=${OLD_ACCESS_KEY}

A user can hold several access keys, which makes RGW key rotation
genuinely incremental: add the new key, move the tenant, remove the old.
That is a property cephx does not have.

radosgw-admin user suspend --uid='acme$acme-prod'
radosgw-admin user enable --uid='acme$acme-prod'

Quota consistency

radosgw-admin quota set --quota-scope=user --uid='acme$acme-prod' \
  --max-size=5T --max-objects=10000000
radosgw-admin quota enable --quota-scope=user --uid='acme$acme-prod'

ceph config get client.rgw rgw_bucket_quota_ttl
ceph config get client.rgw rgw_user_quota_bucket_sync_interval

RGW caches quota state, so a tenant can exceed a quota briefly before
enforcement catches up. The window is governed by those intervals.

So an RGW quota is a soft ceiling by design. Sizing it with headroom
below the hard capacity limit is the correct response, not tuning the
interval to zero.

A tenant exceeding quota

radosgw-admin user stats --uid='acme$acme-prod' --sync-stats
radosgw-admin quota show --quota-scope=user --uid='acme$acme-prod'

The S3 client receives 403 QuotaExceeded on writes. Reads and deletes
continue, which lets the tenant recover by deleting.

Quiz

Production discipline

Rotate RGW keys incrementally — a user can hold several access keys at once, so add, migrate, verify, remove with no window where nothing works. Enable rgw_enable_usage_log before you need it; it does not backfill. Use --sync-stats for any figure that will be billed.

Cross-course references

• Kubernetes: multiple valid credentials are what make rotation non-disruptive anywhere
• Linux: cached enforcement trades exactness for throughput deliberately