Running multiple gateways well

Why this matters in production

Gateways are interchangeable only if they are configured identically. A fleet where one gateway has a different frontend setting, a different TLS certificate, or a stale configuration produces intermittent failures that depend on which gateway a request landed on — the hardest kind to reproduce.

Deploying a fleet

ceph orch apply rgw default \
    --realm=default --zone=default \
    --placement="count:4 label:rgw" \
    --port=8080

ceph orch ps --daemon-type rgw
ceph orch ls --service-type rgw

The orchestrator maintains the declared count, replacing daemons on failed hosts automatically.

Keeping configuration consistent

# set at the client.rgw level so every gateway inherits it
ceph config set client.rgw rgw_thread_pool_size 512
ceph config set client.rgw rgw_max_concurrent_requests 1024

# verify what each daemon actually has
for d in $(ceph orch ps --daemon-type rgw --format json | jq -r '.[].daemon_name'); do
  echo "== $d"
  ceph config show "$d" 2>/dev/null | grep -E 'thread_pool|frontends'
done

Setting configuration in the central config database rather than in per-host files is what keeps a fleet consistent. Per-host ceph.conf entries are the usual source of divergence.

Sizing

gateways = ceil(peak load / per-gateway capacity) + 1

The +1 covers a host failure and a rolling upgrade. A fleet sized exactly to peak has no headroom for either, so every maintenance window becomes a capacity event.

Verifying every gateway serves

for h in rgw-01 rgw-02 rgw-03 rgw-04; do
  printf '%-10s ' "$h"
  curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' "http://$h:8080/"
done

Hitting each gateway directly, bypassing the load balancer, is how a single misbehaving gateway is found. A load balancer with a weak health check keeps it in rotation and the failures look random.

Per-gateway differences that cause trouble

The last is worth noting: S3 signatures include a timestamp checked against a tolerance, so one gateway with a drifting clock rejects a fraction of requests.

Quiz

Production discipline

Set RGW configuration centrally rather than in per-host files, and verify the effective configuration of each daemon after any change — the orchestrator reports a divergent gateway as healthy. Test each gateway directly when failures are intermittent; the fraction failing usually points straight at how many gateways are involved.

Cross-course references

• Kubernetes: configuration drift between replicas produces the same intermittent behaviour
• Linux: comparing effective configuration across a fleet is standard practice for the same reason