Placing and sizing RGW daemons

Intermediate⏱ ~16 mincephradosgw-admin

What you'll learn

Deploy RGW daemons with the orchestrator
Size gateway count and resources for a workload
Decide where gateways should run
Configure per-gateway settings appropriately

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Gateways are stateless and horizontally scalable, which makes the placement decision about resource contention rather than about correctness. The main question is whether they share hosts with OSDs.

Deploying

ceph orch apply rgw default \
    --realm=default --zone=default \
    --placement="count:4 label:rgw" \
    --port=8080

ceph orch ps --daemon-type rgw
ceph orch host label add rgw-01 rgw

Or from a service specification:

service_type: rgw
service_id: default
placement:
  label: rgw
  count: 4
spec:
  rgw_frontend_port: 8080
  rgw_realm: default
  rgw_zone: default

Where to run them

Placement	Suits
Dedicated gateway hosts	production at scale; no contention
Co-located with OSDs	small clusters; contention risk under load
Co-located with monitors	small clusters; monitors are latency-sensitive
In containers on the client side	edge cases; adds a network hop for other clients

Dedicated hosts are the production answer. Co-locating gateways with OSDs means the CPU spent on HTTP handling, signature verification, and encryption competes with OSD work — and both peak at the same time, because gateway load causes OSD load.

Resource characteristics

Resource	Driver
CPU	request rate, signature verification, TLS, encryption
Memory	concurrent request count, buffering
Network	object throughput, doubled by the RADOS hop

Note the network point: a gateway receives an object from the client and sends it to the OSDs, so it moves each byte twice. A gateway on a 25 Gb link delivers roughly half that in client throughput.

Sizing

gateways = ceil(peak request rate / per-gateway capacity) + 1

The +1 is for maintenance and failure. A two-gateway deployment running at capacity has no headroom for a rolling upgrade.

# observe per-gateway load
ceph orch ps --daemon-type rgw
ceph daemon /var/run/ceph/ceph-client.rgw.*.asok perf dump | jq '.rgw'

Per-gateway settings

ceph config set client.rgw rgw_thread_pool_size 512
ceph config set client.rgw rgw_max_concurrent_requests 1024
ceph config set client.rgw rgw_frontends "beast port=8080 num_threads=512"

The thread pool bounds concurrency. Too small and requests queue at the gateway while the cluster is idle; too large and memory grows with each in-flight request.

Quiz

Knowledge check · 4 questions

Q1. A gateway on a 25 Gb link is expected to serve how much client throughput?

The full 25 Gb, since the gateway streams straight through to the OSDs as it receives Roughly half — it receives from the client and sends to the OSDs, moving each byte twice Roughly double, since RGW compresses object payloads before sending them onward It depends only on CPU, since signature verification is what bounds the request rate
Q2. A gateway can exhaust its thread pool while using very little CPU.

True False
Q3. Diagnose gateway request queueing with idle resources.

RGW requests are queueing with high latency. Gateway CPU is at 15%, memory is ample, the network is at 20%, and the cluster reports HEALTH_OK. Adding a fourth gateway made no difference.
Q4. Why are dedicated gateway hosts preferred over co-locating RGW with OSDs?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Size gateway network links against double the expected client throughput, since every byte traverses the gateway twice — this is the most common RGW sizing error. Measure pool latency before adding gateways when requests queue with idle resources; more gateways add more threads to block on the same slow pool.

Cross-course references

Kubernetes: an ingress controller has the same double-traversal bandwidth property
Linux: a reverse proxy sized against backend rather than frontend bandwidth makes the same error