High availability for RGW

Why this matters in production

Because gateways hold no state, RGW high availability is entirely a matter of getting requests to a healthy gateway. That makes it simpler than most HA problems and puts all the subtlety in the health check.

The shape

graph TD
    C[S3 clients] --> V[Virtual IP / DNS]
    V --> L1[Load balancer]
    V --> L2[Load balancer standby]
    L1 --> G1[rgw-01]
    L1 --> G2[rgw-02]
    L1 --> G3[rgw-03]
    G1 --> R[(RADOS)]
    G2 --> R
    G3 --> R

No session affinity is required — any gateway serves any request.

With cephadm

ceph orch apply rgw default --placement="count:3 label:rgw" --port=8080

ceph orch apply ingress --service-id rgw.default \
    --placement="count:2 label:lb" \
    --frontend-port 443 --monitor-port 8999 \
    --virtual-ip 10.20.0.100/24

The ingress service deploys HAProxy and keepalived, giving a virtual IP that fails over between load balancer hosts.

Health checks

A TCP connect check is not sufficient — a gateway can accept connections while unable to reach the cluster. Check something that exercises the path:

curl -sf http://rgw-01:8080/ -o /dev/null -w '%{http_code}\n'

An unauthenticated request to the root returns a well-formed S3 error, which proves the gateway is processing requests. A gateway that cannot reach RADOS fails this while still accepting TCP connections.

TLS

ceph config set client.rgw rgw_frontends \
  "beast port=8080 ssl_port=8443 ssl_certificate=/etc/ceph/rgw.pem"

Terminating at the load balancer is the common choice; terminating at the gateway is right where internal network traffic must be encrypted.

Failure and upgrade behaviour

Gateway failure: the health check fails, the load balancer removes it, in-flight requests on that gateway fail and clients retry. S3 SDKs retry by default, so a single gateway failure is usually invisible.

Rolling upgrade:

ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.1
ceph orch upgrade status

Gateways are upgraded one at a time. With n+1 capacity the upgrade is transparent; at exactly n it is a period of reduced capacity.

Quiz

Production discipline

Use an HTTP health check that exercises the request path and verify it actually fails on a gateway isolated from the cluster; a TCP check keeps broken gateways serving errors. Provision n+1 gateways so rolling upgrades do not reduce capacity, and make the load balancer itself redundant.

Cross-course references

• Kubernetes: readiness probes must exercise the dependency path for the same reason
• Linux: any load-balanced stateless service faces the identical health-check design question