Load balancing RGW: layer 4 versus layer 7

Why this matters in production

The two modes differ in what the balancer can see and therefore in what it can check and log. For RGW specifically, the large-request behaviour and the health-check quality are what usually decide it.

The comparison

Layer 4

frontend s3_tcp
    bind *:443
    mode tcp
    default_backend rgw_tcp

backend rgw_tcp
    mode tcp
    balance leastconn
    option httpchk GET /
    http-check expect status 403
    server rgw1 10.20.0.11:8443 check check-ssl verify none
    server rgw2 10.20.0.12:8443 check check-ssl verify none

TLS passes through to the gateways, so certificates live on each gateway and the balancer cannot see request contents. Note that even in TCP mode HAProxy can perform an HTTP health check, which is what distinguishes a useful configuration from a plain connect check.

Layer 7

frontend s3_http
    bind *:443 ssl crt /etc/haproxy/certs/rgw.pem
    mode http
    option forwardfor
    default_backend rgw_http

backend rgw_http
    mode http
    balance leastconn
    option httpchk GET /
    http-check expect status 403
    timeout server 300s
    server rgw1 10.20.0.11:8080 check
    server rgw2 10.20.0.12:8080 check

TLS terminates at the balancer, per-request logging is available, and client IPs are forwarded in a header.

ceph config set client.rgw rgw_remote_addr_param http_x_forwarded_for

Without that setting, every request appears to originate from the load balancer in RGW’s logs and in any IP-based bucket policy condition.

Large uploads

Multipart part uploads can be large and slow. Two settings matter:

timeout server 300s
timeout client 300s

Default timeouts of 30 or 60 seconds abort large part uploads midway, producing failures that appear random and correlate with object size. Check for request buffering too — a balancer buffering a 5 GB part before forwarding it needs the memory to hold it.

Choosing

Layer 7 for most deployments: better health checks, per-request logging, and centralised certificate management.

Layer 4 when end-to-end encryption is required, when the balancer’s CPU is a constraint, or when the deployment must not have request contents visible at the balancer.

Quiz

Production discipline

Configure the health check to expect the 403 an unauthenticated root request produces, and verify it fails on a gateway isolated from the cluster. Raise proxy timeouts to match your largest expected part transfer — the default values abort large uploads in a way that looks random and correlates with size.

Cross-course references

• Kubernetes: Ingress timeout annotations address exactly this large-upload problem
• Linux: any reverse proxy in front of a bulk-transfer service faces the same timeout tuning