Load balancer failures in front of RGW

Why this matters in production

Adding a load balancer to protect against gateway failure introduces a component whose failure takes out everything. Getting its redundancy and its health checking right is what makes the addition a net gain.

Failure modes

Redundancy

ceph orch apply ingress --service-id rgw.default \
    --placement="count:2 label:lb" \
    --frontend-port 443 --monitor-port 8999 \
    --virtual-ip 10.20.0.100/24

ceph orch ps --service-name ingress.rgw.default

Two instances with a virtual IP managed by keepalived. Verify the failover actually works rather than assuming:

# on the active balancer host
systemctl stop haproxy
# from a client, confirm requests continue

Untested failover is the most common reason a redundant balancer does not help.

Health checks

option httpchk GET /
http-check expect status 403
inter 5s fall 3 rise 2

Too aggressive and a gateway briefly slow is removed, reducing capacity and possibly cascading. Too lenient and a broken gateway serves errors for a long time.

Diagnosing balancer symptoms

# is the VIP where it should be?
ip addr show | grep 10.20.0.100
ssh lb-02 ip addr show | grep 10.20.0.100

# backend state
echo "show stat" | socat stdio /var/run/haproxy.sock | \
  awk -F, '{print $1, $2, $18}'

# is the balancer or a gateway the problem?
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' https://rgw.example.com/
for h in rgw-01 rgw-02 rgw-03; do
  curl -s -o /dev/null -w "$h %{http_code} %{time_total}\n" "http://$h:8080/"
done

The VIP appearing on two hosts simultaneously is a split-brain in keepalived and produces intermittent failures that depend on ARP resolution — worth checking explicitly when symptoms are inconsistent.

Quiz

Production discipline

Test balancer failover and gateway removal explicitly during commissioning; both failure paths are silent until exercised and neither is proven by normal operation. Check for the VIP on more than one host whenever symptoms are intermittent with no pattern — split brain produces exactly that.

Cross-course references

• Kubernetes: verifying that a Service actually removes an unready pod is the same test
• Linux: any VRRP or cluster-manager failover needs the same explicit exercise