RGW failure modes and their responses

Advanced⏱ ~18 mincephradosgw-admin

What you'll learn

Recognise each RGW failure mode from its symptoms
Apply the appropriate first response
Distinguish gateway, pool, and multisite failures
Prioritise correctly during a compound failure

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

RGW failures span the gateway, the pools beneath it, the load balancer in front of it, and multisite replication beside it. Having the catalogue means the first minute is spent narrowing rather than guessing.

The catalogue

Failure	Symptom	First action
Single gateway crash	brief errors, clients retry successfully	verify the balancer removed it
All gateways down	total service outage	check daemon status and cluster health
Index pool slow	LIST and small PUT slow, transfers fine	check index pool latency and device class
Index pool full	writes fail, reads work	free capacity or rebalance
Data pool full	writes fail with 500	free capacity, check per-OSD variance
Load balancer failure	total outage despite healthy gateways	check VIP and balancer health
Weak health check	intermittent errors, one bad gateway in rotation	test gateways directly
Multisite sync stalled	one zone stale, the other fine	`radosgw-admin sync status`
Bucket index corruption	LIST returns wrong or missing objects	`radosgw-admin bucket check`
Clock skew	intermittent SignatureDoesNotMatch	check NTP on gateways

Triage sequence

# 1. is the cluster healthy?
ceph -s

# 2. are the gateways running?
ceph orch ps --daemon-type rgw

# 3. can each gateway serve, directly?
for h in rgw-01 rgw-02 rgw-03; do
  curl -s -o /dev/null -w "$h %{http_code}\n" "http://$h:8080/"
done

# 4. are the pools healthy?
ceph df | grep rgw
ceph osd pool stats default.rgw.buckets.index

# 5. is multisite syncing?
radosgw-admin sync status

Five commands, and they eliminate most of the catalogue.

Bucket index repair

radosgw-admin bucket check --bucket=data
radosgw-admin bucket check --bucket=data --fix
radosgw-admin bucket check --bucket=data --check-objects --fix

--check-objects verifies every index entry against the data pool, which is thorough and slow. Use it when listings disagree with reality, not routinely.

Prioritising a compound failure

Total outage — service down for everyone
Writes failing — data is not being accepted
Partial outage — some clients or buckets affected
Degraded performance — service works, slowly
Replication lag — one zone behind

Multisite lag ranks last: the primary is serving correctly and the secondary catches up. It is a durability concern rather than an availability one, unless the primary then fails.

Quiz

Knowledge check · 4 questions

Q1. A LIST shows an object that a subsequent GET returns 404 for. What has happened?

The object was deleted between the two operations, and the LIST was served from a cached listing An index inconsistency — the index entry exists without a corresponding object The data pool has lost the object, since the LIST reads the index and the GET reads the data pool The bucket policy allows ListBucket but denies GetObject, which RGW reports as a 404
Q2. Multisite replication lag should be prioritised above write failures during a compound incident.

True False
Q3. Triage a compound RGW incident.

Simultaneously: one gateway of four has crashed, multisite sync to the secondary site is four hours behind, and clients report that some uploads are failing with 500 errors. The cluster reports HEALTH_WARN with a nearfull OSD.
Q4. Why does testing gateways directly matter more than testing through the load balancer during an incident?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Keep the five-command triage sequence in the runbook so the first minute of an RGW incident narrows rather than guesses. Prioritise by whether data is being accepted rather than by how alarming a symptom looks — multisite lag reads as urgent and is the lowest priority of the common failures.

Cross-course references

Kubernetes: triaging by user impact rather than by alert severity is the same discipline
Linux: a documented failure catalogue per service is what makes on-call effective