When the pools behind RGW are the problem

Why this matters in production

The gateway reports HTTP errors; the cause is frequently a pool beneath it. Mapping the failing operation to the pool it depends on is what turns a 500 into a specific capacity or latency problem.

Which operation depends on which pool

So a deployment where GET works and PUT fails points at the index or non-ec pool rather than the data pool.

Pool conditions and their gateway symptoms

Checking

ceph -s
ceph df | grep rgw
ceph health detail | grep -i full

radosgw-admin zone get --rgw-zone=default | \
  jq -r '.placement_pools[].val | {index_pool, storage_classes, data_extra_pool}'

for p in $(ceph osd pool ls | grep rgw); do
  printf '%-40s ' "$p"
  ceph osd pool stats "$p" | tail -1
done

Responding to a full pool

# where is the fullness?
ceph osd df | sort -k17 -rn | head

# rebalance before anything else
ceph osd reweight-by-utilization 110
ceph balancer status

# reclaim: garbage collection may be behind
radosgw-admin gc list --include-all | jq 'length'
radosgw-admin gc process --include-all

# reclaim: incomplete multipart uploads
radosgw-admin bucket list | jq -r '.[]' | while read b; do
  radosgw-admin bucket check --bucket="$b" 2>/dev/null | grep -i multipart
done

Garbage collection lagging is a common and easily-missed cause: deleted objects are queued for removal and the queue can grow faster than it drains, so capacity does not return.

ceph config get client.rgw rgw_gc_max_concurrent_io
ceph config get client.rgw rgw_gc_processor_period

Verifying from the gateway’s perspective

ceph config set client.rgw debug_rgw 10
ceph config set client.rgw debug_rados 5
# reproduce the failing operation
ceph config set client.rgw debug_rgw 1

The gateway log shows which RADOS operation failed and against which pool, which settles the mapping definitively.

Quiz

Production discipline

Map the failing operation to the pools it depends on before investigating; which operations work is as informative as which fail. Check the garbage collection backlog whenever capacity does not return after a deletion — it is the most common reason a cleanup appears to have done nothing.

Cross-course references

• Kubernetes: finalizers deferring actual deletion produce the same gone-but-not-freed behaviour
• Linux: unlinked files held open by a process behave identically from a capacity perspective