Tracing RGW latency to its layer

Advanced⏱ ~18 mincurlcephradosgw-admin

What you'll learn

Measure latency at each layer of the path
Use gateway operation logs for per-request timing
Attribute latency correctly
Apply the appropriate remedy per layer

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

“S3 is slow” spans four layers with four different fixes. Measuring each takes a few minutes and prevents an optimisation project aimed at the wrong one.

Measuring the layers

# end to end, with a breakdown
curl -s -o /dev/null -w '
  dns:      %{time_namelookup}
  connect:  %{time_connect}
  tls:      %{time_appconnect}
  ttfb:     %{time_starttransfer}
  total:    %{time_total}
' "https://rgw.example.com/bucket/object"

Gap	Layer
dns	name resolution
connect − dns	network to the balancer
tls − connect	TLS handshake
ttfb − tls	balancer plus gateway plus pools
total − ttfb	data transfer

A large ttfb with a small total − ttfb means the delay is in processing, not transfer.

Isolating the balancer

# through the balancer
curl -s -o /dev/null -w '%{time_starttransfer}\n' https://rgw.example.com/bucket/obj

# directly to a gateway
curl -s -o /dev/null -w '%{time_starttransfer}\n' http://rgw-01:8080/bucket/obj

A large difference is the balancer — queueing, TLS termination cost, or a health check flapping a backend.

Per-request timing at the gateway

LOG_OBJECT=report.pdf
ceph config set client.rgw rgw_enable_ops_log true
ceph config set client.rgw rgw_ops_log_rados true

radosgw-admin log list
radosgw-admin log show --object=${LOG_OBJECT} | \
  jq -r '.log_entries[] | "\(.total_time) \(.op) \(.uri)"' | sort -rn | head -20

The operations log records per-request duration, which identifies slow operations and their patterns — a specific bucket, a specific operation type, or a specific size range.

Attributing to the pools

ceph osd pool stats default.rgw.buckets.index
ceph osd pool stats default.rgw.buckets.data
ceph osd perf | sort -k2 -n | tail -5

Pattern	Layer
LIST and small PUT slow, GET fast	index pool
Large GET slow, small operations fast	data pool or network
Everything slow, one OSD an outlier	that OSD
Slow only through the balancer	balancer
Slow at high concurrency only	gateway thread pool or pool saturation

The remedies

Layer	Remedy
DNS	caching resolver, longer TTL
Network to balancer	topology, not usually tunable
TLS	session resumption, or terminate at the gateway
Balancer	more instances, or layer 4 to reduce cost
Gateway	more gateways, or a larger thread pool
Index pool	flash devices, more shards
Data pool	more OSDs, or address an outlier

Quiz

Knowledge check · 4 questions

Q1. A curl measurement shows a large time_starttransfer and a small transfer time. Where is the delay?

In the network transfer, since time_starttransfer covers the body arriving at the client In processing — the balancer, gateway, or pools — before the first byte is returned In DNS resolution, which curl folds into time_starttransfer rather than reporting separately In the TLS handshake, which is measured from the connection opening to the first byte
Q2. Latency that is fine at low load and degrades sharply past a threshold, with CPU idle, suggests gateway thread pool exhaustion.

True False
Q3. Attribute a latency increase across layers.

S3 request latency has doubled. Requests through the load balancer take 240 ms to first byte; the same requests directly against a gateway take 45 ms. The cluster is healthy and pool latency is normal.
Q4. Why should the RGW ops log be enabled before an incident rather than during one?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Enable the RGW operations log with a retention policy in advance, so a latency investigation starts with per-request evidence rather than with a decision to begin collecting it. Measure through the balancer and directly against a gateway as the first step — the difference attributes the latency in one comparison.

Cross-course references

Kubernetes: distributed tracing across ingress, service, and backend serves the same attribution purpose
Linux: curl timing breakdowns are the standard first measurement for any HTTP service