Working an RBD latency complaint to its cause

Advanced⏱ ~18 mincephrbdfio

What you'll learn

Apply a layered latency investigation
Rule out each layer with a specific measurement
Identify the most common causes quickly
Escalate with evidence rather than impressions

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

“RBD is slow” arrives without qualification and can mean any of six things. Working the layers in order, with one measurement per layer, gets to the cause in minutes and produces evidence rather than a hypothesis.

The descent

1. Is the cluster healthy?

ceph -s
ceph health detail

Recovery, backfill, or scrubbing in progress explains elevated latency immediately. So does a nearfull OSD.

2. Is one OSD an outlier?

ceph osd perf | sort -k2 -n | tail -5

This is the highest-yield single command in the sequence. An OSD an order of magnitude above the median is the answer more often than anything else.

3. Is this image or the whole cluster?

rbd perf image iostat --pool rbd-vms

One image with elevated latency among many normal ones points at that image’s workload or its object placement. All images elevated points at the cluster.

4. Is the workload itself different?

# in the guest
iostat -x 5

Check avgqu-sz and the request rate. A queue depth of 1 explains low IOPS with normal latency, and an increased request rate explains everything.

5. Is the network contributing?

OSD_HOST=stor-04
ping -c 1000 -i 0.01 ${OSD_HOST}

Compare against the recorded baseline. Two round trips of unexpected RTT is a large share of a flash-cluster operation.

6. Where does the time actually go?

# PRIMARY_OSD: the primary for the slow PG, or the outlier from step 2
PRIMARY_OSD=44

ceph daemon "osd.$PRIMARY_OSD" dump_historic_ops

The per-stage timings localise the delay precisely — queue, replica wait, or local device commit.

The common causes, ranked

Cause	Frequency	Settled by
One slow OSD	most common	step 2
Recovery or scrub in progress	common	step 1
Workload change in the guest	common	step 4
nearfull OSD throttling	occasional	step 1
Network degradation	occasional	step 5
Snapshot copy-on-write burst	occasional	correlate with snapshot schedule

Quiz

Knowledge check · 4 questions

Q1. Which single command most often identifies the cause of an RBD latency complaint?

ceph df, to see whether a pool is close to full ceph osd perf, sorted to reveal a latency outlier rbd info on the affected image, to check its features ceph auth ls, to confirm the client's capabilities
Q2. An OSD approaching nearfull degrades write latency gradually, before any warning threshold is crossed.

True False
Q3. Investigate a latency complaint with no obvious cause.

Multiple teams report RBD latency roughly doubled over the past two weeks. The cluster is HEALTH_OK, no OSD is a latency outlier, no recovery is running, and network RTT matches the baseline. Capacity is at 81%.
Q4. Why is a recorded baseline essential for this investigation?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Record baseline values for OSD latency distribution, per-image IOPS, and network RTT during a quiet period and store them where on-call can reach them; every step of this investigation is a comparison and without a baseline none of them conclude. Treat capacity headroom as a performance parameter, not only a safety threshold.

Cross-course references

Kubernetes: a layered descent from pod to node to storage is the same investigative structure
Linux: the USE method — utilisation, saturation, errors per resource — is the general form of this