Connecting OSD latency to what users see

Advanced⏱ ~17 mincephprometheus

What you'll learn

Explain how OSD latency propagates to clients
Quantify the client impact of one slow OSD
Demonstrate the link with evidence
Prioritise which outlier to address

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The link between “osd.44 is at 87 ms” and “our p99 doubled” is not obvious to anyone outside the storage team, and demonstrating it is what gets the device replaced.

How the latency propagates

a write to a PG whose acting set includes osd.44
  → primary sends subops to all replicas
  → completion waits for the slowest
  → the client sees osd.44's latency

The fraction of client operations affected:

affected fraction ≈ (PGs on osd.44) / (total PGs) × size
                  ≈ 1 / (number of OSDs) × size

On a 100-OSD cluster with size=3, roughly 3% of operations touch any given OSD — which is far above the p99 boundary, so the slow OSD sets the p99 entirely.

OSDs	`size`	Fraction touching one OSD	Percentile it dominates
30	3	10%	p90 and above
100	3	3%	p97 and above
300	3	1%	p99 and above
1000	3	0.3%	p99.7 and above

This table is the argument: on a 100-OSD cluster a single slow OSD determines everything above p97, and no amount of tuning elsewhere changes that.

Demonstrating the link

# 1. the outlier
ceph osd perf | sort -k2 -rn | head -3

# 2. the client percentile over the same window
# in Prometheus:
#   histogram_quantile(0.99, rate(ceph_osd_op_w_latency_bucket[5m]))

# 3. the decisive test — remove the OSD and re-measure
ceph osd out 44
# wait for the remap
fio --name=probe --ioengine=rbd --pool=rbd-vms --rbdname=probe \
    --rw=randwrite --bs=4k --iodepth=32 --runtime=120 --time_based \
    --percentile_list=50:95:99

If the p99 falls after marking the OSD out, the link is demonstrated rather than argued.

# restore if the test disproves it
ceph osd in 44

Marking one OSD out is low-risk on a healthy cluster with headroom, and it is the only evidence that settles the question definitively.

Prioritising among outliers

Outlier	Priority
One OSD far above the rest	highest — it sets the p99 alone
Several OSDs moderately high	investigate the shared host or component
A whole device class high	expected; compare against the class baseline
All OSDs high	load, not an outlier — different investigation

ceph osd perf | awk 'NR>1 {print $2}' | sort -n | \
  awk '{a[NR]=$1} END {print "median", a[int(NR/2)], "max", a[NR]}'

A max far above the median is an outlier problem; a high median is a load problem.

Quiz

Knowledge check · 4 questions

Q1. On a 100-OSD cluster with size=3, roughly what fraction of client operations touch any given OSD?

0.3% 10% 33% 3%
Q2. A slow OSD that sets the p99 outright on a 100-OSD cluster may not reach the p99 at all on a 1000-OSD one.

True False
Q3. Make the case for a device replacement.

osd.44 reports 87 ms while the cluster median is 3 ms. Client p99 write latency has doubled. Procurement questions whether the device is really the cause and asks for evidence.
Q4. How do you distinguish an outlier problem from a load problem in `ceph osd perf` output?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Compare the maximum OSD latency against the median before investigating: a high max with a low median is an outlier, a high median is a load problem. To prove an outlier is the cause, mark it out and re-measure the client p99 — a measured drop is evidence where correlated graphs are only an argument.

Cross-course references

Kubernetes: removing a suspect node and re-measuring is the same decisive test
Linux: taking a suspect disk out of an array to confirm attribution is standard