Why one slow OSD dominates RBD latency

Why this matters in production

A single degraded device in a cluster of hundreds can make the entire cluster feel broken. The mechanism is simple and the arithmetic is stark, and once you have seen it, outlier detection becomes the monitoring you care most about.

The mechanism

A write completes when every OSD in the acting set has committed. So:

write latency = 2 × RTT + max(device latency across the acting set)

Not the average — the maximum. One OSD at 200 ms makes every write to every PG it participates in take 200 ms, regardless of how fast its peers are.

The blast radius

cluster: 100 OSDs, 4096 PGs, size 3
PG replicas total: 4096 × 3 = 12,288
PGs per OSD: 12,288 / 100 ≈ 123
share of PGs touched by one OSD: 123 / 4096 ≈ 3%

Three percent of PGs sounds small. But client objects distribute evenly across PGs, so roughly 3% of all operations hit that OSD — and for an application doing thousands of operations, essentially every user transaction touches it at some point. The user-visible effect is not “3% of requests are slow” but “the p99 is terrible and the application feels broken.”

Detecting it

ceph osd perf | sort -k2 -n | tail -10

osd  commit_latency(ms)  apply_latency(ms)
 47                  11                 13
 83                  12                 14
 12                 847                861

The check that matters is not an absolute threshold but the ratio to the cluster median:

ceph osd perf --format json | \
  jq -r '.osdstats.osd_perf_infos[] |
         "\(.id) \(.perf_stats.commit_latency_ms)"' | \
  sort -k2 -rn | head -5

Alert when any OSD exceeds, say, five times the median for its device class. An absolute threshold either misses outliers on slow hardware or fires constantly on fast hardware.

Responding

# 1. confirm it is the device, not the host
HOST=stor-04
DEVICE_ID=12
ceph osd metadata 12 | jq -r '.hostname, .devices'
ssh ${HOST} 'iostat -x 5 3'
ceph device get-health-metrics ${DEVICE_ID}

# 2. reduce its share of traffic while investigating
ceph osd reweight 12 0.5

# 3. if the device is failing, drain and replace
ceph osd out 12

Reweighting is the graduated response: it moves some PGs away, reducing the outlier’s blast radius without the full rebalance that marking it out triggers.

Quiz

Production discipline

Alert on per-OSD latency deviation from the median within each device class; it is the single highest-value Ceph alert and an absolute threshold cannot substitute for it. Use reweighting as the graduated response so an outlier can be relieved immediately without committing to a full rebalance at an inconvenient time.

Cross-course references

• Kubernetes: one slow node in a service backend degrades p99 the same disproportionate way
• Linux: the slowest device in a RAID array bounds every synchronous write identically