Tracing tail latency to its source

Advanced⏱ ~18 mincephfioprometheus

What you'll learn

Trace a latency tail to a specific source
Distinguish the common tail causes
Quantify each source's contribution
Address the identified source

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A tail is not diffuse. It is produced by identifiable events, and the method for finding them is systematic rather than exploratory.

The candidate sources

Source	Signature
One slow OSD	tail affects a fraction ≈ `size`/`n_osds`
Deep scrub	tail correlates with scrubbing PG count
Recovery or backfill	tail correlates with recovery activity
Device garbage collection	periodic tail spikes on flash
Network loss	tail spikes at TCP retransmission timeout, ~200 ms
RocksDB compaction	tail on the DB device, periodic
Client CPU scheduling	tail present at the client only
Full or nearfull OSD	tail plus health warnings

The 200 ms clustering is diagnostic: a tail concentrated near 200 ms is almost always TCP retransmission timeouts.

The method

# 1. is it one OSD?
ceph osd perf | sort -k2 -rn | head -5

# 2. does the tail correlate with scrub?
ceph pg dump pgs | grep -c scrubbing
# with noscrub set and scrubs drained, re-measure

# 3. does it correlate with recovery?
ceph -s | grep -E 'recovery|degraded'

# 4. is it at ~200 ms?
fio ... --percentile_list=99:99.9:99.99
netstat -s | grep -i retransmit

# 5. is it periodic?
# plot the latency over time rather than summarising it
fio ... --log_avg_msec=1000 --write_lat_log=latency

The latency log is the most informative single artefact: plotting latency against time reveals periodicity that percentiles cannot show.

python3 -c '
import sys
vals = [(int(l.split(",")[0]), int(l.split(",")[1]))
        for l in open("latency_lat.1.log")]
# bucket by second and print the max per second
import collections
m = collections.defaultdict(int)
for t, v in vals: m[t//1000] = max(m[t//1000], v)
for s in sorted(m)[:60]: print(s, m[s]//1000, "us")'

Quantifying the contribution

# with the suspected source removed
ceph osd set noscrub
ceph osd set nodeep-scrub
# drain, then re-measure

baseline p99:         14 ms
with scrub disabled:   9 ms      → scrub contributes 5 ms to p99
with osd.44 out:       6 ms      → osd.44 contributes 3 ms further

Subtracting one source at a time attributes the tail quantitatively rather than identifying a single culprit.

Addressing each

Source	Action
Slow OSD	investigate the device and host, replace if warranted
Scrub	reschedule to off-peak, extend the interval, add sleep
Recovery	throttle, or accept during the recovery window
Flash GC	over-provision, or choose drives with better GC behaviour
Network loss	fix the link; check for incast
RocksDB compaction	faster DB device, or fewer OSDs per DB device

Quiz

Knowledge check · 4 questions

Q1. Latency values in the tail cluster tightly around 200 ms rather than spreading smoothly. What does this indicate?

Garbage collection inside the flash devices themselves A deep scrub running across some PGs RocksDB compaction on the DB device TCP retransmission timeouts from network loss
Q2. Percentiles are sufficient to distinguish a periodic tail cause from a continuously slow component.

True False
Q3. Attribute a latency tail quantitatively.

A pool shows p99 write latency of 14 ms against an 8 ms target. Several possible causes are suspected: scrub activity, one slow OSD, and background recovery.
Q4. What is the most informative single artefact for diagnosing a latency tail?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Capture a latency time series with --write_lat_log rather than only percentiles; periodicity distinguishes a compaction or scrub cycle from a persistently slow component, and percentiles discard it. Attribute the tail quantitatively by removing one suspected source at a time and measuring the difference.

Cross-course references

Kubernetes: correlating latency spikes with scheduled jobs uses the same time-series approach
Linux: periodic latency from background daemons is invisible in aggregate statistics