Tail latency — why p99 is the number that matters

Why this matters in production

A dashboard showing 3 ms average write latency can coexist with an application team reporting timeouts, and both are accurate. The average is dominated by the many fast operations; the complaints come from the few slow ones. Storage is a domain where the distribution matters more than its centre, and tail latency is the vocabulary for saying so.

Reading the percentiles

• p50 (median) — half of operations are faster. Close to what the average shows on a healthy system.
• p99 — one operation in a hundred is slower than this. For a database doing 10,000 operations per second, that is 100 slow operations every second.
• p99.9 — one in a thousand. This is where user-visible stalls live for interactive workloads.
• p99.99 — one in ten thousand. Batch jobs and long-running transactions hit this regularly enough to matter.

The arithmetic is the argument. A web request that touches twenty storage operations has a roughly 18% chance of hitting at least one p99 operation — so the p99 of the storage layer becomes something close to the p80 of the user experience. This is why “only 1% of operations are slow” is not the reassurance it sounds like.

Why distribution amplifies the tail

In a single-disk system the tail comes from the disk. In a distributed system it comes from whichever component is slowest at that instant, and there are many components.

A replicated write is acknowledged when the slowest of three OSDs commits it. If each OSD independently has a 1% chance of being slow, the write has roughly a 3% chance of being slow. Replication improves durability and worsens the tail — a trade worth knowing explicitly.

Add the sources of transient slowness and the picture fills in:

• an SSD running internal garbage collection
• a deep scrub reading every object in a PG
• recovery or backfill competing for the same device queue
• a network buffer filling during a burst
• an OSD’s memory target being exceeded, triggering cache trimming

None of these is a fault. All of them produce tail latency, and all of them are things an operator can schedule or throttle.

Finding the tail in Ceph

# the OSD with the outlying latency in `ceph osd perf`:
OSD_ID=13

ceph osd perf                          # commit/apply latency per OSD
ceph daemon "osd.$OSD_ID" perf dump    # full histograms from one OSD
ceph -s                                # slow ops appear here when severe
ceph health detail                     # names the OSDs with blocked requests

ceph osd perf is a snapshot rather than a histogram, but the distribution across OSDs is exactly what you want: a column of numbers around 2 ms with one at 60 ms names the device to investigate. For real histograms, the Prometheus module exports per-OSD latency buckets, which is what a p99 dashboard should be built on.

The operator’s levers

Most tail latency in a healthy Ceph cluster is self-inflicted and schedulable:

• Scrub windows. osd_scrub_begin_hour and osd_scrub_end_hour keep deep scrub out of business hours.
• Recovery throttles. osd_max_backfills, osd_recovery_max_active, and osd_recovery_sleep decide how much of the device budget recovery may take.
• Device replacement. One degrading device is the most common single cause, and it is found in ceph osd perf.
• Separating the cluster network, so replication traffic stops contending with client I/O on the same link.

Quiz

Production discipline

Measure and alert on p99 and p99.9; averages will not tell you the cluster is degrading and slow ops will only tell you once it is an incident. Build the histograms from the Prometheus module rather than relying on the point-in-time ceph osd perf snapshot, though that command remains the fastest way to spot one bad device. Most tail latency in a healthy cluster is schedulable — deep scrub windows, recovery throttles, and network separation — so review those before concluding the hardware is inadequate.

Cross-course references

• Observability: histogram metrics, percentile queries, and alerting on quantiles.
• Ceph: Part LXI (Scrubbing) and Part LX (Recovery Tuning) for the two largest self-inflicted sources.
• Ceph: Part LXXIX (Slow Ops) for what happens when the tail becomes an incident.