Reading a latency distribution

Advanced⏱ ~17 minfioprometheus

What you'll learn

Interpret each percentile correctly
Diagnose from the shape of the distribution
Explain why the mean misleads
Choose which percentile to hold as an objective

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Storage latency distributions are heavily skewed, and the summary statistic chosen determines whether a problem is visible at all.

What each percentile means

Percentile	Describes	At 30k ops/s
p50	the typical operation	—
p95	1 in 20 operations	1500 per second
p99	1 in 100	300 per second
p99.9	1 in 1000	30 per second
p99.99	1 in 10000	3 per second
max	the single worst	once

The right-hand column is the one that changes minds: a p99 of 80 ms on a busy system is not a rare event, it is 300 slow operations every second.

The shape

tight distribution:   p50 3 ms   p95 4 ms    p99 5 ms    p99.9 8 ms
long tail:            p50 3 ms   p95 5 ms    p99 42 ms   p99.9 310 ms
bimodal:              p50 3 ms   p95 3 ms    p99 95 ms   p99.9 98 ms
uniformly slow:       p50 38 ms  p95 44 ms   p99 51 ms   p99.9 62 ms

Shape	Diagnosis
Tight	healthy; the device class sets the level
Long tail	an outlier OSD, contention, or scrub
Bimodal	two populations — e.g. cache hits and misses, or one slow OSD
Uniformly slow	saturation or a device class limit

The bimodal case is distinctive: p95 near p50 with p99 far above means most operations are fast and a specific subset is consistently slow — which points at a particular OSD or a particular code path rather than at general contention.

Why the mean misleads

1000 operations: 999 at 2 ms, 1 at 3000 ms
mean = 5 ms
p99  = 2 ms
max  = 3000 ms

The mean is dragged upward by the outlier and describes no actual operation. Conversely:

1000 operations: 990 at 2 ms, 10 at 500 ms
mean = 7 ms
p99  = 500 ms

Here the mean understates a problem affecting 1% of operations severely. In both directions it fails to describe what users experience.

Choosing an objective

Objective on	Suits
p50	nothing, alone
p95	interactive workloads with retry tolerance
p99	most storage objectives
p99.9	latency-critical systems
max	nothing — a single event defines it

fio ... --percentile_list=50:95:99:99.9

histogram_quantile(0.99, sum(rate(ceph_osd_op_w_latency_bucket[5m])) by (le))

An objective on p99 with p99.9 monitored is the common and defensible combination: the objective is achievable and the additional percentile reveals rare severe events before they become frequent.

Quiz

Knowledge check · 4 questions

Q1. A distribution shows p50 3 ms, p95 3 ms, p99 95 ms, p99.9 98 ms. What does this shape indicate?

General contention across the cluster, which raises every percentile above the median together Device saturation, which lifts the whole distribution including the p50 and the p95 A bimodal distribution — most operations are fast and a specific subset is consistently slow A measurement artefact from too few samples above the p95 to place the higher percentiles
Q2. The mean latency is a reasonable summary when the distribution has a long tail.

True False
Q3. Prioritise a latency problem.

A p99 write latency of 80 ms is reported. Management considers it low priority since "99% of operations are fine". The pool serves 30,000 operations per second.
Q4. Why are percentiles the correct tool for storage latency rather than mean and standard deviation?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Convert a percentile into events per second using the actual operation rate before deciding priority — “1%” and “300 slow operations every second” are the same fact and only one of them gets acted on. Read the shape as well as the level: p95 near p50 with p99 far above is bimodal and points at a specific OSD.

Cross-course references

Kubernetes: request latency SLOs use percentiles for the identical reason
Linux: any queueing system produces the same skewed distribution