Lab: Histograms and Latency

Objective

Every latency panel you will ever read is an estimate produced by histogram_quantile() from a bucket layout somebody chose once and never revisited. This lab makes the true distribution knowable — you set it — so you can measure how far the estimate is from the truth and what moves it.

By the end you will have watched the same distribution reported as 156 ms (the mean), 2.2 s (the p99 under the Go client library’s default buckets) and 1.84 s (the p99 under a layout with boundaries where the data is). The true p99 is 1.84 s. Two of those three numbers would have been on a dashboard.

Architecture

One host, four containers. Three identical generator processes emit a histogram with a distribution you configure by environment variable; one Prometheus scrapes all three.

   +------------------+   +------------------+   +------------------+
   | gen-a  (fast)    |   | gen-b  (fast)    |   | gen-c  (slow)    |
   | 95% @  60-100ms  |   | 95% @  60-100ms  |   | 95% @ 500-900ms  |
   |  5% @ 1200-2000ms|   |  5% @1200-2000ms |   |  5% @2400-4000ms |
   | :8000/metrics    |   | :8000/metrics    |   | :8000/metrics    |
   +--------+---------+   +--------+---------+   +--------+---------+
            |                      |                      |
            +----------+-----------+----------+-----------+
                       |  scrape every 15s
                       v
              +--------+---------+
              | prometheus 2.55  |
              | job="gen"        |
              | :9090            |
              +------------------+

Two of the three instances are healthy and one is not. That ratio matters: it is the shape of a real fleet, and it is what makes the aggregation task in Task 6 produce the answer it does.

Requirements

A Linux host with Docker Engine 28.x and Docker Compose v2, plus network access on first run so the image build can install one Python package.
curl and jq on the host. Every measurement is a Prometheus HTTP API query parsed with jq.
Free TCP ports 9090, 8001, 8002 and 8003 on the host.
Roughly 400 MiB of disk for the images and a few hundred MiB of memory.
No out-of-band access requirement: nothing outside the lab directory and the compose project is touched.

Scenario

A payments team runs a checkout API with a stated SLO of “p99 below 300 ms”. The service was instrumented on its first day with prometheus.DefBuckets, because that is the default and nobody had a reason to change it. The p99 panel has read between 90 ms and 120 ms for a year.

Support tickets say otherwise. A small but persistent group of customers reports checkout taking “a couple of seconds”. The traces confirm it. The panel does not. Nobody has been able to reconcile the two, and the current working theory on the incident channel is that the tracing is sampling badly.

Your job is to reproduce the shape on a bench, establish which number is wrong and why, and produce the change that makes the panel agree with the customers.

Tasks

Task 1: Write the generator

LABDIR="$HOME/rb-obs-histogram"
mkdir -p "$LABDIR"
cd "$LABDIR"

gen.py — a self-driving histogram source. The bucket layout, the two modes of the distribution and the mix between them are all environment variables, so one image serves every task in this lab:

#!/usr/bin/env python3
"""Synthetic latency source for the histogram lab.

Observations are DRAWN from a bimodal distribution rather than measured from
real work. That makes the true quantiles computable by hand, which is what
lets the lab measure the error in histogram_quantile() rather than trusting it.
"""
import os
import random
import time

from prometheus_client import Histogram, start_http_server


def env_float(name, default):
    return float(os.environ.get(name, default))


# The bucket layout is fixed when the Histogram is constructed. Changing this
# variable requires recreating the container - see Task 7 for why that matters.
BUCKETS = tuple(
    float(x)
    for x in os.environ.get(
        "BUCKETS", "0.005,0.01,0.025,0.05,0.1,0.25,0.5,1,2.5,5,10"
    ).split(",")
)

ROUTE = os.environ.get("ROUTE", "/checkout")
RATE = env_float("RATE", 200)          # observations per second
FAST_MS = env_float("FAST_MS", 80)     # centre of the fast mode
SLOW_MS = env_float("SLOW_MS", 1600)   # centre of the slow mode
SLOW_RATIO = env_float("SLOW_RATIO", 0.05)  # fraction landing in the slow mode

# client_python appends the +Inf bucket itself; do not list it here.
REQUESTS = Histogram(
    "http_request_duration_seconds",
    "Simulated time spent handling HTTP requests.",
    labelnames=("method", "route", "status"),
    buckets=BUCKETS,
)


def draw():
    """One observation, in seconds, uniform within +/-25% of its mode."""
    centre = SLOW_MS if random.random() < SLOW_RATIO else FAST_MS
    return random.uniform(centre * 0.75, centre * 1.25) / 1000.0


def main():
    start_http_server(8000)
    interval = 1.0 / RATE
    while True:
        REQUESTS.labels("GET", ROUTE, "200").observe(draw())
        time.sleep(interval)


if __name__ == "__main__":
    main()

Dockerfile:

FROM python:3.12-slim
RUN pip install --no-cache-dir prometheus_client==0.21.0
COPY gen.py /app/gen.py
EXPOSE 8000
CMD ["python", "-u", "/app/gen.py"]

Task 2: Write the Prometheus config and start the stack

prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: gen
    static_configs:
      - targets: ['gen-a:8000', 'gen-b:8000', 'gen-c:8000']

compose.yaml:

name: rb-obs-histogram

# The three generators share a build context and differ only in environment.
x-gen: &gen
  build: .
  restart: unless-stopped

services:
  gen-a:
    <<: *gen
    container_name: rb-hist-gen-a
    environment:
      RATE: '200'
      ROUTE: '/checkout'
      FAST_MS: '80'
      SLOW_MS: '1600'
      SLOW_RATIO: '0.05'
    ports:
      - '8001:8000'

  gen-b:
    <<: *gen
    container_name: rb-hist-gen-b
    environment:
      RATE: '200'
      ROUTE: '/checkout'
      FAST_MS: '80'
      SLOW_MS: '1600'
      SLOW_RATIO: '0.05'
    ports:
      - '8002:8000'

  # The degraded instance. Same code, same bucket layout, worse latency.
  gen-c:
    <<: *gen
    container_name: rb-hist-gen-c
    environment:
      RATE: '200'
      ROUTE: '/checkout'
      FAST_MS: '700'
      SLOW_MS: '3200'
      SLOW_RATIO: '0.05'
    ports:
      - '8003:8000'

  prometheus:
    image: prom/prometheus:v2.55.1
    container_name: rb-hist-prom
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=2h'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    ports:
      - '9090:9090'

volumes:
  prom-data:

Service impact possiblelab host— Builds the generator image and binds four host ports

$ docker compose up -d --build

Let it run for five minutes before taking any measurement. Every query in this lab uses a [5m] rate window, and a partially-filled window produces numbers that move while you read them.

docker compose ps --format 'table {{.Name}}\t{{.Status}}'
curl -sf http://localhost:9090/-/ready && echo PROMETHEUS-READY

Task 3: Read the raw exposition and reconcile it by hand

Before any PromQL, look at what the producer actually emits. This is the object every later query operates on:

curl -s http://localhost:8001/metrics | grep '^http_request_duration_seconds'

You are looking at eleven _bucket series with finite le values, one with le="+Inf", one _sum and one _count — fourteen series from one histogram with one label set. Three facts to confirm with your own eyes, because every later step depends on them:

The bucket values never decrease as le grows. They are cumulative counters, each a superset of the one below.
_count equals the le="+Inf" bucket, exactly. If they differ you are looking at two scrapes.
_sum divided by _count is the mean observation — around 0.156 seconds for the fast instances, given a 95/5 mix of an 80 ms mode and a 1600 ms mode.

Check that third one directly:

curl -s http://localhost:8001/metrics \
| awk '/^http_request_duration_seconds_sum/ {s=$2}
       /^http_request_duration_seconds_count/ {c=$2}
       END {printf "sum=%.3f count=%d mean=%.4f s\n", s, c, s/c}'

Now count the series Prometheus is storing for this one metric:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=count by (instance) (http_request_duration_seconds_bucket)' \
| jq -r '.data.result[] | "\(.metric.instance) \(.value[1]) bucket series"'

Twelve per instance: eleven finite boundaries plus +Inf. Add _sum and _count and one label set costs fourteen series. That is the number to multiply when somebody proposes adding a customer_id label.

Task 4: Watch the mean lie

The panel the payments team has been reading is an average. Reproduce it:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=rate(http_request_duration_seconds_sum{instance="gen-a:8000"}[5m])
                          / rate(http_request_duration_seconds_count{instance="gen-a:8000"}[5m])' \
| jq -r '.data.result[] | "mean = \(.value[1]) s"'

Approximately 0.156 s. Now ask the question the SLO actually asks:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=histogram_quantile(0.99,
                            sum by (le) (
                              rate(http_request_duration_seconds_bucket{instance="gen-a:8000"}[5m])))' \
| jq -r '.data.result[] | "p99 = \(.value[1]) s"'

Approximately 2.2 s. Same instance, same five minutes, same data. One number is fourteen times the other, and neither is wrong: the mean is dragged up by the slow 5% and dragged back down by the fast 95%, landing in a region where no actual request lives. Not one observation in this distribution took 156 ms.

Task 5: Measure the interpolation error

Here is what makes this lab different from reading the panel: you know the true answer. The distribution is 95% uniform over 60-100 ms and 5% uniform over 1200-2000 ms. The 99th percentile therefore sits inside the slow mode, at the point where the top 1% begins — that is (0.99 - 0.95) / 0.05 = 0.8, the 80th percentile of the slow mode:

true p99 = 1200 ms + 0.8 * (2000 - 1200) ms = 1840 ms = 1.84 s

histogram_quantile() reported 2.2 s. Work out why from the layout. The default boundaries near the tail are 1, 2.5, 5, 10. Every slow observation lands in the bucket spanning 1 s to 2.5 s, so 5% of the mass is smeared uniformly across a 1.5-second-wide bucket, which is the assumption the function is forced to make. The rank falls 80% of the way into that bucket:

estimate = 1 + 0.8 * (2.5 - 1) = 2.2 s

The function did exactly the right thing with the information it had. The 20% error is not in the query. It is in the layout. Confirm the mass really is where the arithmetic says by reading the bucket rates directly:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum by (le) (rate(http_request_duration_seconds_bucket{instance="gen-a:8000"}[5m]))' \
| jq -r '.data.result[] | "le=\(.metric.le) \(.value[1])"' | sort -g -k1.4

Read it as a cumulative distribution: the value at le="0.1" is about 95% of the value at le="+Inf", nothing changes between 0.25 and 1, and the remaining 5% appears all at once at le="2.5". A layout with nothing between 1 and 2.5 cannot resolve a tail that lives at 1.84.

Task 6: Watch the fleet aggregation hide the bad instance

gen-c is the degraded instance. The p99 query most teams have on their dashboard aggregates the whole job:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=histogram_quantile(0.99,
                            sum by (le) (rate(http_request_duration_seconds_bucket{job="gen"}[5m])))' \
| jq -r '.data.result[] | "job p99 = \(.value[1]) s"'

Now the same question, per instance:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=histogram_quantile(0.99,
                            sum by (instance, le) (rate(http_request_duration_seconds_bucket{job="gen"}[5m])))' \
| jq -r '.data.result[] | "\(.metric.instance) p99 = \(.value[1]) s"' | sort

Two instances at roughly 2.2 s and one well above it — gen-c should read somewhere near 4.5 s. The job-level number lands between the two, because pooling the bucket rates builds a distribution whose tail is gen-c’s tail diluted by two instances that do not have it.

That in-between number is the problem. It overstates the latency of the two healthy instances and understates the latency of the degraded one, so it describes nobody’s experience. Any threshold you set from it is simultaneously too high to catch gen-c early and too low to be quiet when the fleet is healthy. One third of your users are on the bad instance and the panel that is supposed to represent them shows a number none of them see.

The discipline from lesson 02, stated as a rule you can apply without thinking: decide what you want the p99 of, keep those labels in the inner sum by (...), drop the rest, and always keep le. For alerting, take the worst:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=max(histogram_quantile(0.99,
                            sum by (instance, le) (rate(http_request_duration_seconds_bucket{job="gen"}[5m]))))' \
| jq -r '.data.result[] | "worst instance p99 = \(.value[1]) s"'

Then prove the most common query bug in production by dropping le from the inner aggregation:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=histogram_quantile(0.99,
                            sum(rate(http_request_duration_seconds_bucket{job="gen"}[5m])))' \
| jq '{status, result_count: (.data.result | length), warnings}'

The aggregation removed the boundaries the function needs, so there is nothing left to invert. In a Grafana panel this renders as “No data” — indistinguishable at a glance from a service that has stopped serving traffic.

Task 7: Move the boundaries and measure the estimate converge

The fix for Task 5’s 20% error is boundaries where the data is. Recreate gen-a with a layout that keeps the default’s coverage of the fast mode and adds resolution across 1 s to 2 s:

docker compose stop gen-a
docker compose rm -f gen-a

Edit compose.yaml and add a BUCKETS line to the gen-a environment block:

  gen-a:
    <<: *gen
    container_name: rb-hist-gen-a
    environment:
      RATE: '200'
      ROUTE: '/checkout'
      FAST_MS: '80'
      SLOW_MS: '1600'
      SLOW_RATIO: '0.05'
      # Dense where the SLO and the tail actually live; coarse beyond, because
      # nothing operational changes between 5s and 10s.
      BUCKETS: '0.01,0.025,0.05,0.075,0.1,0.15,0.25,0.5,1,1.25,1.5,1.75,2,3,5,10'
    ports:
      - '8001:8000'

docker compose up -d gen-a

Wait a full five minutes so the rate window contains only post-change scrapes, then re-run the p99 query from Task 4 against gen-a. It should now read approximately 1.84 s — the true value, because the rank now falls in the bucket spanning 1.75 s to 2 s, which is narrow enough that the uniform assumption inside it is nearly true.

Now look at what the change cost you. Ask Prometheus which le values exist for this instance across a window that straddles the restart:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=count by (le) (count_over_time(http_request_duration_seconds_bucket{instance="gen-a:8000"}[30m]))' \
| jq -r '.data.result[] | .metric.le' | sort -g

Both layouts are present. The old boundaries stopped receiving samples but the series still exist for the retention window, and any query whose range window spans the restart is mixing two different histograms.

Task 8: Find the ceiling

The default layout’s highest finite boundary is 10 s. histogram_quantile() never reports a value above the highest finite boundary, because there is nothing above it to interpolate towards. Make that visible by giving an instance a layout that ends well below its own tail. Recreate gen-b with a ceiling of 1 s while it continues to emit observations up to 2 s:

docker compose stop gen-b
docker compose rm -f gen-b

Add to gen-b’s environment in compose.yaml:

      BUCKETS: '0.01,0.025,0.05,0.1,0.25,0.5,1'

docker compose up -d gen-b

After five minutes:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=histogram_quantile(0.99,
                            sum by (le) (rate(http_request_duration_seconds_bucket{instance="gen-b:8000"}[5m])))' \
| jq -r '.data.result[] | "p99 = \(.value[1]) s"'

Exactly 1. Not approximately — exactly, and it will read exactly 1 whether the slow mode is at 1.6 s or 16 s. The rank falls in the +Inf bucket, and the documented behaviour there is to return the upper bound of the highest finite bucket. Cross-check against a statistic that has no ceiling:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=rate(http_request_duration_seconds_sum{instance="gen-b:8000"}[5m])
                          / rate(http_request_duration_seconds_count{instance="gen-b:8000"}[5m])' \
| jq -r '.data.result[] | "mean = \(.value[1]) s"'

The mean is unchanged by the layout, because _sum records the real values. A p99 pinned flat on a round number while the mean moves is the signature of a tail that has left the histogram, and it is the one histogram pathology you can diagnose from the panel alone.

Validation

Four checks. Each one proves a claim rather than repeating a step.

1. The exposition is internally consistent. _count equals the +Inf bucket on every instance:

for port in 8001 8002 8003; do
  curl -s "http://localhost:$port/metrics" \
  | awk -v p="$port" '/_bucket.*le="\+Inf"/ {b=$2}
                      /^http_request_duration_seconds_count/ {c=$2}
                      END {printf "%s inf=%d count=%d match=%s\n", p, b, c, (b==c ? "yes" : "NO")}'
done

Every line must end match=yes.

2. The SLO-aligned layout is measurably closer to the truth. Query both instances at once and compare against the computed 1.84 s:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=histogram_quantile(0.99,
                            sum by (instance, le) (rate(http_request_duration_seconds_bucket{job="gen"}[5m])))' \
| jq -r '.data.result[] | "\(.metric.instance) \(.value[1])"' | sort

gen-a (dense layout) near 1.84; gen-c (default layout) materially further from its own true value. Compute gen-c’s true p99 from its configured modes and check the direction of its error matches the reasoning in Task 5.

3. The per-instance query surfaces what the job query hides. The maximum of the per-instance p99 exceeds the job-level p99:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=max(histogram_quantile(0.99, sum by (instance, le) (rate(http_request_duration_seconds_bucket{job="gen"}[5m]))))
                          - histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="gen"}[5m])))' \
| jq -r '.data.result[] | "hidden by aggregation: \(.value[1]) s"'

A positive number is the size of the problem the dashboard is not showing.

4. The ceiling is a hard stop, not an estimate. Sample gen-b’s p99 three times a minute apart; all three read exactly 1.

Expected Outcome

Four containers running; three histogram producers on three different bucket layouts, all scraped by one Prometheus.
A recorded set of four numbers for the same distribution: mean 0.156 s, default-layout p99 2.2 s, dense-layout p99 1.84 s, arithmetic truth 1.84 s.
A per-instance p99 query that identifies gen-c as degraded, and a job-level query that does not.
A gen-b whose p99 is pinned at its highest finite boundary while its mean continues to move.

Troubleshooting

Queries return an empty result. The rate window is not full. Wait five minutes after any container restart before believing a [5m] query.
The build fails at pip install. The host has no network access to PyPI. Build once on a connected host and copy the image, or vendor the wheel into the build context.
_count and the +Inf bucket disagree. You have read two different scrapes, or grep matched a second metric family. Re-run the awk check, which reads one response.
A generator container exits immediately. docker compose logs gen-a shows the Python traceback. The usual cause is a malformed BUCKETS value: the boundaries must be comma-separated, strictly increasing, and must not include +Inf.
The p99 does not change after editing BUCKETS. The container was restarted rather than recreated, so it is running the old process, or you edited the wrong service block. docker compose rm -f then up -d recreates.
Both old and new le values appear indefinitely. That is correct until the old series age out of the 2-hour retention configured here. In production it is the full retention window.

Cleanup

Everything the lab created is one directory, one compose project and one named volume.

Data-loss risklab host— Removes the four containers and this project's Prometheus volume

$ cd ~/rb-obs-histogram && docker compose down -v

docker volume ls | grep rb-obs-histogram || echo "volumes gone"

# The generator image was built locally and is not used by anything else.
docker image rm rb-obs-histogram-gen-a rb-obs-histogram-gen-b \
  rb-obs-histogram-gen-c 2>/dev/null || true

rm -rf "$HOME/rb-obs-histogram"

Production notes

A bucket layout change is a producer deployment, and it invalidates a window. In a change window, treat it like any other code deploy: it needs a restart of every replica, it lands progressively, and for the width of one rate window after each replica restarts, the quantile for that replica is not meaningful. Announce the timestamp in the change record so the next person to read a chart across it knows why it steps.

Put a boundary at the SLO, and one either side of it. A 300 ms SLO with DefBuckets has boundaries at 250 ms and 500 ms and nothing between, so the panel cannot distinguish a healthy 280 ms from a breached 480 ms. Boundaries at 250, 300 and 400 make the breach visible at the moment it happens. This is the cheapest reliability improvement available to most teams and it costs four extra series per label set.

Alert on the worst instance, chart the aggregate. The job-level p99 is the right number for a capacity conversation and the wrong number for a page. An alert rule built on the Task 6 pattern:

groups:
  - name: latency-slo
    rules:
      - alert: P99AboveSLO
        expr: |
          max by (route) (
            histogram_quantile(
              0.99,
              sum by (route, instance, le) (
                rate(http_request_duration_seconds_bucket[5m])
              )
            )
          ) > 0.3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'p99 above the 300ms SLO on {{ $labels.route }}'

Keep the rate window at least four times the scrape interval. At a 15s scrape interval, [5m] gives twenty samples per window. [1m] gives four, and the quantile visibly jitters. The jitter is not the service; it is the window.

Count the series before adding a label. Fourteen series per label set was the number you measured in Task 3. A histogram sliced by method, route and status at 4 by 12 by 6 is 3,456 series before anyone adds anything. One unbounded label makes that number a capacity incident.

What You Learned

The mean and the p99 of one distribution can differ by more than an order of magnitude, and only one of them is the SLO. You measured 0.156 s and 2.2 s from the same five minutes of the same metric.
histogram_quantile() is exact about the buckets and approximate about the data. The 20% error you measured came entirely from a 1.5-second-wide bucket with all the mass in one part of it, and it disappeared when you put a boundary where the mass was.
The inner sum by (...) decides what the quantile is of. Dropping le returns nothing at all; dropping instance hides a degraded instance behind its healthy peers.
The highest finite boundary is a ceiling, not a measurement. A p99 sitting flat on a round number while the mean moves is a tail that has left the histogram.
A layout change is fixed at process start and leaves the old series behind. Plan it as a deployment, and expect one rate window of meaningless quantiles on either side of it.

Objectives

Prerequisites