histogram_quantile()

At 09:51 the on-call engineer is told that the API is “slow for some users”. She opens the latency panel. The p99 line is at 80 ms — well under the 200 ms SLO. She pages a senior engineer. The senior engineer notices that the panel is per cluster. There are four. One of them is at 740 ms p99. The other three are at 70 ms. The cluster-level aggregation hides the bad one. The SLO is breached; the alert is silent.

This is the operational shape of histogram_quantile(): it is the most-used query in any Prometheus deployment and the most-misused. The function is correct; the queries built on top of it are frequently not. This lesson is the inverse math, the aggregation discipline, and the failure modes that panels routinely hide.

What it is

histogram_quantile(φ, b) estimates the φ-quantile of a distribution from a vector b of bucket counts. The vector must include the le="+Inf" bucket; without it, the result is NaN.

The function takes two arguments:

• φ — the desired quantile, a real number between 0 and 1. 0.5 is the median; 0.95 is the 95th percentile; 0.99 is the 99th percentile.
• b — a vector of bucket counts. In practice, this is sum by (le) (rate(*_bucket[5m])) so that all instances contribute to a single per-le rate vector.

The canonical query is:

histogram_quantile(
  0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

histogram_quantile() is the inverse of a histogram’s forward operation. The forward operation is: given an observation, increment every counter whose boundary is at or above the observed value. The inverse is: given the counts, where did the φ-th observation fall?

Why a sysadmin cares

A latency SLO is the operational commitment the team has made to users. The SLO is breached when the p99 crosses a threshold. The threshold is meaningless without a quantile query that is correct. Three production failure shapes repeat:

1. Quantile that hides the worst cluster. A single sum by (le) across instances or clusters hides the tail because the larger cluster dominates the count. The p99 panel reads “global p99” and is, in fact, the p99 of the larger cluster’s distribution.
2. Quantile that returns NaN. A missing +Inf bucket makes the function return NaN. The panel reads “no data” rather than “broken query”; the on-call engineer is told to investigate an empty panel.
3. Quantile from a summary. A team uses summary and writes quantile{quantile="0.99"}. The metric is pre-computed at the source; the quantile function in PromQL cannot recombine across instances. The panel is the average of pre-computed quantiles, which has no operational meaning.

The single most important operational lesson of this part is: histogram_quantile() is per the grouping you keep in the inner sum by (...). The default is per cluster if you sum across instances and per instance if you keep the instance label.

Mental model

A simple example with five observations: 1, 2, 3, 4, 5. The true median is 3. A histogram with boundaries at 2, 4 and +Inf records these cumulative counts:

le="2"    2  <-- 1 and 2 land here
le="4"    4  <-- 3 and 4 are counted here too
le="+Inf" 5  <-- every observation is counted here

For the median, rank = 0.5 * 5 = 2.5. The first bucket whose cumulative count reaches 2.5 is le="4". That bucket spans 2 to 4 and holds 4 - 2 = 2 observations, and the rank sits 2.5 - 2 = 0.5 observations into it, so the estimate is 2 + (4 - 2) * (0.5 / 2) = 2.5. The true median is 3; the 0.5 error is the price of a two-boundary layout.

For the 99th percentile, rank = 0.99 * 5 = 4.95. The only bucket whose cumulative count reaches 4.95 is +Inf. That bucket has no finite upper bound to interpolate towards, so the function returns the upper bound of the second highest bucket: 4. It never reports a value above the highest finite boundary.

How it works

histogram_quantile() walks the bucket list to find the bucket where the φ-quantile falls, then interpolates within that bucket. The implementation, in plain prose:

1. Sort the buckets by le. The server sorts them before the function runs.
2. Compute the rank and find its bucket. The rank is φ multiplied by the count of the +Inf bucket. Walking from the smallest boundary upward, the quantile bucket is the first whose cumulative count is at or above the rank.
3. If the rank falls in the +Inf bucket, return the upper bound of the second highest bucket. There is no finite upper boundary to interpolate towards, so the highest finite boundary is the answer.
4. Otherwise interpolate inside that bucket. For classic histograms the function assumes the observations in the bucket are spread uniformly between its lower bound (the previous boundary) and its upper bound, and places the quantile at the matching fraction of the width. The lowest bucket is treated as starting at 0 when its upper bound is above 0; when that upper bound is at or below 0, the upper bound itself is returned.

The interpolation uses the count of the quantile bucket itself, not the running total:

quantile = lower_bound
         + (rank - lower_count)
           / (upper_count - lower_count)
           * (upper_bound - lower_bound)

lower_bound and lower_count come from the previous bucket, or 0 and 0 for the lowest bucket; upper_bound and upper_count come from the quantile bucket. Because the quantile bucket is by definition the first whose cumulative count reaches the rank, rank - lower_count is never larger than upper_count - lower_count. The fraction is therefore between 0 and 1, and the result can never leave the bucket. There is no clamping step inside the interpolation; the +Inf bucket in step 3 is the only special case.

Two worked examples

The values below are one sum by (le) (rate(...)) result. They are written as whole numbers totalling 1000 because the quantile depends only on their ratios, not on their scale.

  le        cumulative value
  0.01                    30
  0.05                   100
  0.1                    200
  0.5                    600
  1                      800
  5                      900
 10                      950
 +Inf                   1000

p93, interpolated. rank = 0.93 * 1000 = 930. The first cumulative value at or above 930 is 950, at le="10", so the quantile bucket runs from 5 to 10. Its predecessor le="5" holds 900, so the bucket itself holds 950 - 900 = 50 observations and the rank sits 930 - 900 = 30 observations into it:

quantile = 5 + (930 - 900) / (950 - 900) * (10 - 5)
         = 5 + (30 / 50) * 5
         = 5 + 3
         = 8

The p93 estimate is 8 seconds.

p99, at the highest finite boundary. rank = 0.99 * 1000 = 990. No finite bucket reaches 990; the first cumulative value at or above 990 is the +Inf bucket at 1000. The rank falls in the open-ended bucket, so the function returns the upper bound of the second highest bucket:

quantile = upper bound of le="10"
         = 10

The p99 estimate is 10 seconds, and it stays at 10 seconds however slow the slowest 5 percent of requests actually are. The highest finite boundary is the ceiling of every value histogram_quantile() can report.

The practical lesson: a p99 line sitting exactly on the highest finite boundary for hours is not a measurement, it is the ceiling. The tail has moved off the end of the histogram, and the layout needs boundaries above the SLO, or a native histogram, which has no fixed top boundary. Coarse boundaries lower down produce the same shape on a smaller scale: p50 and p99 panels that track each other because both are interpolating across the same wide bucket.

How to configure it

The query, not the file.

Per-instance p99 across all routes:

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

Per-route p99 across all instances in a job:

histogram_quantile(
  0.99,
  sum by (route, le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

Per-cluster p99 across all routes and instances:

histogram_quantile(
  0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

The pattern is: decide what you want p99 of, keep those labels in the sum by (...), drop the rest, and always keep le. Removing le from the sum by (...) is the most common query bug; the aggregation removes the boundaries the function needs, and no quantile comes back at all.

SLO burn-rate alert on p99:

# alert.rules.yml — alert when p99 of the worst route is
# above the SLO for 5 minutes.
groups:
  - name: latency-slo
    rules:
      - alert: P99AboveSLO
        expr: |
          histogram_quantile(
            0.99,
            sum by (route, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          ) > 0.2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'p99 latency above 200ms SLO for {{ $labels.route }}'
          description: 'p99 = {{ $value | humanizeDuration }}'

Heatmap panel in Grafana:

A heatmap plots the bucket rates themselves, not histogram_quantile(). Set the panel data source to Prometheus and use:

sum by (le) (
  rate(http_request_duration_seconds_bucket[$__rate_interval])
)

The $__rate_interval is Grafana’s auto-selected rate window based on the panel’s time range. The heatmap shows the bucket distribution directly and is the best tool for understanding whether the bucket layout is appropriate.

How to validate it

Three validation steps, each catching a different shape of broken query.

1. Confirm the function returns a finite value.

# The p99 should be a real number, not NaN (READ-ONLY)
histogram_quantile(
  0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

If the result is NaN, the +Inf bucket is missing or the histogram has no observations in the window.

2. Confirm the per-instance and per-cluster queries disagree when one instance is bad.

# Per-instance p99 (READ-ONLY)
histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

# Per-cluster p99 (READ-ONLY)
histogram_quantile(
  0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

In a healthy fleet, the per-cluster value is close to the per-instance median. When one instance is degraded, the per-instance max is far above the per-cluster value. The gap is the magnitude of the hidden problem.

3. Confirm the rate window is at least 4x the scrape interval.

A histogram_quantile query over rate(_bucket[1m]) against a 30s scrape interval is sampling 2 scrape intervals per evaluation, which makes the rate noisy. The rule of thumb is rate_window at least 4 * scrape_interval. A 15s scrape interval pairs with rate(_bucket[1m]) minimum.

How it can fail

Six failure modes, in the order they appear in real incidents.

1. Missing +Inf bucket. A producer emits _bucket only for the finite values. The function returns NaN. Symptom: the panel reads “no data”; the alert is silent; the team is unaware the metric is broken until an incident reveals the gap.
2. sum by () without le. A panel sums buckets without keeping le, so the aggregation strips the boundary label off every sample. histogram_quantile() has no bucket boundaries left to walk and cannot use the input. Symptom: the panel renders nothing and the query carries a warning, rather than showing a wrong number.
3. Pre-computed summary used as a quantile. A team confuses summary{quantile="0.99"} with a per-fleet p99. The summary is pre-aggregated at the source; quantile() over it averages the pre-computed values, which has no statistical meaning. Symptom: the panel renders a number; the number is the average of per-instance p99s, which can be lower than every per-instance p99.
4. Coarse bucket layout hides the tail. A service with a 150 ms SLO uses DefBuckets. The boundaries are at 100 ms and 250 ms with nothing in between. A regression to 200 ms goes into the le="0.25" bucket — the same bucket as a 240 ms regression. Symptom: the p99 panel shows no movement between 100 ms and 250 ms; the SLO is breached silently.
5. Buckets summed across heterogeneous layouts. Two services emit http_request_duration_seconds_bucket with different le boundaries. The team sums them with sum by (le). Symptom: the resulting le vector is the union of the two boundaries; histogram_quantile() interpolates against a malformed distribution; the p99 is nonsense.
6. Rate window too short for the scrape interval. A panel uses rate(_bucket[30s]) against a 15s scrape interval. The rate is sampled over one or two scrape intervals and is too noisy to be useful. Symptom: the p99 panel jumps around; the alert fires intermittently; the team loses trust in the metric.

How to troubleshoot it

When a p99 panel is wrong, the diagnosis order matters.

1. **Confirm the underlying histogram exists. **count by (le) (http_request_duration_seconds_bucket) should list the boundaries the source declared.
2. Confirm the +Inf bucket is present. It should be the last row, with the highest count.
3. Confirm the rate window is sane. rate(_bucket[1m]) against a 15s scrape interval covers four scrapes and survives a missed one. rate(_bucket[30s]) covers two, which is the arithmetic minimum and leaves no headroom: one failed scrape and the panel gaps.
4. Confirm the sum by (...) keeps le. Open the panel query inspector. The inner sum by (...) must include le.
5. Confirm the metric is a histogram, not a summary. http_request_duration_seconds{quantile="0.99"} is a summary. histogram_quantile() cannot combine it.
6. Compare per-instance and per-cluster quantiles. A wide gap is a flag that the panel is hiding a tail.

Security implications

histogram_quantile() is a query-side function. It does not expose data the metric does not already expose. The security implications are inherited from the underlying histogram (lesson 01) and from the query surface that runs the function.

The operational concern is who can read the panel. A panel that shows per-tenant p99 exposes the latency distribution of every tenant. Treat the query surface as part of the access model: a panel that a junior engineer can read is also a panel an attacker with the dashboard URL can read.

The platform security part of the course covers Grafana permissions and Prometheus query logging.

Performance implications

histogram_quantile() is not free. The cost is paid per evaluation, not per scrape.

• Bucket walk. The function walks the bucket vector linearly. With 12 buckets and one label set, the cost is negligible. With 12 buckets and 10,000 label sets, the cost is 120,000 evaluations per evaluation cycle.
• Rate window. A 5m rate window against a 15s scrape interval is 20 samples. Each sample contributes to the rate calculation; the calculation is the dominant cost.
• Recording rule cache. Every dashboard panel that uses histogram_quantile() evaluates the query on render. A panel that runs every 30s for 10 dashboards is 20 evaluations per minute. A recording rule that precomputes the per-route p99 collapses this to a single evaluation per evaluation interval.

The trade-off is the standard one: dashboards that load fast and stay correct rely on recording rules for any quantile or rate that is computed more than a handful of times per minute.

Production guidance

• Always keep le in the sum by (...). A panel query that loses le is silently wrong.
• Validate p99 panels against a known distribution. Inject a synthetic load (lesson 06 covers loadtest-style tools) and confirm the panel reads the expected value.
• Use a rate window that is at least 4x the scrape interval. A 15s scrape interval pairs with rate(_bucket[1m]) minimum.
• Compare per-instance and per-cluster quantiles in every review. A wide gap is a sign the panel is hiding a tail.
• Move p99 panels behind recording rules. A panel that re-evaluates histogram_quantile() per render is an unnecessary cost.
• Document the aggregation level. The team’s instrumentation guide should say “the per-route p99 is histogram_quantile() over sum by (route, le)” so the next panel author does not reinvent (wrongly).

Verification

You should now be able to answer:

• What does histogram_quantile() assume about the distribution within a single bucket?
• Why does a missing +Inf bucket return NaN?
• What is the difference between a per-instance p99 and a per-cluster p99 query?
• What does histogram_quantile() return when the rank falls in the +Inf bucket, and why is that the ceiling of every value it can report?
• Why does quantile() over a summary return a wrong value?
• What rate window should be paired with a 15s scrape interval?

Quiz