Quantile Aggregation

An SLO review board asks the on-call team: “What is the p99 checkout latency across the fleet for the last 30 days?” The operator opens Grafana. The dashboard panel is titled “p99 checkout latency.” The number is 320ms. The SLO is 500ms. The team claims the SLO is met. The compliance reviewer pulls the raw traces and finds that one in twenty checkout requests took 2.4 seconds. The dashboard was not wrong in the way the operator thought. It was averaging per-instance quantiles, and the “average p99” is a quantity the math does not support. The SLO is not met. The audit fails.

Quantile aggregation is the place where the math, the implementation, and the dashboard convention disagree. The correct form is two lines of PromQL that look almost like the wrong form. The wrong form is the more common one in production.

What quantile aggregation is

Three concepts sit under the heading:

1. Histogram quantiles. A histogram is a counter of bucket counts. histogram_quantile(φ, vector) takes a vector of bucket counts and computes the φ-quantile by linear interpolation within the bucket that contains it. The vector must carry the le label.
2. Summary quantiles. A summary is a pre-aggregated set of φ-quantile values computed inside the source process. The quantile="0.99" label carries the value. They cannot be combined across instances by any operator.
3. group() modifier. A label-manipulation operator that collapses every series into one and emits a synthetic constant label group="<constant>" (or whatever name you supply). It is not an aggregation in the reducer sense; it is a label modifier.

The first two are what most operators reach for. The third is occasionally useful when a downstream system requires a label the aggregation would otherwise drop.

Why a sysadmin cares

Three production questions that quantile aggregation answers:

• “What is the p99 latency across the fleet?” — histogram_quantile(0.99, sum by (le) (rate(bucket[5m])))
• “What is the worst-offender p99 per host?” — histogram_quantile(0.99, sum by (instance, le) (rate(bucket[5m])))
• “What does the latency distribution look like?” — plot the bucket counts directly, or expose an approximate_percentile from the OTel collector.

The first two are daily vocabulary. The third is what the on-call engineer reaches for when the histogram buckets reveal a shape the mean hides.

How it works

The mental model for histogram_quantile is two steps: combine the buckets, then interpolate.

input: bucket counters per (job, instance, le)
                        |
                        v
        sum the bucket counters across the grouping set
        (so one histogram represents the whole group)
                        |
                        v
        find the bucket whose [low, high] contains φ
                        |
                        v
        linearly interpolate within that bucket
                        |
                        v
        emit the value as the φ-quantile

The input to histogram_quantile must be a vector with one series per bucket boundary (le label). The reducer that prepares the input — almost always sum by (le) — combines the bucket counters across the grouping dimension. If you skip the combination and pass the raw per-instance buckets, the operator interpolates within each instance’s histogram separately and emits one series per instance; that is useful for per-host panels, not for fleet-wide p99.

wrong (averages per-instance p99):
  per-instance p99:  histogram_quantile(0.99, rate(bucket{job="api"}[5m]))
  fleet "average":  avg(per-instance p99)

correct (combines buckets, then computes one quantile):
  fleet p99:        histogram_quantile(0.99, sum by (le) (rate(bucket{job="api"}[5m])))

The two expressions return different numbers even on identical input data. The first is the arithmetic mean of three p99s. The second is the 99th percentile of the combined distribution. In a fleet where one host is slow and twelve are fast, the first returns “200ms-ish” and the second returns “2.4 seconds.” The audit difference is the difference between “SLO met” and “SLO breached.”

What you cannot do with summaries

A summary’s quantile="0.99" value is computed inside the source process. The process has access to the raw observation stream and can compute the quantile exactly (within the configured window). Prometheus cannot combine those pre-computed quantiles across instances because the underlying distributions have been discarded. The math is irreversible: the p99 of the combined distribution is not derivable from the per-instance p99s without the raw observations.

The on-the-fly alternatives that summaries preclude:

  approximate_percentile (OTel collector, t-digest):
    combines across hosts via a mergeable sketch

  classic histogram + histogram_quantile:
    combines by summing bucket counters

  native histogram (Prometheus 2.50+):
    combines by summing zero-threshold and sparse bucket counters

All three are mathematically valid combinations. The summary’s pre-aggregated quantile is not, because the source has thrown away the data needed to combine.

group() versus aggregation

group() is occasionally confused with an aggregation operator. It is a label modifier. sum by (instance) (x) reduces the input to one series per instance. group without (instance) (x) collapses the input into one series whose only label is group="..." (the constant you supply). The vector length drops to one; the sample value is the input vector’s sample at the timestamp.

The use case is downstream systems (alertmanager, recording rules, pushgateway consumers) that require a specific label to be present. group() is the cheapest way to add a constant label to a vector.

How to configure it

The production pattern is to precompute the fleet-wide quantile in a recording rule, with le kept in the by clause.

# /etc/prometheus/rules/quantiles.yml
groups:
  - name: per-instance-quantile
    interval: 30s
    rules:
      # Per-instance p99. Keep `le` so histogram_quantile can interpolate.
      - record: instance:http_request_duration:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (job, instance, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

      # Per-instance p50.
      - record: instance:http_request_duration:p50
        expr: |
          histogram_quantile(
            0.5,
            sum by (job, instance, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

  - name: per-job-quantile
    interval: 30s
    rules:
      # Fleet p99 per job. Combine buckets across instances, then quantile.
      - record: job:http_request_duration:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

      # Cluster-wide p99. Combine across all dimensions.
      - record: cluster:http_request_duration:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

The pattern is consistent across levels: combine buckets at the level you want, then call histogram_quantile. The le label must survive the combination — it is the bucket boundary that the operator needs to interpolate within.

Choosing the histogram type

The summary row is the trap. A summary seems cheaper and simpler. It is, until the operator tries to compute fleet-wide quantiles. The choice is irreversible on the source side — summaries cannot be combined. The default in new code is the classic histogram, with native histograms where supported.

How to validate it

# 1. Static check.
promtool check rules /etc/prometheus/rules/quantiles.yml
# SUCCESS: /etc/prometheus/rules/quantiles.yml

# 2. Confirm the per-instance p99 emits one series per instance.
curl -s 'http://prometheus:9090/api/v1/query?query=instance:http_request_duration:p99' \
  | jq '.data.result | length'
# 17

# 3. Confirm the per-job p99 emits one series per job.
curl -s 'http://prometheus:9090/api/v1/query?query=job:http_request_duration:p99' \
  | jq '.data.result | length'
# 3

# 4. Confirm the per-job p99 is NOT the avg of per-instance p99s.
#    If the per-job number equals the avg of per-instance numbers,
#    the rule is computing the wrong form.
curl -s 'http://prometheus:9090/api/v1/query?query=job:http_request_duration:p99' \
  | jq -r '.data.result[].value[1]' | sort -n
# 0.183
# 0.214
# 0.087

curl -s 'http://prometheus:9090/api/v1/query?query=avg by (job) (instance:http_request_duration:p99)' \
  | jq -r '.data.result[].value[1]'
# 0.156
# 0.198
# 0.094

#    The two rows differ. The second is wrong.

The fourth step is the diagnostic check. The correct rule combines buckets; the wrong rule averages per-instance quantiles. Both numbers exist in the data; only the first reflects the fleet-wide p99.

How it can fail

The high-frequency failure modes for quantile aggregation:

1. Averaging per-instance p99. avg by (job) (histogram_quantile(0.99, ...)) is the most common production mistake. The number it returns is the arithmetic mean of the per-instance p99s, which is unrelated to the fleet-wide p99. The SLO claim built on this number is wrong.
2. Histograms with sparse buckets. A histogram with buckets at 0.01, 0.1, 1, 10 has too few buckets to interpolate accurately. The p99 the operator reads is approximate, and the approximation is coarse when the latency distribution is dense. Add buckets where the distribution lives.
3. histogram_quantile on a histogram with one bucket. The operator calls histogram_quantile on a metric that has only {le="+Inf"}. The result is +Inf. The dashboard renders infinity; the alert does not fire. The metric needs bucket boundaries the operator can interpolate within.
4. Combining summaries. The operator tries to “average p99 across instances” with summaries. The result is the arithmetic mean of pre-computed quantiles. The math is wrong. The only fix is to change the source metric from summary to histogram.
5. NaN propagation. histogram_quantile returns NaN when the input has zero observations. An alert that reads histogram_quantile(0.99, ...) > 1 evaluates NaN > 1, which is false. The alert does not fire even though there are no requests to alert on. Filter NaN explicitly.
6. Native histogram interpretation. A native histogram returns quantile values differently from a classic histogram. Tools that consume the metric may expect one form. Verify that the panel queries work with the new format before deploying native histograms.

How to troubleshoot it

When a quantile panel “looks low” or an SLO claim feels unsupported:

1. Inspect the per-instance p99 distribution. A fleet where eleven hosts have p99 = 100ms and one host has p99 = 5 seconds will produce an average around 540ms. The fleet-wide p99 is around 5 seconds. The discrepancy tells you whether the rule is averaging or combining.
2. Compare the rule output to a hand-computed value. Pick a 5-minute window. Pull all bucket counters from the /api/v1/query?query=... endpoint. Sum them by le. Interpolate manually. Compare to the recording-rule output. They should match within rounding.
3. Inspect the histogram bucket coverage. A histogram with {le="0.01", le="0.1", le="1", le="10", le="+Inf"} cannot represent a 99th percentile between 1 and 10 seconds with much accuracy. The fleet-wide p99 is in that range. Add buckets.
4. For alerts, prefer the upper bound of the bucket. When the p99 is uncertain due to coarse buckets, alert on the upper bucket boundary, which is a safe over-estimate.

Security implications

• Histograms expose the distribution of latencies. They do not expose payload contents. The risk is cardinality (one series per bucket boundary per label combination), not data leakage.
• Native histograms in Prometheus 2.50+ are denser but expose the same distribution shape. Treat them with the same cardinality discipline as classic histograms.
• Summaries cannot be combined; if a service emits summaries for any sensitive measurement, the team has committed to per-instance dashboards for that measurement. The cardinality cost may grow.

Performance implications

• histogram_quantile is O(B log B) per evaluation, where B is the number of buckets. A classic histogram with 12 buckets per metric, 5 metrics, and 50 label combinations is 12 × 5 × 50 = 3,000 input series per timestamp. The rule evaluator handles this in microseconds. Native histograms are denser and faster.
• A panel that re-evaluates histogram_quantile(0.99, sum by (le) (rate(bucket[5m]))) on every refresh is a panel that re-sums the bucket counters every refresh. Move the aggregation to a recording rule.
• Bucket cardinality is the largest single performance lever. 12 buckets is the typical default; 30+ is common for fine-grained latency analysis. Each bucket is a series.

Verification

You should now be able to answer:

• Why is averaging per-instance p99s mathematically wrong?
• What is the correct form for fleet-wide quantiles?
• What is the difference between a histogram and a summary?
• What does group() do, and when is it useful?

Quiz