Cumulative Buckets

At 06:47 the SRE team is called into a post-incident review. A user reported a slow checkout at 03:12. The p99 panel showed nothing abnormal at that timestamp. The on-call engineer pulled a Grafana explore view of the bucket metric and read the values. They were increasing — every bucket count was higher than it had been at 03:11. But the per-bucket count (the number of observations that fell in each interval) was flat. The engineer had been looking at cumulative counts, which always grow. The bucket rate was flat. The service was not slow. The user was. The post- incident review asks: how do we make sure the next engineer sees the per-bucket rate, not the cumulative count, the first time?

This lesson is the cumulative-bucket math: why le="..." is defined as it is, why the bucket list is monotonic, and what the difference between le and a “leftover” boundary is.

What it is

A Prometheus histogram bucket is cumulative. The label le="X" declares an upper bound; the value at le="X" is the count of observations whose value is less than or equal to X (in the unit of the metric). Every bucket count is a superset of the bucket count below it:

http_request_duration_seconds_bucket{le="0.005"}  24054
http_request_duration_seconds_bucket{le="0.01"}   33444   > 24054
http_request_duration_seconds_bucket{le="0.025"}  100392  > 33444
http_request_duration_seconds_bucket{le="0.05"}   120193  > 100392
http_request_duration_seconds_bucket{le="0.1"}    130044  > 120193
http_request_duration_seconds_bucket{le="0.25"}   138210  > 130044
http_request_duration_seconds_bucket{le="0.5"}    141022  > 138210
http_request_duration_seconds_bucket{le="1"}      143110  > 141022
http_request_duration_seconds_bucket{le="2.5"}    144201  > 143110
http_request_duration_seconds_bucket{le="5"}      144310  > 144201
http_request_duration_seconds_bucket{le="10"}     144318  > 144310
http_request_duration_seconds_bucket{le="+Inf"}   144320  >= 144318

The monotonic property is enforced by the OpenMetrics specification and validated by the Prometheus scrape parser. A scrape that emits le boundaries out of order is rejected as malformed; the entire family is dropped.

The shape has a name: cumulative. Each bucket count contains all the observations in the bucket below it. The number of observations that fall strictly between two boundaries is the difference:

le="0.01"   33444
le="0.005"  24054
between (0.005, 0.01] = 33444 - 24054 = 9390 observations

The difference is sometimes called the leftover or per-bucket count. It is the histogram’s “this is how many observations fell in this interval” number. It is not exposed as a separate metric — it is computed by subtraction in the query.

Why a sysadmin cares

The cumulative shape is what makes the histogram queryable. Three operational disciplines follow from it.

1. Always compute the per-bucket count by subtraction, not by reading the bucket directly. A panel that reads http_request_duration_seconds_bucket{le="0.1"} shows the cumulative count of every observation at or below 0.1s. That number always grows. It tells the operator nothing about the current rate of slow requests. The operator wants rate(...) of the per-bucket count, which is `rate(http_request_duration_seconds_bucket{le=“0.1”}[5m])
   • rate(http_request_duration_seconds_bucket{le=“0.05”}[5m])`.
2. Always apply rate() inside histogram_quantile(). The function expects a vector of rates, not a vector of cumulative counts. A panel that reads histogram_quantile(0.99, http_request_duration_seconds_bucket) treats the cumulative counts as instantaneous counts and produces nonsensical p99 values that grow forever.
3. Never sum buckets across heterogeneous sources. Two services emit the same metric with different boundaries. The le vectors are not aligned; the sum is malformed. The platform enforces this only at the scrape level, not at the query level.

The lesson is short because the math is short. The cost of not knowing it is panels that lie and post-incident reviews that start with “what was the engineer looking at, exactly?”

How it works

Three rules follow from the cumulative definition.

Rule 1: the bucket list is monotonic in le.

The OpenMetrics text format requires that, within a single histogram family, the le boundaries be strictly increasing (or strictly equal at a duplicate boundary, which is a parser error). Prometheus validates this on scrape. A producer that emits le="0.5" before le="0.1" causes the entire family to be dropped with a parse error in the Prometheus logs.

Rule 2: the +Inf bucket is the total observation count.

The +Inf bucket is the cumulative count of every observation. The _count series emits the same value as the +Inf bucket at every scrape. Both are monotonically increasing; both reset on a process restart. The relationship holds in the OpenMetrics specification and is checked by the Prometheus client libraries in their test suites.

Rule 3: within-bucket observations are assumed uniform.

The histogram does not store where in the bucket each observation fell. histogram_quantile() assumes a uniform distribution within the bucket width. The smaller the bucket, the smaller the assumption error. A bucket from 0.05 to 0.1 has 50 ms of width; a bucket from 1 to 2.5 has 1.5 s. The same fractional uncertainty in the rank calculation produces a larger absolute error in the wider bucket.

le vs “leftover”

The boundary label le="X" means “less than or equal to X”. The number of observations that fell strictly above the previous boundary and at or below this one is the leftover or per-bucket count:

# Cumulative counts (what the source emits)
le="0.005"  24054
le="0.01"   33444

# Per-bucket count (what the panel wants)
rate((http_request_duration_seconds_bucket{le="0.01"}[5m])
   - (http_request_duration_seconds_bucket{le="0.005"}[5m]))

The PromQL idiom for the per-bucket rate uses histogram_quantile() directly: it walks the cumulative vector internally and subtracts to find the within-bucket count. A panel that wants to show which buckets are getting hit (a heatmap) sums the per-bucket count across labels:

# Heatmap data — per-bucket rate summed across instances
sum by (le) (
  rate(http_request_duration_seconds_bucket[5m])
)

The query returns one row per le, with the value being the rate of observations that fell at or below that boundary. The heatmap renders the bucket distribution directly.

How to configure it

The cumulative shape is not configured; it is a property of the histogram metric type. The configurable knobs are the boundary layout (lesson 06) and the rate window that is applied at query time.

Validate the cumulative property on a live metric:

# Bucket values must be non-decreasing in le (READ-ONLY)
sort(
  http_request_duration_seconds_bucket{service="checkout"}
) by (le)

The sort() function returns the bucket vector in ascending order of le. If the resulting values are strictly non-decreasing, the cumulative property holds.

Compute the per-bucket count by subtraction:

# Per-bucket rate of observations between le="0.05" and le="0.1"
# (READ-ONLY)
rate(http_request_duration_seconds_bucket{le="0.1"}[5m])
- rate(http_request_duration_seconds_bucket{le="0.05"}[5m])

The result is the number of observations per second that fell in the (0.05, 0.1] interval over the last 5 minutes.

Heatmap of bucket rates:

# Sum across instances; each row is one bucket's rate
# (READ-ONLY)
sum by (le) (
  rate(http_request_duration_seconds_bucket[5m])
)

This is the heatmap data source. The values are cumulative counts per second, but the heatmap visualisation treats each row as a bucket and shades it by the rate.

Alert on a high rate of observations in a tail bucket:

# alert.rules.yml
groups:
  - name: checkout-tail-latency
    rules:
      - alert: HighRateSlowRequests
        expr: |
          (rate(http_request_duration_seconds_bucket{le="1"}[5m])
           - rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
          > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'More than 10 requests/sec in (0.5s, 1s] bucket on checkout'

How to validate it

Four checks, each catching a different mistake.

1. Confirm le boundaries are present and sorted.

# One row per le boundary (READ-ONLY)
count by (le) (http_request_duration_seconds_bucket)

If a row is missing, the producer was restarted with a different layout. If the order is wrong in the /metrics output, the producer is violating OpenMetrics.

2. Confirm the cumulative property holds.

# The le="0.25" count must be >= the le="0.1" count, etc.
# (READ-ONLY)
http_request_duration_seconds_bucket{le="0.1"}
  > http_request_duration_seconds_bucket{le="0.05"}

The Prometheus server evaluates this and emits a row per label set. If the result contains any zero values, the cumulative property is violated; check the producer.

3. Confirm _count equals the +Inf bucket.

# _count and the +Inf bucket should agree (READ-ONLY)
http_request_duration_seconds_count
  == http_request_duration_seconds_bucket{le="+Inf"}

The expected result is 1 (true) for every label set. A zero indicates the metric is broken; a 2 (false) indicates the producer is emitting inconsistent state.

4. Confirm rate() is applied before histogram_quantile().

# The query should always look like this
histogram_quantile(
  0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

If the inner expression is missing rate(...), the histogram_quantile() is operating on raw counters and returning meaningless values.

How it can fail

Five failure modes that show up in production.

1. Panel reads raw cumulative count instead of rate. The query is http_request_duration_seconds_bucket{le="0.5"}. The panel renders the cumulative count of every observation at or below 0.5s. The number grows forever. Symptom: the panel climbs during healthy operation; on-call engineers stop looking.
2. Buckets summed across heterogeneous layouts. Two services emit http_request_duration_seconds_bucket with different le boundaries. The team sums them with sum by (le) (rate(...)). Symptom: the resulting le vector is the union of the two layouts; some boundaries are missing from one source; histogram_quantile() interpolates against a malformed distribution and the p99 is nonsense.
3. Bucket list reordered by accident. A refactor in the producer iterates over a map (which has non-deterministic order) and emits boundaries in random order. Symptom: the scrape fails; the family is dropped; the panel reads “no data” until the bug is fixed.
4. Rate window too short for the rate to be stable. rate(_bucket[15s]) against a 15s scrape interval samples one or two points; the rate is too noisy to drive an alert. Symptom: the alert fires intermittently; the team loses trust and silences it.

How to troubleshoot it

When a panel is wrong, the diagnosis order matters.

1. Open the panel query inspector. The query should apply rate() before histogram_quantile(). If it does not, fix it.
2. Confirm the cumulative property holds. The bucket values at scrape time must be non-decreasing in le. If they are not, the producer is broken.
3. Confirm +Inf and _count agree. A mismatch means the producer is emitting inconsistent state.
4. Confirm the boundaries are sorted in /metrics. If they are not, the producer is violating OpenMetrics; the family is being dropped.
5. Confirm all aggregated sources have the same layout. group by (le) (...) across all sources should yield the same le vector for every source. A mismatch is the operator for “this is not safe to sum”.

Security implications

The cumulative shape does not introduce a new attack surface. The metric exposes the same labels whether the operator reads it as cumulative or per-bucket. The risk is the same as for any histogram (lesson 01): high- cardinality labels become a per-person observation count when the label is a user identifier.

The operational concern is that a query that emits raw cumulative counts is rarely the query an attacker wants to scrape; the attacker wants per-tenant quantiles, which are computed from the same buckets. The mitigation is the same as for the underlying histogram.

Performance implications

The cumulative shape is the cheapest shape for storage and the most expensive for the producer.

• Storage. A histogram with N boundaries is N counters. There is no separate per-bucket-down counter; the per-bucket count is recovered by subtraction.
• Producer CPU. Every observation walks the bucket array and increments every counter whose boundary is at or above the observation. The cost per observation is the expected bucket index, which for a uniform distribution is N/2. A 30-bucket histogram costs roughly 15 atomic increments per observation.
• Query cost. histogram_quantile() walks the bucket vector linearly. The cost per evaluation is O(N) where N is the number of boundaries. A 12-bucket histogram is negligible; a 100-bucket histogram is noticeable.

The trade-off is unavoidable: the cumulative shape is what makes the histogram queryable, and it is what costs producer CPU.

Production guidance

• Always apply rate() before histogram_quantile(). The function operates on rates; the raw cumulative counts are not its input.
• Use histogram_quantile() for quantiles, not the raw bucket value. The raw value grows forever and has no operational meaning.
• Validate the cumulative property in staging. A refactor that iterates over a map is a common cause of re-ordered boundaries.
• Document the boundary layout in the producer. The team’s instrumentation guide should list the le boundaries every histogram uses, with the rationale for each.
• Audit the bucket boundaries across services. Two services emitting http_request_duration_seconds_bucket with different layouts cannot be aggregated safely.

Verification

You should now be able to answer:

• Why are Prometheus histogram buckets cumulative, and how is that enforced?
• What is the difference between a bucket count and a per-bucket count?
• Why must rate() be applied before histogram_quantile()?
• Why can two services with the same metric name but different bucket layouts not be summed safely?
• What does the _count series have in common with the +Inf bucket?

Quiz