Histogram Buckets

At 04:18 a checkout service paged on a “p99 latency above 1s” alert. The on-call engineer opened the panel and saw the line flat at 60 ms. The alert was wrong. The engineer escalated to the metric. The metric was a summary, not a histogram, and the team had configured Grafana to aggregate pre-computed quantiles across instances — an operation that does not mean what it appears to mean. The team changed the source to emit a histogram. They chose 11 default buckets. They restarted the service. The alert went quiet.

This lesson is the first of six on Prometheus histograms. It covers what a bucket is, how the bucket layout shapes every follow-up query, and what the per-observation memory trade-off actually costs.

What it is

A Prometheus histogram is a set of counters that record how many observations fell within a series of ranges. Each range is a bucket, and each bucket is identified by a label le (“less than or equal to”) on the upper boundary of the bucket. The boundary is expressed in the unit of the metric — seconds for http_request_duration_seconds_bucket, bytes for http_request_size_bytes_bucket.

A real exposure from a Go service:

# HELP http_request_duration_seconds Time spent handling HTTP requests.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{method="GET",route="/checkout",status="200",le="0.005"} 24054
http_request_duration_seconds_bucket{method="GET",route="/checkout",status="200",le="0.01"}  33444
http_request_duration_seconds_bucket{method="GET",route="/checkout",status="200",le="0.025"} 100392
http_request_duration_seconds_bucket{method="GET",route="/checkout",status="200",le="0.05"}  120193
http_request_duration_seconds_bucket{method="GET",route="/checkout",status="200",le="0.1"}   130044
http_request_duration_seconds_bucket{method="GET",route="/checkout",status="200",le="0.25"}  138210
http_request_duration_seconds_bucket{method="GET",route="/checkout",status="200",le="0.5"}   141022
http_request_duration_seconds_bucket{method="GET",route="/checkout",status="200",le="1"}     143110
http_request_duration_seconds_bucket{method="GET",route="/checkout",status="200",le="2.5"}   144201
http_request_duration_seconds_bucket{method="GET",route="/checkout",status="200",le="5"}     144310
http_request_duration_seconds_bucket{method="GET",route="/checkout",status="200",le="10"}    144318
http_request_duration_seconds_bucket{method="GET",route="/checkout",status="200",le="+Inf"} 144320
http_request_duration_seconds_sum{method="GET",route="/checkout",status="200"}               53423.214
http_request_duration_seconds_count{method="GET",route="/checkout",status="200"}               144320

Three things to read from this:

1. There is one time series per <basename>_bucket per unique combination of the other labels. In the example above, method, route and status are the slicing labels; the bucket count is fixed at 12 (the 11 finite boundaries plus +Inf).
2. The _count series is the total number of observations. It equals the value of the le="+Inf" bucket at every scrape.
3. The _sum series is the sum of all observation values. It is monotonically increasing, exactly like a counter, because observations are always positive.

The bucket layout is the set of finite le values that the library was told to use. The layout is fixed when the instrumentation is initialised. Changing it later requires a restart of the producer; an in-process change after the metric is registered is a violation of the OpenMetrics specification that Prometheus will reject on scrape.

Why a sysadmin cares

Histograms are how Prometheus exposes distribution-shaped measurements: “how long do requests take, how big are responses, how full is the queue.” The alternative is a summary, which computes quantiles at the source and exposes them as gauges. Summaries cannot be aggregated correctly across instances. The histogram trades more storage for the ability to answer quantile questions across the fleet — and the trade-off is paid in series count.

The bucket layout is the bit the operator owns. The wrong layout makes the panel look healthy when the service is degraded, because the resolution in the SLO region is too coarse to show the problem. The right layout makes the incident visible at the moment it happens. A team that has never revisited its bucket layout since the first commit has a dashboard that lies at the tail.

The second operational cost is cardinality. A histogram is n+1 series per label set, plus _sum and _count. A histogram with 11 finite buckets exposed across 100 label combinations creates 1,300 time series. Add one more unbounded label — customer_id — and that number goes to 1.3 million. The cardinality budget lesson returns to the maths.

How it works

The mental model is a stack of counters, each one a superset of the one below:

le="+Inf"   144320  <-- every observation lands here
le="10"     144318  <-- 2 observations were above 10s
le="5"      144310  <-- 8 more between 5s and 10s
le="2.5"    144201  <-- 109 more between 2.5s and 5s
le="1"      143110  <-- 1091 more between 1s and 2.5s
le="0.5"    141022  <-- 2088 more between 0.5s and 1s
le="0.25"   138210  <-- 2812 more between 0.25s and 0.5s
le="0.1"    130044  <-- 8166 more between 0.1s and 0.25s
le="0.05"   120193  <-- 9851 more between 0.05s and 0.1s
le="0.025"  100392  <-- 19801 more between 0.025s and 0.05s
le="0.01"   33444   <-- 66948 more between 0.01s and 0.025s
le="0.005"  24054   <-- 9390 more between 0.005s and 0.01s
                ^
                bucket boundary

The histogram does not store individual observations. It stores one cumulative counter per boundary. Every observation increments every counter whose boundary is at or above the observed value.

Two consequences of this layout:

1. Approximation, not truth. The histogram says “144,201 observations were at most 2.5s” but does not say how the observations were distributed within the 0.05s-0.1s bucket. The histogram_quantile() function assumes observations are uniformly distributed within a bucket. The smaller the bucket, the smaller the error from this assumption. Lesson 02 covers the inverse math.
2. Counting is constant-time per observation. Each observation walks the bucket list from the boundary downwards and increments the relevant counters. The cost is proportional to the number of buckets, not the number of observations. This is why clients pre-register the layout: it is hot-path code in the instrumented service.

The slicing labels

A histogram does not exist in isolation. Every bucket, the sum and the count carry the same other labels: method, status, route in the example above. The bucket layout slices across the label domain. With three labels that have cardinalities of 4, 12 and 6, the histogram exposes 22 * 13 = 286 time series for the _bucket metric alone, plus 22 each for _sum and _count — 330 series in total.

http_request_duration_seconds_bucket
  /- method (4 values: GET, POST, PUT, DELETE)
  /- status (6 values: 200, 201, 204, 400, 404, 500)
  /- route (12 values: /, /checkout, /cart, ...)
  /- bucket (11 finite boundaries + +Inf)
   = 4 * 6 * 12 * 12 = 3,456 time series

The bucket count is fixed by the layout. Everything else is the slicing cardinality — and the slicing cardinality is the thing a deployment can quietly change in one line of code. Lesson 04 returns to the per-observation cost when the slicing labels are unbounded (user_id, request_id, uuid).

Layout shapes

Three practical layouts and their uses:

The Go client library default — prometheus.DefBuckets — is five milliseconds to ten seconds in 11 exponentially-spaced steps:

.005 .01 .025 .05 .1 .25 .5 1 2.5 5 10

It is appropriate for typical web request latency. It is not appropriate for batch jobs, IO, or queue waits, which span milliseconds to minutes.

How to configure it

Three layers. Each layer has its own conventions and its own failure mode.

1. Source code (the instrumentation library).

The Go client library default is a sensible starting point:

import "github.com/prometheus/client_golang/prometheus"

var requestDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "Time spent handling HTTP requests.",
        // Buckets default to prometheus.DefBuckets:
        // .005 .01 .025 .05 .1 .25 .5 1 2.5 5 10
        // Override here for non-web workloads.
        Buckets: []float64{
            0.001, 0.005, 0.01, 0.025, 0.05,
            0.1,   0.25,  0.5,  1,     2.5,
            5,     10,    30,
        },
    },
    []string{"method", "route", "status"},
)

For Python (prometheus_client):

from prometheus_client import Histogram

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'Time spent handling HTTP requests',
    buckets=(
        0.001, 0.005, 0.01, 0.025, 0.05,
        0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30,
    ),
    labelnames=('method', 'route', 'status'),
)

The convention every histogram in the platform should follow:

• _seconds for latencies in seconds.
• _bytes for sizes in bytes.
• _ratio is not a histogram — use a gauge.

2. Prometheus scrape config (rate vs bucket accuracy).

prometheus.yml controls how Prometheus scrapes, not the bucket layout itself. The relevant options are scrape interval (which sets the resolution of the rate window) and honor_labels (which decides whether the producer’s labels override Prometheus’s job/instance labels). Histograms do not need anything special here:

scrape_configs:
  - job_name: checkout
    scrape_interval: 15s     # CONFIGURATION
    honor_labels: true       # preserve producer labels
    static_configs:
      - targets: ['checkout.svc:8080']

For remote-write targets that emit native histograms (Prometheus 2.50+ experimental), the relevant option is in the receiving Prometheus:

# prometheus.yml on the receiver (CONFIGURATION)
global:
  external_labels:
    cluster: payments

The receiver must be started with --enable-feature=native-histograms to ingest them. Without the flag, native histogram observations are dropped on scrape.

3. Grafana dashboards.

A histogram panel does not draw buckets directly. It calls histogram_quantile() over rate() of buckets:

# p99 latency over 5 minutes, per route
histogram_quantile(
  0.99,
  sum by (route, le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

The dashboard layer has no bucket configuration of its own. The buckets are fixed by the source. The lesson returns to this query in lesson 02.

How to validate it

Three layers of validation, each catching a different failure mode.

1. The metric is exposed at the source.

# READ-ONLY
curl -sf http://checkout.svc:8080/metrics \
  | grep '^http_request_duration_seconds_bucket' \
  | head

Expected output: 11 *_bucket lines per label set, with le boundaries increasing from the smallest to +Inf. If the boundaries are not monotonic, the OpenMetrics parser will reject the scrape.

2. The metric is in Prometheus.

# Count series per metric name (READ-ONLY)
count by (__name__) ({__name__=~"http_request_duration_seconds.*"})

Expected: three rows. _bucket should dominate the count, because of n+1 series per label set. _sum and _count should be equal to the number of unique label sets.

# Sample of the smallest finite bucket (READ-ONLY)
http_request_duration_seconds_bucket{le="0.005"}

The value should be a non-decreasing counter. A value that resets indicates a process restart; that is expected, but should be paired with a sane restart rate (rate(process_start_time_seconds[1h])).

3. The bucket layout is what the source was configured with.

# List unique le values in the histogram (READ-ONLY)
count by (le) (http_request_duration_seconds_bucket)

Expected: one row per boundary the source configured, including le="+Inf". If a boundary is missing from the output, the source was restarted with a different layout — the alert panels will silently change meaning.

The error shape when bucket boundaries differ across two sources of the same metric is the lesson of 03-cumulative-buckets. Two services that emit http_request_duration_seconds_bucket with different le boundaries cannot be summed safely.

How it can fail

Six failure modes, ordered by frequency in real environments.

1. Wrong unit suffix. A producer emits http_request_duration_milliseconds_bucket with le="500". The dashboard labels it “ms”, but Grafana plots the raw number as “ms”. The unit is milliseconds, not the seconds the label promised. Every downstream aggregation, alert and SLO is off by a factor of 1000. Symptom: p99 panel shows numbers that look plausible but disagree with the source code by three orders of magnitude.
2. Bucket layout too coarse for the SLO region. The default DefBuckets has no boundary between 0.1s and 0.25s. A service whose SLO is 150 ms gets no resolution at the boundary. Symptom: histogram_quantile() cannot distinguish 100 ms from 200 ms; the SLO breach alert fires only after the breach is gross.
3. Unbounded label added to a histogram. A developer adds customer_id to a histogram label set. The cardinality jumps from thousands to millions. Symptom: the head block memory grows within minutes, rule evaluation slows, and the Prometheus host OOMs.
4. Producer restarted with a different layout. A configuration change moves the bucket boundaries. The new series have new label sets; the old ones become stale and age out. Symptom: histogram_quantile() returns NaN for a few minutes until the old buckets expire from the head block, and the panel drops to “no data” mid-incident.
5. Buckets emitted unsorted. A bug in a custom instrumentation library emits boundaries in the wrong order. Symptom: scrape fails; Prometheus logs out of order sample and rejects the whole histogram family. The metric disappears from the platform until the bug is fixed.
6. Layout mismatch across aggregation. Two services emit the same metric name but with different bucket layouts. The team sums them with sum by (le). Symptom: the sum by (le) step is mathematically wrong because the boundaries do not align; the result is nonsense and the p99 panel cannot be trusted.

How to troubleshoot it

Diagnostic steps, ordered from cheapest to most expensive.

1. Confirm the metric is exposed. Curl the source /metrics. If the bucket lines are missing, the producer was never instrumented, or the process was restarted with a flag that disabled the histogram.
2. Confirm the layout matches the code. count by (le) (http_request_duration_seconds_bucket) should show exactly the boundaries the source declared.
3. Confirm Prometheus is scraping the source. up{job="checkout"} should be 1. The scrape_samples_scraped metric tells you how many samples per scrape; if it falls after a layout change, the producer is rejecting some boundaries.
4. Confirm the panel is reading the right metric. Open the panel’s query inspector. If it reads ..._bucket{le="+Inf"} and ignores the other boundaries, it is showing the request count, not the latency.
5. Confirm cardinality is within budget. topk(10, count by (__name__) ({__name__=~"http_request_duration_seconds.*"})) shows the spread. A histogram whose _bucket count dwarfs its _count is a slicing-cardinality problem.

Security implications

A histogram exposes labels. The same labels that slice the histogram into thousands of series are labels that may carry sensitive values: customer_id, email, tenant_id. The OpenMetrics scrape is unauthenticated by default on every client library; if the producer is reachable, the histogram is readable.

Three operational disciplines:

• Treat high-cardinality labels as PII risk before adding them to a histogram. A label that resolves to a unique person becomes a unique series per person.
• Restrict /metrics to the scrape network. Bind the exporter to the internal interface only. Use a sidecar scraper if the scrape source is public.
• Strip or hash sensitive labels before the histogram is exposed. A customer_id of 42 is a label; a hash of 42 is the same cardinality budget cost without the identifier leak.

The platform security part of the course covers authentication of scrape endpoints and the abuse shape of a public /metrics.

Performance implications

The histogram is the most expensive metric type in Prometheus. The cost is paid in three places:

1. Producer CPU. Each observation walks the bucket array. A 12-bucket histogram is roughly 10x more expensive than a counter; a 30-bucket histogram is roughly 25x more. The cost matters in hot paths.
2. Producer memory. The bucket array is allocated once per label set. A histogram with 1000 unique label sets and 12 buckets holds 12000 atomic uint64s (96 KiB) for the bucket arrays, plus the _sum and _count cells. Most of the memory cost is the bucket array itself; the rest is the label-keyed map.
3. Platform storage. Each _bucket is a separate time series in the head block. A 12-bucket histogram across 1000 label sets is 12,000 _bucket series plus 2000 _sum/_count — roughly 14,000 series per histogram metric. The cardinality budget lesson covers the cost per series.

The trade-off is unavoidable: finer buckets cost more, both in the producer and the platform. The lesson returns to bucket layout choice in lesson 06.

Production guidance

• Default to DefBuckets for web request latency. It is appropriate for the typical 5 ms to 10 s range. Do not reinvent it.
• Customise only when the workload is not a web request. Batch jobs, IO, queue waits and disk-backed operations need wider or differently-spaced layouts. Lesson 06 covers the choice.
• Unit suffix is part of the contract. A histogram named *_seconds_bucket must be in seconds. A histogram named *_bytes_bucket must be in bytes. The unit in the name and the unit in the buckets must agree.
• Slicing labels are a cardinality decision. Every label added to a histogram is multiplied by n+1 series. Lesson 04 covers the per-observation cost.
• Validate before relying. promtool check metrics is not a thing, but curl /metrics | grep is. Validate the layout in staging before the service ships.
• Document the choice. The instrumentation guide for the service should name the histogram, its boundaries, its slicing labels, and its unit.

Verification

You should now be able to answer:

• What does the label le="..." mean in a Prometheus histogram bucket?
• How many time series does a 12-bucket histogram with 100 unique label sets create?
• What is the difference between DefBuckets, an exponential layout and a custom layout?
• Which slicing labels multiply the series count?
• How do you validate that the live bucket layout matches what the source was configured to emit?

Quiz