Choosing the Bucket Layout

At 14:08 a team deploys a new payments service. The instrumentation uses prometheus.DefBuckets. The SLO is “p99 below 300 ms”. A regression at 16:22 lifts the p99 to 180 ms. The SLO alert does not fire because 180 ms is below 300 ms. The next regression at 17:01 lifts the p99 to 290 ms — also below the SLO. The customer-impacting regression at 17:45 lifts the p99 to 410 ms, which trips the SLO and the alert. Three regressions were missed because the default bucket layout had no boundary between 250 ms and 500 ms.

This lesson is the bucket layout choice. The default layout is appropriate for typical web request latency. For other workloads — DB queries, IO, queue waits, batch jobs — the layout needs to be chosen with the SLO and the tail in mind.

What it is

The bucket layout of a Prometheus histogram is the sorted set of finite le boundaries the producer was configured with. The layout is fixed at process start; it is not a runtime configuration. The choice of boundaries determines three things:

1. The resolution at the SLO region. A boundary at 200 ms means the p99 panel can distinguish a 100 ms regression from a 300 ms regression. A boundary at 250 ms with the next at 500 ms means the panel cannot.
2. The cost on the producer. Each observation walks the bucket array; the cost is O(N) where N is the number of buckets. A 12-bucket histogram is roughly 10x more expensive than a counter; a 30-bucket histogram is roughly 25x more.
3. The cardinality cost on the platform. Each boundary is one more time series per label set. A 30-bucket histogram with 1,000 label sets is 32,000 series (30 buckets + +Inf + _sum + _count per label set).

The trade-off is between resolution and cost. The wrong choice shows up in two shapes:

• Too few boundaries in the SLO region. The panel cannot distinguish a healthy p99 from a breached p99; the SLO alert is silent during regressions.
• Too many boundaries in the tail. The producer pays for buckets that almost never get hit; the platform pays for series that record almost nothing.

The art is the choice.

Why a sysadmin cares

The bucket layout is the bit of the histogram the operator owns. The wrong layout makes the dashboard lie; the right layout makes the incident visible at the moment it happens. Three operational disciplines follow:

1. Default to DefBuckets for web request latency. The Go client library default is five milliseconds to ten seconds in eleven exponential steps. It is appropriate for the typical web request range.
2. Customise the boundaries for non-web workloads. DB queries, IO, queue waits and batch jobs span ranges that the default does not cover.
3. Put boundaries at the SLO. A panel that needs to distinguish 200 ms from 300 ms needs a boundary at 250 ms (or both). A panel that needs to distinguish 5 s from 10 s needs a boundary at 7.5 s (or both).

The lesson is short because the choice is short: pick the boundaries that the SLO needs, in the range the workload spans, and document the choice.

Mental model

For a web request with SLO 200 ms:

DefBuckets:
  5ms  10ms 25ms 50ms 100ms 250ms 500ms 1s 2.5s 5s 10s
                        ^
                        SLO boundary? 250ms is too coarse
                                      for a 200ms SLO.

A better layout for a 200 ms SLO:

SLO-aligned:
  10ms 25ms 50ms 75ms 100ms 150ms 200ms 250ms 500ms 1s 2.5s 5s
                  ^       ^      ^      ^     ^
                  dense in the SLO region, coarse in the tail

The SLO region (50 ms to 250 ms) has boundaries every 25-50 ms; the tail (250 ms to 5 s) has boundaries every 250-1000 ms. The histogram resolves the SLO breach precisely and stops resolving once the breach is past 500 ms — because nothing operational changes between 500 ms and 1 s.

How it works

The bucket layout is a sorted slice of float64 values in the producer. Three helpers in the Go client library and the Python client cover most workloads.

prometheus.DefBuckets — eleven exponential boundaries from 5 ms to 10 s:

.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10

Appropriate for typical web request latency. The boundaries are spaced by factors of 2^(1/2) between 0.1 and 1 s, and by factors of 2.5 between 1 and 10 s. The shape is the default and should be the choice unless the workload is unusual.

prometheus.LinearBuckets(start, width, count) — N evenly-spaced boundaries starting at start:

LinearBuckets(0, 100, 30) ->
  0, 100ms, 200ms, ..., 2900ms

Appropriate for narrow distributions with known step size: a single microservice hop, a queue with predictable drain time, a periodic batch operation with known cadence.

prometheus.ExponentialBuckets(start, factor, count) — N exponentially-spaced boundaries starting at start:

ExponentialBuckets(0.001, 2, 16) ->
  1ms, 2ms, 4ms, 8ms, 16ms, ..., 32.768s

Appropriate for wide distributions: IO latency, queue waits, batch jobs, anything that spans three or more orders of magnitude.

Custom boundaries — a hand-picked slice:

SLOBuckets ->
  10ms, 25ms, 50ms, 75ms, 100ms, 150ms, 200ms,
  250ms, 500ms, 1s, 2.5s, 5s

Appropriate when the SLO region is known and the tail is bounded.

The “your tail is bounded” heuristic

The right layout for a workload has three parts:

1. A lower bound. The smallest value the metric meaningfully takes. Below this, the value is noise.
2. A dense region around the SLO. Boundaries every 10-25% of the SLO target.
3. A bounded tail. The largest value the metric is expected to take. Beyond this, the value is pathological and the operator is paging anyway.

The “your tail is bounded” heuristic says: pick the largest boundary to be the value above which the operator does not need to distinguish. A web request with SLO 200 ms and a “the user has given up” timeout at 30 s does not need boundaries between 1 s and 30 s; one boundary at 5 s and another at 30 s is enough.

Default Pause-Resume pattern

A common shape in Go services is to start with DefBuckets and revisit the layout when the first SLO review shows that the default is wrong. This is the “Default Pause-Resume” pattern:

1. Default to DefBuckets — get the histogram emitted and scraped with a known-working set of boundaries.
2. Pause to instrument the panel — observe the bucket distribution in production for a week; identify the SLO region and the tail.
3. Resume with a custom layout — replace DefBuckets with a layout that resolves the SLO and bounds the tail.

The discipline is to not start with a custom layout. The default is the right starting point; the custom layout is the result of measurement.

How to configure it

Three patterns, each matched to a workload class.

1. Web request latency (the default case).

Use DefBuckets. No customisation needed:

import "github.com/prometheus/client_golang/prometheus"

var requestDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "Time spent handling HTTP requests.",
        Buckets: prometheus.DefBuckets,
    },
    []string{"method", "route", "status"},
)

The default is fine for most web services. The boundary at 250 ms and the next at 500 ms is too coarse for a 200 ms SLO; in that case, add a boundary at 200 ms.

2. Database query latency (the wide-tail case).

DB queries span 1 ms (cached lookup) to 30 s (table scan on a hot row). The default’s upper bound of 10 s is too low; the lower bound of 5 ms is too high. A custom layout:

var dbQueryDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "db_query_duration_seconds",
        Help: "Time spent executing database queries.",
        Buckets: []float64{
            0.001, 0.005, 0.01, 0.025, 0.05,
            0.1,   0.25,  0.5,  1,     2.5,
            5,     10,    30,
        },
    },
    []string{"query_type"},
)

The lower bound is 1 ms; the upper bound is 30 s; the boundaries in between are exponential with extra resolution at 100 ms and 1 s (typical SLO regions for DB queries).

3. Disk IO latency (the millisecond-range case).

Disk IO spans microseconds (cached read) to seconds (page fault). A fine exponential layout:

var ioDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "disk_io_duration_seconds",
        Help: "Time spent in disk IO operations.",
        Buckets: []float64{
            0.0001, 0.0005, 0.001, 0.005, 0.01,
            0.05,   0.1,    0.5,   1,     5,
            10,
        },
    },
    []string{"operation", "device"},
)

The lower bound is 100 microseconds; the upper bound is 10 s. The boundaries are exponential with a factor of roughly 5 between 0.001 and 10. The dense region is 1 ms to 1 s (typical SLO region for IO).

4. Queue wait latency (the seconds-range case).

Queue waits span milliseconds to minutes. A wide exponential layout:

var queueWaitDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "queue_wait_duration_seconds",
        Help: "Time spent waiting in the work queue.",
        Buckets: []float64{
            0.01, 0.05, 0.1, 0.5, 1,
            5,    10,   30,  60, 300,
            600,  1800, 3600,
        },
    },
    []string{"queue"},
)

The lower bound is 10 ms; the upper bound is 1 hour. The boundaries are dense around the expected wait time (seconds to minutes) and sparse at the pathological tail (above 10 minutes, which is paged on anyway).

5. SLO-driven custom layout (the precision case).

When the SLO is on a specific value and the panel must distinguish breaches, a custom layout with boundaries every 10-25% of the SLO:

var checkoutDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "checkout_duration_seconds",
        Help: "Time spent in checkout.",
        // SLO is 200 ms; boundaries every 25 ms in the SLO region.
        Buckets: []float64{
            0.01, 0.025, 0.05, 0.075, 0.1,
            0.125, 0.15, 0.175, 0.2, 0.225,
            0.25, 0.3, 0.4, 0.5, 1,
            2.5, 5,
        },
    },
    []string{"status"},
)

This layout resolves a 175 ms regression from a 200 ms regression from a 225 ms regression. The SLO alert fires at the right granularity.

How to validate it

Three validations, each catching a different mistake.

1. Confirm the layout is what the source was configured with.

# One row per le boundary (READ-ONLY)
count by (le) (http_request_duration_seconds_bucket)

The output should list exactly the boundaries the source declared. If a row is missing, the producer was restarted with a different layout.

2. Confirm the SLO region has boundaries.

# Are there boundaries every 25 ms between 100 and 300?
# (READ-ONLY)
http_request_duration_seconds_bucket{le=~"0\\.1|0\\.125|0\\.15|0\\.175|0\\.2|0\\.225|0\\.25|0\\.275|0\\.3"}

If the result is empty, the SLO region has no boundaries; the p99 panel will read the bucket boundary, not the actual p99.

3. Confirm the tail is bounded.

# What fraction of observations fall in the largest bucket?
# (READ-ONLY)
1 - (
  rate(http_request_duration_seconds_bucket{le="5"}[5m])
  /
  rate(http_request_duration_seconds_bucket{le="+Inf"}[5m])
)

If the fraction is large (above 10%), the tail is being clipped. Add a higher boundary; the operator needs to distinguish “slow” from “very slow”.

How it can fail

Six failure modes, in the order they appear in real environments.

1. SLO region too coarse. The default layout has no boundary between 250 ms and 500 ms. A regression to 300 ms is invisible to the p99 panel. Symptom: the customer report fires; the SLO alert does not.
2. Upper bound too low. A histogram with the largest finite boundary at 10 s clips every observation above 10 s into the same bucket. A regression that lifts the p99 to 30 s looks the same as one that lifts it to 60 s. Symptom: the operator cannot distinguish a slow regression from a stuck regression.
3. Lower bound too high. A histogram with the smallest finite boundary at 10 ms records every observation below 10 ms in the same bucket. A fast service has all observations in the lowest bucket; the p99 panel reads the boundary. Symptom: the panel cannot distinguish 1 ms from 9 ms.
4. Too many boundaries. A 30-bucket histogram in a hot path is a real CPU cost. A 60-bucket histogram is a worse CPU cost and rarely buys resolution. Symptom: the producer’s CPU profile shows the histogram walk dominating; the operator scales the host unnecessarily.
5. Wrong unit in the boundaries. A histogram named http_request_duration_seconds_bucket with boundaries in milliseconds (500, 1000, 2500). Symptom: the panel is off by a factor of 1000 (lesson 01).

How to troubleshoot it

Diagnostic order, from cheapest to most expensive.

1. Confirm the SLO region has boundaries. Pull up the bucket distribution in production; identify the region where the operator needs resolution; check that the layout has boundaries in that region.
2. Confirm the tail is bounded. Plot the cumulative distribution; identify the value above which the operator does not need to distinguish; check that the layout’s largest boundary is at that value.
3. Confirm the producer is not over-budget. Check the producer’s CPU profile for the histogram walk; reduce the bucket count if it is dominant.
4. Confirm the layout matches the SLO. Open the SLO alert and the dashboard panel; check that the bucket boundaries in the SLO region match what the alert expects.
5. Confirm the layout is documented. The team’s instrumentation guide should list the bucket boundaries for every histogram, with the rationale.

Security implications

The bucket layout choice does not introduce a new attack surface. The metric exposes the same labels regardless of the boundaries.

The operational concern is the cardinality cost. A layout with too many boundaries on a high-cardinality histogram is a per-bucket fan-out that can blow the cardinality budget. The mitigation is the standard one: validate the cardinality cost in staging before the layout ships.

Performance implications

The bucket layout choice is paid on the producer and on the platform.

• Producer CPU. Each observation walks the bucket array. The cost per observation is O(N/2) for a uniform distribution; O(N) in the worst case (an observation above the largest boundary increments every bucket). A 12-bucket histogram in a hot path (10,000 req/s) is roughly 60,000 atomic increments per second; a 30-bucket histogram is 150,000.
• Platform cardinality. Each boundary is one more time series per label set. A 30-bucket histogram with 1,000 label sets is 32,000 series; the same metric with a 12-bucket layout is 14,000. The difference matters when the histogram is on a hot path.
• Query cost. histogram_quantile() walks the bucket vector linearly. The cost per evaluation is O(N). With a 30-bucket layout, the cost is roughly 30 operations per evaluation; with a 12-bucket layout, 12. The difference is small compared to the rate calculation cost.

The trade-off is between resolution and cost. The right layout is the smallest one that resolves the SLO and bounds the tail.

Production guidance

• Default to DefBuckets for web request latency. Do not customise until measurement shows the default is wrong.
• Customise the boundaries for non-web workloads: DB queries, IO, queue waits, batch jobs. Use the workload’s natural range.
• Put boundaries every 10-25% of the SLO target in the SLO region. The SLO alert needs resolution at the boundary.
• Bound the tail at the largest value the operator needs to distinguish. Beyond that, one bucket is enough.
• Document the choice in the instrumentation guide. The rationale is the bit that survives the next refactor.
• Validate the layout in staging before shipping. A regression that goes unnoticed because the layout is too coarse is a panel that lies during the next incident.
• Re-examine the layout when the SLO changes, when a new dependency shifts the latency distribution, or when the cardinality budget comes under pressure.

Verification

You should now be able to answer:

• What is the default Prometheus client library bucket layout, and when is it appropriate?
• What is the “your tail is bounded” heuristic, and why does it matter?
• What changes when you choose too few boundaries vs too many?
• How do you put boundaries at the SLO?
• Why should the bucket layout be documented in the instrumentation guide?

Quiz