Native Histograms

At 03:55 a service deploys a new Go binary that emits native histograms. The new producer is instrumented with NativeHistogramBucketFactor: 1.1 (an exponential layout that resolves every bucket to within 10%). The classic-histogram version of the same metric was emitting 12 buckets per label set. The native version emits 1 series per label set, with the distribution encoded in a sparse protobuf payload. The head block series count for this metric drops from 3,456 to 288. The team gets back cardinality budget they had spent on the bucket fan-out.

This is the operational promise of native histograms: the same distribution shape, much smaller series footprint, exact (within bucket resolution) quantile estimation. The operational cost is a feature flag, a remote-write protocol version, and a migration plan.

What it is

A native histogram is an alternative representation of a histogram metric. Instead of one time series per _bucket{le="..."} value, the histogram is encoded as a single time series whose value carries the full bucket distribution in a sparse protobuf payload.

A native histogram in the Prometheus exposition format appears as:

# HELP http_request_duration_seconds Time spent handling HTTP requests.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_sum{route="/checkout"}   53423.214
http_request_duration_seconds_count{route="/checkout"} 144320
# the native histogram is encoded in a special line; the
# client_golang library emits it as a separate `native_`
# prefixed metric when running with --enable-feature
http_request_duration_seconds{route="/checkout",le="+Inf"} 144320
# native-histogram line — sparse, encoded
http_request_duration_seconds_native{route="/checkout"} {schema:0,zero_threshold:2.93874e-39,zero_count:0,
  count:144320,sum:53423.214,positive_spans:[{offset:0,length:331}],positive_deltas:[1,1,1,1,...]}

The *_native line carries the full distribution in a single sample. The bucket boundaries are exponential with a configurable factor (default 2^(2^-4) ≈ 1.0443 for the schema 0 layout); each bucket is a delta from the previous count, and empty buckets are omitted (sparse). A 1,000-bucket distribution can be encoded in a few hundred bytes.

Native histograms were introduced as a feature-flagged capability in Prometheus 2.40 (2022). They remain experimental in Prometheus 2.55.x — the feature is GA-quality but still behind --enable-feature=native-histograms. The flag is on the producer (to enable emission) and on the receiver (to enable ingestion).

Why a sysadmin cares

The cardinality cost of a classic histogram is the bucket fan-out. A 12-bucket histogram with 1,000 unique label sets is 14,000 series. A native histogram is one series per label set, regardless of the bucket resolution. The cardinality saving is proportional to the bucket count.

The operational consequences:

1. Cardinality budget restored. A team that has been spending budget on bucket fan-out can re-spend it on slicing labels (e.g. adding status to a histogram that only had route). The native histogram is one series per (route, status), not twelve.
2. Exact quantiles within the bucket resolution. histogram_quantile() over a native histogram interpolates against the actual bucket boundaries, not against the user’s chosen layout. A native histogram with bucket_factor = 1.1 (10% relative width) resolves quantiles to within 10% of the true value, which is finer than most classic layouts.
3. Better aggregation. A native histogram can be merged across instances by simple addition of the bucket counts; the resulting native histogram has the combined distribution. Classic histograms with different bucket layouts cannot be merged.

The cost of native histograms is paid at three points:

1. Producer compatibility. The producer must be on a client library version that supports native histograms. Most modern libraries do (Go, Python, Java, Rust); some do not.
2. Receiver compatibility. The receiver must be on Prometheus 2.50+ with --enable-feature=native-histograms (or a backend that supports remote-write 2.0 native histograms: Mimir, Thanos, Prometheus server itself for subquery).
3. Migration discipline. A mixed environment — some producers emitting classic, some emitting native — has two representations of the same metric. The dashboards must be written to handle both.

How it works

The native histogram uses a sparse representation of an exponentially-spaced bucket layout. The encoding has five parts:

• schema — the exponent spacing. Schema 0 uses 2^(2^-4) (≈ 1.0443) per bucket. Schema 1 uses 2^(2^-3) (≈ 1.0905), and so on, doubling the relative bucket width per schema. Most producers use schema 0.
• zero_threshold — observations below this absolute value go into a separate zero_count bucket. This is the only “weird” bucket; the rest are positive.
• zero_count — the count of observations that fell below zero_threshold.
• positive_spans — a list of (offset, length) pairs describing the ranges of populated buckets in the positive domain.
• positive_deltas — a list of deltas; bucket[i] count = previous_count + delta[i]. Sparse; only populated buckets appear.

schema=0            bucket boundaries: 1, 1.0443, 1.0905, 1.1386, ...
zero_threshold=0    no observations below this value
zero_count=0
positive_spans=[(offset=0, length=331)]
positive_deltas=[1, 1, 1, 1, ...]

result: a sparse representation of ~331 populated buckets
        in a few hundred bytes of protobuf.

The encoding is documented in the Prometheus native histogram specification. The decode is straightforward: walk the spans, walk the deltas, reconstruct the bucket counts. The native histogram can be converted to a classic histogram with any desired le layout; the histogram_quantile() function on the Prometheus server does this on the fly.

histogram_quantile(0.99, native_histogram_metric) is a direct query — the function reconstructs the bucket layout from the sparse encoding, locates the p99, and interpolates within that bucket. The result is more accurate than a classic histogram with the same number of buckets, because the native histogram’s bucket boundaries are finer.

Remote-write 2.0

Native histograms are sent over the wire as part of the Prometheus remote-write 2.0 protocol. The protocol extended the original remote-write 1.0 to carry histogram samples natively; receivers that only support 1.0 see the classic representation (if available) or drop the native observations entirely.

A receiver on Prometheus 2.55 with the flag enabled ingests native histograms from a producer that emits them. A receiver on an older Prometheus or a remote-write backend without 2.0 support silently drops them. This is the silent failure mode of native histograms in mixed-version clusters.

How to configure it

Three layers: producer flag, receiver flag, dashboard.

1. Producer (the instrumentation library).

In the Go client library:

import "github.com/prometheus/client_golang/prometheus"

var requestDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "http_request_duration_seconds",
        Help: "Time spent handling HTTP requests.",
        // Native histogram configuration:
        // - NativeHistogramBucketFactor: 1.1 means each bucket
        //   is 10% wider than the previous (relative to lower bound).
        // - 1.1 is the typical fine setting; 1.2 is acceptable; 1.05
        //   is finer but increases the encoded payload size.
        NativeHistogramBucketFactor: 1.1,
    },
    []string{"route", "status"},
)

With NativeHistogramBucketFactor set, the producer emits both the classic and the native representations unless explicitly told to drop the classic one. The classic emission is for compatibility with older receivers; the native emission is for receivers that support it.

2. Receiver (the Prometheus server).

The Prometheus server must be started with --enable-feature=native-histograms to ingest the native representation. Without the flag, the server silently drops native observations and ingests only the classic representation (if present).

# /etc/default/prometheus — set the feature flag
PROMETHEUS_ARGS="--enable-feature=native-histograms \
                  --storage.tsdb.path=/var/lib/prometheus/data \
                  --web.listen-address=:9090"

For remote-write targets:

# prometheus.yml — receiver side
remote_write:
  - url: https://mimir.example.com/api/v1/push
    # Mimir / Thanos / Prometheus subquery receivers
    # require remote_write 2.0 to carry native histograms.
    # Set the protocol explicitly:
    protobuf_message: io.prometheus.write.v2.Request
    send_exemplars: true

The protobuf_message setting is what activates remote-write 2.0. Without it, the receiver falls back to 1.0 and native histograms are dropped.

3. Grafana dashboards.

A panel that queries a native histogram looks identical to one that queries a classic histogram:

histogram_quantile(
  0.99,
  sum by (route, le) (
    rate(http_request_duration_seconds[5m])
  )
)

The query engine detects the sample type and converts the native histogram to a virtual classic histogram with the boundaries needed by the function. The panel works against mixed environments without modification.

How to validate it

Four validations, each catching a different mistake.

1. Confirm the producer is emitting native histograms.

# READ-ONLY — look for the _native suffix line
curl -sf http://checkout.svc:8080/metrics \
  | grep -E '^http_request_duration_seconds(_native|_bucket|_sum|_count)?' \
  | head

Expected: classic _bucket, _sum, _count lines plus a *_native line. If only the classic lines are present, the producer was not configured with NativeHistogramBucketFactor.

2. Confirm the receiver is ingesting native histograms.

# Look for the histogram metadata (READ-ONLY)
count({__name__="http_request_duration_seconds_native"})

Expected: one row per label set (route, status). If zero rows, the receiver is dropping native observations — the flag is off, or the remote-write protocol is 1.0.

3. Confirm histogram_quantile() works against the native histogram.

histogram_quantile(
  0.99,
  sum by (route, le) (
    rate(http_request_duration_seconds[5m])
  )
)

Expected: a real number per route. If NaN, the function is not seeing the native observations; check the receiver flag.

4. Confirm the cardinality saving.

# Active series for the classic histogram (READ-ONLY)
count({__name__="http_request_duration_seconds_bucket"})

# Active series for the native histogram (READ-ONLY)
count({__name__="http_request_duration_seconds_native"})

Expected: the native series count is the number of unique label sets (e.g. 12 if (route, status) has 12 values). The classic series count is that number times (N+1) where N is the number of classic boundaries.

How it can fail

Six failure modes that show up in production.

1. Receiver flag missing. A producer emits native histograms; the receiver is on Prometheus 2.55 but without --enable-feature=native-histograms. Symptom: the producer emits both representations; the receiver ingests only the classic. The cardinality saving is zero; the producer pays the encoding cost for nothing.
2. Remote-write 1.0 backend. A receiver remote-writes to Mimir / Thanos / Prometheus 2.40. Symptom: the remote-write protocol is 1.0; the backend drops the native observations; the cardinality saving is lost at the storage layer.
3. Producer flag set but library version too old. A Go binary is rebuilt with NativeHistogramBucketFactor: 1.1 but the client_golang version does not support native histograms. Symptom: the producer compiles but ignores the option; the /metrics output has only the classic representation.
4. Mixed classic and native across producers. Two services emit http_request_duration_seconds — one classic, one native. Symptom: histogram_quantile() works against both; the panel is consistent. But the per-route cardinality saving is partial; the budget calculation is misleading.
5. Dashboard written against le only. A panel query sums by (le) only — this works against native histograms (the function converts them to virtual classic histograms) but a panel that explicitly sums across le to compute the rate window may return a different value than expected. Symptom: the panel renders a different number than the per-route p99 in the staging environment.
6. Migration without rollback plan. A team flips the producer flag in production. Symptom: an older downstream consumer (e.g. a custom aggregation service) fails to parse the native representation and either crashes or silently drops the samples. The fix is coordinated; the producer flag is not safe to flip without auditing every downstream consumer.

How to troubleshoot it

Diagnostic order, from cheapest to most expensive.

1. Confirm the producer flag is set. grep NativeHistogramBucketFactor in the producer source.
2. Confirm the producer emits the native line. curl /metrics | grep _native.
3. Confirm the receiver flag is set. curl /api/v1/status/runtimeinfo (Prometheus 2.55+) lists the enabled features. The native-histograms feature must be present.
4. Confirm the remote-write protocol is 2.0. prometheus_remote_write_* metrics expose the protocol version; if the metric is missing, the receiver is on 1.0.
5. Confirm the downstream consumer handles native histograms. A consumer that expects the classic representation will see the native one as “unparseable”.

Security implications

Native histograms expose the same labels as classic histograms; the security implications are inherited. There is no new attack surface introduced by the sparse representation.

The operational concern is who can read the histogram. A panel that shows per-tenant p99 exposes the latency distribution of every tenant. Treat the query surface as part of the access model.

Performance implications

The native histogram is significantly cheaper than a classic histogram in three ways:

1. Producer CPU. The cost per observation is O(1) amortized (the bucket index is computed by integer division). A classic histogram with N buckets costs O(N) per observation (the bucket walk).
2. Platform storage. One series per label set, not N+1. A 12-bucket histogram with 1,000 label sets is 14,000 classic series or 1,000 native series. The saving is proportional to N+1.
3. Query cost. histogram_quantile() against a native histogram reconstructs the bucket layout from the sparse encoding. The cost is O(spans + deltas), which is roughly O(N) where N is the number of populated buckets.

The trade-off is the encoding payload size. A native histogram with 1,000 populated buckets is a few hundred bytes of protobuf per sample. A classic histogram with 12 buckets is 12 lines of text per sample. For high-volume scrapes, the native payload is larger per series but smaller per distribution.

Production guidance

• Audit the downstream consumer before flipping the flag. A native histogram in a custom aggregation service that does not handle the new sample type is a crash waiting to happen.
• Flip one producer at a time. The migration is per-producer. Each flip needs to be observed against the receiver and the downstream consumer.
• Document the migration in the instrumentation guide. The team needs to know which producers are on classic, which are on native, and which are mixed.
• Validate the cardinality saving. count by (__name__) (\{__name__=~"http_request_duration_seconds.*"\}) before and after the flag flip should show the expected drop.
• Keep the classic representation until the migration is complete. The dual emission costs nothing in the producer; the receiver flag controls ingestion. The receiver can be flipped first, then the producers.

Verification

You should now be able to answer:

• What is a native histogram, and how does it differ from a classic _bucket histogram at the data-model level?
• What flag enables native histogram ingestion in Prometheus 2.55.x?
• What is remote-write 2.0, and why does it matter for native histograms?
• What is the cardinality saving of a native histogram vs a classic 12-bucket histogram?
• What is the migration path from classic to native histograms across a mixed-version cluster?

Quiz