Exemplars Cost

At 03:14 a Prometheus server on the payments cluster consumed 28 GB of disk on the exemplar appender. The WAL was 2 GB. The head block was 6 GB. The exemplar appender was 20 GB. The team had enabled exemplar storage a week earlier; the team had not budgeted for it. The appender file grew at 120 MB per hour during business hours and plateaued at 20 GB because the retention was the default 15 minutes — the plateau was the steady-state size for the labels in flight.

The diagnostics were straightforward. The metric prometheus_tsdb_head_exemplar_storage_size_bytes reported 20 GB. The histogram cardinality was 1.2 million active series. The per-exemplar cost was 160 bytes. The steady-state size was 1.2 million * 160 bytes * 60 exemplars per minute = 11.5 GB per minute of high-water traffic.

The fix was two parts: raise the head sampler to 0.1% (only) on the payments service, and add a label denylist to the exporter to strip the customer_id label from the bucket. The denylist cut the active series by 80%. The sampler cut the exemplar rate by 99.9%. The appender shrank to 200 MB within 15 minutes.

This lesson is about the cost of exemplars — the wire, the appender, the trace backend — and the configuration knobs that bound it.

What it is

The cost of an exemplar is paid in three places:

1. The wire. Every exemplar is one trailer on a histogram bucket line. The trailer is 100-160 bytes. The cost is per scrape, per bucket, per series.
2. The appender. Every exemplar is one entry in the Prometheus exemplar appender file. The entry is roughly the same size as the wire trailer; the appender is append-only and bounded by the retention window.
3. The trace backend. Every exemplar click is one query against the trace backend. The trace backend must materialise the trace; the cost is the trace materialisation, not the trace ingest.

The cost is bounded by configuration. The appender has a retention window. The trace backend has a capacity. The operator owns the bounds.

Why a sysadmin cares

The default exemplar configuration is safe. The exemplar appendage is bounded by the 15-minute retention window. The wire cost is small. The trace backend cost is bounded by the trace sampling rate. A team that enables exemplars with the defaults does not see the cost.

The cost grows when the team:

• Raises the trace sampling rate to 100% on a high-volume service.
• Adds high-cardinality labels to a histogram.
• Extends the exemplar retention window.
• Pins the exemplar emission to a AlwaysOnExemplarFilter on a high-traffic service.

The cost is not visible until the bill arrives. The default configuration is a hedge; the operator who breaks the hedge should know the cost.

How it works

The mental model is a budget with three line items:

Exemplar cost
  |
  +-- wire cost
  |     = buckets_per_scrape * exemplars_per_bucket * 160 bytes
  |     = bounded by the histogram cardinality and the active-scrape rate
  |
  +-- appender cost
  |     = active_series * 160 bytes * scrape_every_15s * retention_in_minutes
  |     = bounded by the retention window (default 15 minutes)
  |
  +-- click cost
        = clicks_per_hour * trace_query_cost
        = bounded by the trace backend capacity

The three costs are independent. The wire cost is paid on every scrape. The appender cost is paid on every ingest. The click cost is paid on every Grafana panel click.

The wire cost

The wire cost is the simplest. A scrape response is one HTTP body. The bucket lines are the bulk of the body. The exemplar trailers are the trailing bytes. The wire cost of exemplars is the fraction of the body that is trailers.

A scrape response with 10,000 buckets and 1,000 exemplars is 1.6 MB of bucket lines and 160 KB of exemplar trailers. The exemplar overhead is 10% of the body. The CPU cost on the Prometheus server is the parse cost; the bandwidth cost is the trailer serialisation.

The wire cost is paid even when the operator is not looking at the panel. The trailers are emitted on every scrape. The team that disables exemplars on the panel does not disable the wire cost.

The appender cost

The appender is a 15-minute retention window of exemplar entries. The size is bounded by the number of buckets exemplars per bucket bytes per exemplar scrape frequency.

The default retention in Prometheus 2.55.x is 15 minutes. The Prometheus server does not write the appender to a file by default; the appender is in-memory. The appender size is bounded by the retention window.

The on-disk size is the WAL-replayed size. The WAL holds the appender entries for replay; the WAL is bounded by the WAL segment size (default 128 MB) and the WAL retention. The exemplar WAL is a separate file from the metric WAL.

The appender cost is paid in CPU on the Prometheus server. The appender is a map lookup per scrape; the cost is proportional to the bucket count. The map is written in memory and flushed to the WAL on every scrape.

The click cost

The click cost is the cost of querying the trace backend. The trace backend stores the trace; the click is the query. The cost is the trace materialisation; the trace backend must read the trace from storage and return it to the Grafana browser.

The click cost is paid by the trace backend, not by Prometheus. The trace backend is sized for the peak click rate. The peak click rate is set by the number of active on-call engineers and the number of dashboards they have open.

The cost of the defaults

The Prometheus server default retention is 15 minutes. The appender is in-memory. The wire cost is 10% of the body. The click cost is the trace backend capacity.

A team that enables exemplars with the defaults sees a small wire cost, a small in-memory cost, and a small click cost. The cost is bounded.

A team that raises the trace sampling rate to 100% on a high-volume service sees a wire cost that scales with the request rate, an in-memory cost that scales with the request rate, and a click cost that scales with the on-call rate. The cost is unbounded.

How to configure it

The configuration is at three levels: the Prometheus server, the OpenTelemetry SDK, and the trace backend.

1. Prometheus server.

The appender is enabled by the --enable-feature=exemplar-storage flag. The retention is set on the Prometheus binary command line. The appendage is in-memory by default; the WAL is shared with the metric WAL.

The Prometheus server does not have a per-metric exemplar cap. The appender accepts one exemplar per scrape per series. The cap is at the SDK level.

2. OpenTelemetry SDK.

The SDK has the exemplarFilter and the reservoir. The filter controls which observations are eligible; the reservoir controls which observation is selected.

# Python: opentelemetry-sdk (verified on 1.27.x)
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.exemplar import (
    AlwaysOnExemplarFilter,
    TraceBasedExemplarFilter,
)

# Option A: bounded by the trace sampling rate
provider = MeterProvider(exemplar_filter=TraceBasedExemplarFilter())

# Option B: bounded by the SDK reservoir size
provider = MeterProvider(
    exemplar_filter=AlwaysOnExemplarFilter(),
    # The reservoir is created per metric; the cap is
    # one exemplar per bucket per cycle. The cycle is
    # the export interval.
)

The SDK reservoir is the right place to bound the exemplar rate. The Prometheus server has no rate limit; the trace backend has no rate limit. The SDK is the only place the rate is owned.

3. Trace backend.

The trace backend is sized for the click rate. The click rate is the number of Grafana clicks on exemplar diamonds per hour. The trace backend is not the trace ingest; the trace backend is the trace retrieval.

# Tempo trace backend (CONFIGURATION)
auth_enabled: false
storage:
  trace:
    backend: s3
    s3:
      bucket_name: tempo-traces
      endpoint: s3.amazonaws.com
    pool:
      max_workers: 100
    wal:
      path: /var/tempo/wal
search:
  max_duration: 0   # 0 = no limit on the query window

The trace backend capacity is sized for the peak click rate. The peak click rate is set by the on-call cadence. The team should plan for 10 clicks per minute on a busy incident.

How to validate it

Three layers of validation, each catching a different cost-related failure mode.

1. The Prometheus appender size is within budget.

# READ-ONLY
curl -sf http://prometheus:9090/metrics \
  | grep '^prometheus_tsdb_head_exemplar_storage_size_bytes'

Expected: a number below the budget. The budget is the team’s planning number (e.g., 1 GB). A number above the budget is a signal that the histogram cardinality has grown or the sampling rate has been raised.

2. The wire cost is within budget.

# READ-ONLY
# Count exemplars on the last scrape
curl -sf http://checkout.svc:8080/metrics \
  | grep -c '# {trace_id'

Expected: a number equal to the number of buckets * the exemplar rate. A number above the budget is a signal that the sampler is misconfigured.

3. The trace backend is sized for the click rate.

The click rate is observed in the Grafana audit log or the trace backend’s query metric. The team should monitor the rate of trace queries against the trace backend’s capacity.

# READ-ONLY
# Tempo query rate
curl -sf http://tempo:3200/metrics \
  | grep '^tempo_querying_throughput'

Expected: a number below the trace backend capacity. A number above the capacity is a signal that the trace backend is the bottleneck.

How it can fail

Six failure modes, ordered by cost-to-the-team.

1. Unbounded exemplar retention. The Prometheus server is configured with a 24-hour retention window. The appender grows at 1 GB per hour. The disk fills; Prometheus crashes. Symptom: the appender size exceeds the disk budget.
2. High-cardinality label on the histogram. A developer adds customer_id to the histogram. The bucket count grows by 5 orders of magnitude. The appender size explodes. Symptom: the Prometheus server is OOM-killed within minutes of the deploy.
3. AlwaysOnExemplarFilter on a high-volume service. The team enables the filter on a service that handles 100,000 requests per second. The exemplar rate is 100,000 per second. The wire cost is 16 MB per scrape. The appender is 1.6 GB per minute. The trace backend receives 100,000 trace IDs per second. Symptom: everything is the bottleneck.
4. Trace backend auth disabled. The team disables auth on the trace backend to fix a click failure. The trace backend is now open to the internet. Symptom: the trace backend is publicly readable.
5. WAL retention extended to match the metric retention. The team sets the WAL retention to 30 days to match the metric retention. The WAL is 30 days of exemplar entries. The disk is filled. Symptom: the WAL is 100 GB.
6. Trace backend downscaled. The team right-sizes the trace backend for the average click rate. The peak incident-time click rate is 100x the average. The trace backend is overloaded during the incident. Symptom: the trace queries time out; the click handler returns a 504.

How to troubleshoot it

Steps in order from cheapest to most expensive.

1. Check the appender size. The prometheus_tsdb_head_exemplar_storage_size_bytes metric is the first signal. The team should monitor this metric and alert on a budget breach.
2. Check the exemplar rate. The prometheus_target_scrape_pool_exemplar_appended_total metric is the second signal. The rate should be proportional to the request rate * the sampling rate.
3. Check the histogram cardinality. The count of *_bucket series is the third signal. The cardinality should be within the platform budget.
4. Check the trace backend capacity. The trace backend’s query throughput metric is the fourth signal. The capacity should be sized for the peak click rate.
5. Check the label set on the exemplar. The exemplarLabels field in the API response is the fifth signal. The label set should be the bucket label set plus trace_id and span_id. Additional labels are a sign of a misconfigured exporter.

Security implications

The exemplar is a stream of trace IDs. The trace ID is a handle; the trace is the payload. The trace backend must be authenticated; the trace backend must be on a network the team controls.

The exemplar label set is the bucket label set. A label that resolves to a customer, user, or session becomes a unique exemplar per user. The exemplar is a PII stream; the same privacy discipline applies as for the metric labels.

The wire cost is a network cost. The trailers are unencrypted on the wire unless the scrape is over HTTPS. The Prometheus server should scrape over HTTPS; the exporter should reject plain HTTP.

The appender file is on the Prometheus server’s disk. The file is plaintext; the file is readable by anyone with read access to the Prometheus data directory. The team should treat the data directory as a PII directory.

Performance implications

The wire cost is paid on every scrape. The cost is proportional to the number of buckets * the number of exemplars per bucket. The default is 1 exemplar per bucket; the cost is the bucket count * 160 bytes.

The appender cost is paid in memory and on disk. The appender is in-memory by default; the WAL is on disk. The memory cost is the active bucket count * 200 bytes (approximately). The disk cost is the WAL retention * the exemplar rate.

The click cost is paid by the trace backend. The materialisation cost is the trace storage read. The materialisation is O(trace size); the cost is bounded by the trace size, not the trace count.

The total cost is bounded by the configuration. The default configuration is cheap. The unbounded configuration is expensive.

Production guidance

• Default to TraceBasedExemplarFilter. The filter is bounded by the trace sampling rate. The diamond count is proportional to the trace sampling rate.
• Use AlwaysOnExemplarFilter only for SLO metrics. The SLO metrics are the ones the team clicks during incidents. The diamond is worth the storage cost.
• Set the label denylist on the exporter. The exporter should strip PII and high-cardinality labels from the bucket label set. The exemplar inherits the denylist.
• Monitor the appender size. The prometheus_tsdb_head_exemplar_storage_size_bytes metric is the first signal. Alert on a budget breach.
• Size the trace backend for the peak click rate. The peak click rate is the on-call incident rate. The trace backend should be sized for 100x the average click rate.
• Document the cost. The cost of exemplars is a planning number. The team should document the appender size budget, the wire cost budget, and the trace backend capacity budget.

Verification

You should now be able to answer:

• What are the three cost centres of exemplars?
• How does the wire cost scale with the histogram cardinality?
• What is the default retention window of the Prometheus appender?
• Which OpenTelemetry filter bounds the exemplar rate by the trace sampling rate?
• How do you size the trace backend for the peak click rate?

Quiz