Exemplar Emission

At 14:02 a payment service started losing 0.5% of requests to a transient Redis timeout. The dashboard showed the error counter climbing. The trace backend showed a flood of error spans. The on-call engineer wanted to follow one specific request from the metric into the trace, but the histogram had no exemplars. The team had configured the OpenTelemetry SDK with a head sampler dropping 99.9% of traces. The histogram was emitting every observation; the exemplar required the request to be sampled by the trace pipeline, and the trace pipeline was discarding 1,000 of every 1,001 requests. The exemplar reservoir found no eligible trace context. The diamond was absent on every bar.

The fix was to add an exemplar-specific reservoir that records the trace context even when the trace is not otherwise sampled. The team enabled the OTel AlwaysOnExemplarReservoir only for the metrics pipeline and kept the head sampler at 0.1% for the trace pipeline.

This lesson is about where the exemplar comes from at observation time, how the reservoir picks the trace, and where the configuration knobs are in each client library.

What it is

An exemplar is emitted by the producer library at the moment the histogram cell is incremented. The library inspects the active span context, captures the trace_id and span_id, and records them on the bucket cell. The emission is a side effect of the Observe call; the application code does not invoke the exemplar system directly.

The emission is governed by three policies:

1. The trace context policy. If the call is inside an active span, the trace ID and span ID are available. If not, the exemplar is dropped. The metric is recorded either way.
2. The sampling policy. Some libraries require the sampled flag on the span to be true before the exemplar is recorded. Others record the exemplar regardless of the sampling decision. The OpenTelemetry spec gives the library the choice.
3. The reservoir policy. The reservoir decides which observation in a window gets the honour of being the exemplar. The default reservoir in OTel is the AlignedBucketBucketReservoir (a typo in the spec; the name is what it is). It keeps a reservoir per bucket and records observations at a rate proportional to the bucket traffic.

The three policies combine to produce a representative but not exhaustive exemplar stream. The team that understands the policies can tune the rate.

Why a sysadmin cares

The exemplar is the connection between the metric and the trace. The reservoir is the part of the platform that decides whether the connection is alive. A team that configures the metric pipeline without configuring the reservoir gets a histogram with no exemplars on every bar. A team that configures the reservoir with the default sampler gets a histogram with one exemplar per thousand bars. Both teams see the same diamond count on the panel — zero — and the on-call engineer is back to grep.

The reservoir is also where the cost is paid. Every exemplar is a trace ID and a span ID that must be stored in the producer, transmitted on the wire, and stored in the Prometheus appender. The reservoir decays the count back to the bucket rate; the operator owns the cap.

How it works

The mental model is a histogram bucket with a side pocket that holds the trace ID of the most recent observation:

Observation arrives
  -> library increments bucket counter
  -> library inspects active span context
       (no context) -> nothing to record
       (context present) -> record trace_id and span_id
  -> reservoir overwrites previous trace_id and span_id
  -> bucket counter is committed
  -> next scrape reads bucket counter and side pocket

The library never records more than one trace context per bucket per scrape. The reservoir is the LRU cell.

The OpenTelemetry SDK path

The OpenTelemetry SDK attaches an ExemplarReservoir to every histogram instrument. The reservoir is supplied by the SDK and is configurable per-meter. The default is the AlignedBucketBucketReservoir, which keeps one exemplar per bucket per measurement cycle. The cycle is the aggregation interval of the metric pipeline.

When the SDK receives an observation, the reservoir inspects the active span context, decides whether the observation qualifies (based on the configured filter), and records the trace ID and span ID if the observation is selected. The selection is governed by the reservoir itself, not by the application’s instrumented code.

The Prometheus client library path

The Go prometheus/client_golang library attaches an exemplar cell to every histogram bucket. The cell is written from the active span context using the Go tracing package; the cell is read by Collect and emitted on the /metrics endpoint.

The Python prometheus_client library uses the OpenTelemetry bridge to read the active span context. The bridge is configured separately; without it, the exemplar is not recorded.

The Java simpleclient library records exemplars through the OpenTelemetry bridge. The native path is also available; the bridge is the recommended path.

The OpenTelemetry Collector path

The Collector has a histogramstoexemplars processor and an exemplars processor. The histogramstoexemplars processor scans incoming OTLP histograms for paired exemplar attachments and forwards them as Prometheus exemplar trailers. The exemplars processor is the reverse: it extracts exemplars from the trailers and attaches them to the OTLP histograms.

The Collector path is the easiest to configure for fleets that already have Alloy or the Collector deployed. The library path is the easiest for fleets that have only the client library.

The timing mismatch

The exemplar is recorded at the moment the observation arrives. The trace sampling decision is made at the moment the span is created. The two moments are microseconds apart, but the trace sampling decision may not yet be available when the observation is recorded. In practice, the OpenTelemetry SDK waits for the sampling decision before emitting the exemplar. The Prometheus client library does not: it records the exemplar unconditionally if a span context is active.

The mismatch produces a discrepancy. The histogram may record an observation whose trace was subsequently dropped by the sampler. The exemplar points at a trace that does not exist in the trace backend. The diamond renders; the click opens a 404.

How to configure it

The configuration is per library. The OpenTelemetry SDK is the most explicit; the Prometheus client libraries have defaults.

1. OpenTelemetry SDK (recommended path).

# Python: opentelemetry-sdk (verified on 1.27.x)
from opentelemetry import trace, metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.exemplar import (
    AlwaysOnExemplarFilter,
    TraceBasedExemplarFilter,
)

# Option A: default — record exemplars only when the
# trace is sampled. Cheap; can produce zero exemplars
# under head sampling.
provider = MeterProvider()

# Option B: record exemplars regardless of the sampling
# decision. Pay the storage cost; gain the diamond.
provider = MeterProvider(
    exemplar_filter=AlwaysOnExemplarFilter(),
)

// Go: opentelemetry-go (verified on 1.28.x)
import "go.opentelemetry.io/otel/sdk/metric"

provider := metric.NewMeterProvider(
    metric.WithExemplarFilter(metric.AlwaysOnExemplarFilter),
)

2. Prometheus client library (Go).

// Go: prometheus/client_golang (verified on 1.20.x)
import "github.com/prometheus/client_golang/prometheus"

var requestDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "Time spent handling HTTP requests.",
        Buckets: prometheus.DefBuckets,
    },
    []string{"method", "route", "status"},
)

// Exemplars are recorded by default when an active
// span context is present. The library inspects
// the trace.SpanFromContext(ctx) under the hood.

3. Prometheus client library (Python).

# Python: prometheus_client (verified on 0.20.x)
from prometheus_client import Histogram

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'Time spent handling HTTP requests',
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10),
    labelnames=('method', 'route', 'status'),
    # enable_exemplars defaults to True since 0.18.
    # The library requires the OpenTelemetry bridge to
    # read the active span context. Without the bridge,
    # the exemplar is empty.
)

4. Grafana Alloy / OpenTelemetry Collector.

# alloy/config.alloy (PROCESSOR block — drop into the
# pipelines section)
processors:
  # Extract exemplars from OTLP histograms and forward
  # them as Prometheus exemplar trailers.
  exemplars: {}

  # The reverse direction: extract exemplars from
  # Prometheus trailers and attach them to OTLP histograms.
  histogramstoexemplars: {}

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [exemplars, batch]
      exporters: [prometheus]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

The Collector path is the cleanest for a fleet. The library path is the right choice for a single service where the team owns the code.

How to validate it

Three layers of validation, each catching a different failure mode.

1. The producer emits exemplars at the expected rate.

# READ-ONLY
# Count exemplars on the last 1000 bucket lines
curl -sf http://checkout.svc:8080/metrics \
  | grep '^http_request_duration_seconds_bucket' \
  | tail -1000 \
  | grep -c '# {trace_id'

Expected: roughly equal to the active bucket count divided by the open-span rate. If the count is zero, no request with an active span is reaching the endpoint.

2. The emitted trace IDs exist in the trace backend.

# READ-ONLY
# Extract trace IDs and check Tempo
TRAJECT_ID=$(curl -sf http://checkout.svc:8080/metrics | \
  grep '^http_request_duration_seconds_bucket' | \
  grep -o 'trace_id="[a-f0-9]*"' | head -1 | cut -d'"' -f2)

curl -sf "http://tempo:3200/api/traces/$TRAJECT_ID" \
  | jq '.batches | length'

Expected: a non-zero count. A zero count means the trace was dropped by the sampler or never recorded by the backend.

3. The Prometheus query API returns exemplars.

# READ-ONLY
curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="1.0"}' \
  --data-urlencode 'start=2026-08-13T14:00:00Z' \
  --data-urlencode 'end=2026-08-13T14:30:00Z' \
  | jq '.data | length'

Expected: a non-zero count. A zero count means the Prometheus server is not storing exemplars, even though they are being emitted.

4. The exemplar rate matches the trace rate.

The exemplar rate should be roughly equal to the trace sample rate times the request rate. If the rates diverge by an order of magnitude, one of the three policies is misconfigured.

How it can fail

Six failure modes, ordered by frequency.

1. Head sampler at 0.1% with trace-based exemplar filter. The default OpenTelemetry exemplar filter records exemplars only when the active span is sampled. A 0.1% head sampler produces an exemplar on 1 in 1,000 requests. The diamond is sparse on the panel. Symptom: the diamond count is far below the bar count.
2. No active span on the observation path. The producer instrumented the metric but the request reaches the metric call outside a span. The histogram is incremented; the exemplar is empty. Symptom: the bucket line is present without a trailer.
3. OpenTelemetry bridge not configured in Python. The prometheus_client library cannot read the active span context without the OTel bridge. The exemplar is recorded with an empty trace ID. Symptom: the trailer is present but the trace ID is empty or 00000000.
4. Wrong exporter label set. The exporter is configured to emit a histogram with a different label set than the trace context. The exemplar picks up the bucket labels but the trace ID is from the wrong context. Symptom: the diamond appears, but the trace ID points at a different service.
5. Reservoir cycle too short. The OTel reservoir aggregates over a 10-second cycle. If the histogram is exported every 5 seconds, the reservoir is reset twice per cycle. The exemplar is recorded twice. Symptom: the appender file size grows at 2x the expected rate.
6. Span context enrichment lag. The instrumented code calls histogram.Record(value) before the trace context is enriched with the request attributes. The exemplar is recorded with a span that has no attributes. Symptom: the diamond appears; the trace opens; the span has no labels.

How to troubleshoot it

Steps in order from cheapest to most expensive.

1. Confirm the trace sampler is on. TraceBasedExemplarFilter requires the trace to be sampled. If the trace sampler is ParentBased(TraceIDRatio(0.001)), the exemplar rate is limited to 0.1%. Switch to AlwaysOnExemplarFilter for the metric pipeline.
2. Confirm the active span is present. Add a debug log on the producer side that prints the trace ID before the histogram call. If the trace ID is empty, the span is not active.
3. Confirm the OpenTelemetry bridge is configured for Python. The bridge is a separate package. Without it, the exemplar is empty.
4. Confirm the reservoir cycle matches the export interval. The cycle should be an integer multiple of the export interval. A mismatched cycle produces duplicate exemplars.
5. Confirm the trace context propagates. Open a trace in Tempo for one of the exemplars. The trace should span the metric call. If the trace is from a different request, the context is being passed by value somewhere in the call chain.

Security implications

The exemplar carries the trace ID. The trace ID is a handle, not a secret. But the trace ID, once captured, can be used to retrieve the trace from the trace backend. The trace backend must be authenticated; the /metrics endpoint must be on a network the team controls.

The reservoir also records the label set on the bucket. A label that resolves to a unique user, customer, or session becomes a unique exemplar per user. The exemplar is not a log line; the same privacy discipline applies.

The reservoir’s AlwaysOnExemplarFilter does not change the privacy posture. The exemplar is recorded regardless of the sampling decision; the trace ID is the same. The operator should treat the exemplar stream as a PII stream and protect it accordingly.

Performance implications

The exemplar emission is cheap on the producer side. The reservoir is a map lookup per observation; the cost is one atomics read per bucket. The total cost is proportional to the bucket count, not the observation count.

The cost is paid on the wire and in the appender. The appendage cost is the same as lesson 02: 100-160 bytes per exemplar, bounded by the histogram cardinality and the retention window.

The cost is significant on the trace backend. The AlwaysOnExemplarFilter produces an exemplar for every observation. The trace backend receives one trace ID per exemplar click. The backend is not required to materialise the trace; the click is the load. A team that expects 1,000 clicks per hour on a metric should size the trace backend for 1,000 trace retrievals per hour, not 1,000 trace ingests per hour.

Production guidance

• Default to TraceBasedExemplarFilter for high-volume services. The filter is cheap; the storage is bounded by the trace sampling rate. The diamond count is proportional to the trace sample rate.
• Switch to AlwaysOnExemplarFilter for the SLO metrics. The SLO metrics are the ones the team clicks during incidents. The diamond is worth the storage cost.
• Tune the reservoir to the export interval. A 10-second reservoir cycle on a 5-second export interval produces duplicate exemplars. Match the cycle to the export interval.
• Validate the bridge in CI. The OTel Python bridge is a separate dependency. The team should add a CI check that the bridge is present and the exemplar is non-empty.
• Monitor the exemplar rate. The metric prometheus_target_scrape_pool_exemplar_appended_total reports the exemplar rate per scrape. A rate that diverges from the active-span rate is a misconfiguration.

Verification

You should now be able to answer:

• Where does the producer library get the trace ID for the exemplar?
• What is the difference between TraceBasedExemplarFilter and AlwaysOnExemplarFilter?
• Why is the exemplar absent on a histogram whose observations are sampled at 0.1%?
• How does the OpenTelemetry Collector forward exemplars from OTLP to Prometheus?
• What is the timing mismatch between observation and trace sampling decision?

Quiz