Exemplars Overview

At 02:47 a checkout service paged on “p99 latency above 1s”. The on-call engineer opened the latency histogram in Grafana. The curve was flat at 80 ms. The alert had fired because a single 1.4 s outlier had crossed the rolling threshold. The engineer needed to see the trace of that slow request, not the average of 20,000 others. Grafana drew a small diamond on the histogram bar at the timestamp of the breach. The engineer clicked the diamond. Tempo opened the trace: a 1.3 s pause inside a Redis connection-pool exhaustion on the cart-svc dependency. The fix was a config change to the pool size. Total time-to-root-cause: 3 minutes.

The diamond on the bar is an exemplar. This lesson is about what an exemplar is, what it is not, and what the platform must do to put one under the engineer during the next incident.

What it is

An exemplar is a reference from a single histogram datapoint to one specific trace. It is not full trace data, not a log line, and not a request UUID. It is a small piece of metadata appended to the bucket counter at the moment the observation was recorded: enough to find the trace inside the trace backend, and that is all.

The OpenMetrics text format exposes an exemplar as a suffix on the bucket line, after a # separator and inside curly braces:

# TYPE http_request_duration_seconds histogram
# HELP http_request_duration_seconds Time spent handling HTTP requests.
http_request_duration_seconds_bucket{method="POST",route="/checkout",status="200",le="0.5"} 18432 # {trace_id="abc123def456...",span_id="7890abcd"} 0.421 1715638800.123
http_request_duration_seconds_bucket{method="POST",route="/checkout",status="200",le="1.0"} 18434 # {trace_id="aa11bb22cc33...",span_id="44556677"} 0.948 1715638801.456
http_request_duration_seconds_bucket{method="POST",route="/checkout",status="200",le="+Inf"} 18440 # {trace_id="ee99ff88dd77...",span_id="22334455"} 1.412 1715638802.044
http_request_duration_seconds_count{method="POST",route="/checkout",status="200"} 18440
http_request_duration_seconds_sum{method="POST",route="/checkout",status="200"} 1438.221

Read the suffix left-to-right:

• trace_id — the W3C traceparent of the request that landed in this bucket. Identifies the trace inside Tempo.
• span_id — the span that produced the observation. Useful when the trace is huge and the engineer needs to jump to the slow span directly.
• The numeric value — the actual observation that produced this exemplar, 0.421 s in the first line. Not a fraction of the bucket, but the exact value.
• The timestamp — when the observation was recorded, in seconds since the epoch with sub-second precision.

The exemplar is bound to the bucket, not the count. A bucket that received 500 observations holds one exemplar (representative), chosen by the client library, not 500.

Why a sysadmin cares

The histogram tells the operator that the service is slow at 02:47. The trace tells the operator why the service is slow at 02:47. Exemplars are the join between the two. Without them, the operator must either grep logs by timestamp (slow, incomplete, requires the team to have logged the trace ID at the right level) or open every trace in a five-minute window and look for the slow one by hand. The latter is what teams without exemplars actually do in the small hours of the morning.

The two operational shapes an exemplar eliminates:

1. The guesswork pivot. A histogram bucket shows 300 observations above 1 s. The engineer opens Tempo, types status = error, scrolls until they find a slow trace. The exemplar skips the search. The trace is already pointed at.
2. The unknown failure mode. A retry storm at 03:14 produces 4,000 traces in 200 ms. None of them have a pre-defined label. The error counter shows the volume; the traces show the cause. The exemplar is the only way to find one representative request inside the noise.

The exemplar is not a substitute for full tracing. It is a fast path to the right trace.

How it works

The mental model is a histogram with at most one diamond on each bar at each scrape:

le="+Inf"  bar with optional diamond
le="5"     bar with optional diamond
le="2.5"   bar with optional diamond
le="1"     bar with optional diamond  <-- representative trace
le="0.5"   bar with optional diamond
le="0.25"  bar with optional diamond
le="0.1"   bar with optional diamond
le="0.05"  bar with optional diamond
le="0.025" bar with optional diamond
le="0.01"  bar with optional diamond
le="0.005" bar with optional diamond
              ^
              diamond = trace_id of a real observation

The exemplar is chosen by the producer’s instrumentation library at observation time. The library inspects the active span context; if a span is currently active, the library records the trace_id and span_id on the bucket cell that the observation just incremented. If no span is active, the bucket is updated without an exemplar.

Three consequences of this design:

1. At most one exemplar per bucket per scrape. The producer does not record every observation that lands in the bucket. It records the last one for which an active span existed. The trace_id field is overwritten on each subsequent matching observation.
2. Exemplars require a trace context. If the request is not instrumented for tracing, the histogram still records the observation, but no exemplar is attached. The metric is fine; the link is missing.
3. Exemplars are not “the trace.” They are a pointer. The trace itself lives in the trace backend (Tempo, Jaeger, Honeycomb). The exemplar is the address. The trace backend is the building.

Where the exemplar lives

The exemplar lives in three places at three times:

Observation time (microseconds)
  -> stored in the producer library's in-memory bucket cell
Scrape time (seconds)
  -> serialised into the OpenMetrics text on the scrape endpoint
  -> sent over HTTP to Prometheus
Prometheus ingestion (one-shot)
  -> written to the dedicated exemplar appender file
  -> queried by Grafana at panel render time

The exemplar is not stored in the regular TSDB head block. The Prometheus server, when started with --enable-feature=exemplar-storage, opens a separate appender file (<data_dir>/exemplar/) and writes one entry per scrape per time series. The appender is a write-once, append-only file; the storage is bounded by configuration, not by the number of incoming exemplars.

How Grafana uses it

When a Grafana panel renders a histogram, the data source query returns both the bucket values and the exemplars attached to them. Grafana 11.x plots the diamond at the (bucket, timestamp) coordinate and registers a click handler that opens the trace in the configured trace backend. The operator clicks; Tempo loads; the trace is on screen.

The chain requires every link. A missing link breaks the chain silently. The diamond does not appear, and the operator has no reason to know the exemplar exists.

How to configure it

Two configurations: the Prometheus server, and the Grafana data source.

1. Prometheus server flag.

The feature flag is a command-line argument on the prometheus binary:

# CONFIGURATION — add to the systemd unit or pod spec
# Verified on prometheus 2.55.x
/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --enable-feature=exemplar-storage

Verify the flag was parsed:

# READ-ONLY
curl -sf http://prometheus:9090/api/v1/status/runtimeinfo \
  | jq '.data.featureFlags'

Expected: the response includes a feature called exemplar-storage with enabled: true. If the field is absent, the server was started without the flag and the append-only exemplar file is not being written.

2. Grafana data source.

The Prometheus data source in Grafana 11.x has an “Exemplars” section. The minimum configuration is the trace backend data source UID and the internal-link label name:

# Grafana provisioning file (CONFIGURATION)
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prom
    url: http://prometheus:9090
    jsonData:
      httpMethod: POST
      # The data source to open when the user clicks a diamond
      internalLink:
        tracing:
          dataSourceUid: tempo
          # Optional: override the label name Tempo expects
          label: traceID
          # Optional: when the trace ID is in a different label
          spanId: spanID

The dataSourceUid must match the UID of the Tempo data source configured elsewhere in the same Grafana instance. Grafana builds the link as /explore?ds=tempo&traceID=<trace_id> when the user clicks the diamond.

3. Producer SDK (the histogram call).

The default configuration is sufficient for most client libraries. In OpenTelemetry, the SDK attaches an exemplar reservoir to each histogram metric by default; the reservoir records the active trace context at observation time. For Go’s prometheus/client_golang, exemplars are recorded on every observation that has an active span. No additional configuration is required.

For the Python prometheus_client library, exemplars are recorded by default if a trace context is supplied via the OpenTelemetry bridge. The lesson 03 covers the per-library configuration in detail.

How to validate it

Three layers of validation, each catching a different failure mode.

1. The producer emits exemplars on /metrics.

# READ-ONLY
curl -sf http://checkout.svc:8080/metrics \
  | grep '^http_request_duration_seconds_bucket' \
  | grep '# {trace_id'

Expected: at least one line per histogram family per scrape with a # {trace_id="..."} suffix. If every line is missing the suffix, either no request with an active span has hit the endpoint, or the producer is not configured to record exemplars.

2. Prometheus is storing exemplars.

# READ-ONLY
curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="1.0"}' \
  --data-urlencode 'start=2026-08-13T02:40:00Z' \
  --data-urlencode 'end=2026-08-13T02:50:00Z' \
  | jq '.data'

Expected: an array of one or more exemplar objects, each with seriesLabels, exemplarLabels (containing trace_id and span_id), value (the numeric observation) and timestamp. An empty array means the flag is off or the producer is not emitting.

3. Grafana shows the diamond.

Open the histogram panel in Grafana. Hover the bar at the timestamp of a recent scrape. A diamond appears on the bar if the producer emitted an exemplar and Grafana is configured to render them. Click the diamond. The trace opens in Tempo.

The diamond is the only end-to-end check. The API responses can be present without the diamond appearing in the panel; the panel options can be enabled without the API returning data.

How it can fail

Six failure modes, ordered by frequency in real environments.

1. Feature flag not enabled on Prometheus. The server was started without --enable-feature=exemplar-storage. The producer emits exemplars on /metrics; Prometheus parses them; the appender is not configured to write. The query_exemplars API returns an empty array. Symptom: the diamond never appears; the API never returns data.
2. Producer lacks an active span context. The histogram is recorded but no span is active at the moment of the observation. The bucket is incremented; the trace ID is empty; the # {trace_id=...} suffix is absent. Symptom: the metric is fine; the exemplar is missing on every line.
3. Producer sampler is too aggressive. The OpenTelemetry SDK rejects the trace at the head sampler, but the histogram is updated before the decision is reported back. The request is observed by the metric but not by the trace. Symptom: the histogram has activity; the trace backend has nothing for the exemplar to point at.
4. Internal link not configured in Grafana. The data source has the exemplars section, but the dataSourceUid field is empty or points to a non-existent data source. Symptom: the diamond appears; clicking it opens a blank Explore page or a 404 from the trace backend.
5. Trace ID label mismatch. Grafana 11.x expects the trace ID in a label named traceID by default. The producer uses trace_id. Symptom: the diamond renders; the link is built with the wrong field; Tempo cannot find the trace.
6. Exemplar retention expired. The appender file holds exemplars for a bounded time (default 15 minutes in Prometheus 2.55.x). The operator opens a panel at 03:20 for an incident at 03:14. The exemplars have aged out. Symptom: the diamond is absent only for older timestamps; recent activity is fine.

How to troubleshoot it

Steps in order from cheapest to most expensive.

1. Confirm the producer emits exemplars. curl /metrics and grep for # {trace_id. If the suffix is absent, the problem is upstream. The flag, the producer, and the SDK are the next places to look.
2. Confirm the Prometheus flag is enabled. curl /api/v1/status/runtimeinfo and inspect featureFlags. If exemplar-storage is not enabled, the appender is not being written. Restart the server with the flag.
3. Confirm the API returns data. query_exemplars with the same query the panel uses. If the API returns an empty array but the /metrics is fine, the appender file may be unreadable; check the Prometheus data directory permissions.
4. Confirm the data source link is configured. In Grafana, open the Prometheus data source settings; the “Internal link” section must list a trace data source with a valid UID. Cross-check the UID against the Tempo data source configuration.
5. Confirm the trace backend holds the trace. Take a trace ID from one of the exemplars and curl the trace backend directly. If the trace is not there, the head sampler is rejecting the request or the trace is in a different tenant.
6. Confirm the panel options are enabled. Grafana panel options have an “Exemplars” toggle in the histogram visualisation. If it is off, the diamond is suppressed even when exemplars are present.

Security implications

Exemplars carry trace IDs. Trace IDs are not secrets — they are designed to be passed across trust boundaries — but they are correlation handles. An attacker who reads the exemplar stream can enumerate the trace IDs in a time window and attempt to load the corresponding traces from the trace backend. If the trace backend is not authenticated, the attacker reads traces from production.

The labels attached to the exemplar are the same labels that appear on the histogram. A label that resolves to a unique user, customer, or session becomes a unique exemplar per user. The exemplar is not a log line and not a metric, but the same privacy discipline applies. Do not put PII in a histogram label; the exemplar will carry it forward.

The /metrics endpoint is the source. If the endpoint is exposed to a network the team does not control, the trace IDs are exposed too. The platform security part of the course covers the authentication of scrape endpoints.

Performance implications

The exemplar is small: a trace_id (16 bytes), a span_id (8 bytes), the value, the timestamp, and the label set. A single exemplar is roughly 100-300 bytes on the wire depending on the labels. The appender file stores it once; the in-memory cache keeps the most recent exemplar per bucket per series.

The cost is paid in three places:

1. Producer. The library must look up the active span context on every observation. The cost is one map lookup per observation. It is negligible relative to the histogram increment itself.
2. Prometheus appender. The appender is bounded by the exemplar-storage configuration. The default retention is 15 minutes. With at most one exemplar per scrape per series, the steady-state size of the appender is roughly (series_with_exemplars * 200 bytes). A platform with 10,000 histogram-bearing series holds a 2 MiB appender — small in absolute terms.
3. Grafana query load. Each histogram render pulls the exemplars for the visible time range. The cost is per-panel, not per-second. A 50-panel dashboard with histogram-type panels produces 50 exemplar queries per refresh. The query is cheap; the rate can multiply.

The trade-off is favourable: a small write cost buys a single-click pivot from a metric to a trace. The cost is worth paying on every histogram that the team uses for incident investigation.

Production guidance

• Enable the flag at deployment time. The flag is off-by-default. Bake it into the systemd unit or the Helm chart for the Prometheus server. Validate the flag is active after every restart.
• Limit exemplar labels to the platform label set. The producer code should not lift local variables into the exemplar label. Use the same denylist the histogram uses.
• Configure the internal link on every Prometheus data source. A data source without the link is invisible to the click-through. The provisioning file is the right place for this.
• Monitor the appender size. The metric prometheus_exemplar_storage_size_bytes reports the appender size. A growing series-by-series count is normal; a runaway growth is a misconfigured producer.
• Plan for the 15-minute retention. The default is short. If the team investigates incidents that are 30 minutes old, the exemplar window is too short. The retention is set on the Prometheus command line; verify it matches the incident response cadence.

Verification

You should now be able to answer:

• What is an exemplar in the Prometheus 2.55.x data model?
• Which four components must be configured for an exemplar to surface in a Grafana 11.x panel?
• What is the difference between trace_id and span_id in the OpenMetrics exemplar suffix?
• How do you confirm that Prometheus is storing exemplars when the panel shows no diamond?
• Why is the exemplar a “pointer” rather than a trace?

Quiz