Exemplar Troubleshooting

At 22:14 a checkout service started erroring on 8% of requests. The on-call engineer opened the latency histogram in Grafana. The bars were present. The diamond was absent. The engineer opened the request dashboard. The trace was present in Tempo. The exemplar was missing in the panel.

The team had no runbook for the missing-exemplar symptom. The engineer spent 50 minutes before reaching the right diagnostic step: the exporter was emitting exemplars, the Prometheus API was returning empty, the appender file was empty. The Prometheus server had been restarted without the --enable-feature=exemplar-storage flag. The flag was the link. The flag was missing.

The fix was one flag and a restart. The lesson is the diagnostic order that gets the engineer to the right answer in five minutes, not fifty.

What it is

The exemplar chain is end-to-end. The chain has six links: the producer instrumentation, the exporter, the Prometheus server, the trace backend, the Grafana data source, and the Grafana panel. The failure of any one link breaks the surface. The diagnostic order is a sequence of checks that walks the chain from the source to the panel.

The diagnostic order is cheap-first. The first check is the producer; the last check is the panel. Each check narrows the failure domain. The chain is broken at the first check that fails; the engineer fixes the link and re-runs the chain.

Why a sysadmin cares

The exemplar is the fastest pivot from a metric to a trace. The symptom of the missing exemplar is the on-call engineer who opens the histogram and finds no diamond. The fix is not the absence of the diamond; the fix is the link in the chain that is broken.

The team that has a runbook for the missing-exemplar symptom resolves the failure in five minutes. The team that has no runbook resolves the failure in fifty. The five-minute fix is the runbook.

How it works

The mental model is a chain with six links. Each link is a configuration; each link is a check; each link is a fix:

[Producer] -> [Exporter] -> [Prometheus] -> [Trace Backend] -> [Grafana DS] -> [Grafana Panel]
    (1)           (2)            (3)               (4)                (5)              (6)
    check         check          check             check              check            check
    span          trailer        flag              trace              data             toggle
    context       on /metrics    enabled           present            source           enabled

The diagnostic order walks the chain from left to right. The first check that fails is the broken link. The fix is to repair the link and re-run the chain.

The six checks

Check 1: Producer active span context.

The producer must have an active span context at the moment of the histogram observation. The check is to inspect the raw scrape response.

# READ-ONLY
curl -sf http://checkout.svc:8080/metrics \
  | grep '^http_request_duration_seconds_bucket' \
  | grep '# {trace_id' \
  | head -1

Expected: a line with a # {trace_id="..."} trailer. If the trailer is absent, the producer is not recording exemplars.

Check 2: Exporter trailer on /metrics.

The exporter must emit the OpenMetrics trailer on the scrape response. The check is the same as Check 1 with a different scope.

# READ-ONLY
curl -sf http://checkout.svc:8080/metrics \
  | grep -c '# {trace_id'

Expected: a non-zero count. If the count is zero, the exporter is not emitting exemplars.

Check 3: Prometheus flag enabled.

The Prometheus server must be started with the --enable-feature=exemplar-storage flag. The check is the runtime info API.

# READ-ONLY
curl -sf http://prometheus:9090/api/v1/status/runtimeinfo \
  | jq '.data.featureFlags'

Expected: an entry for exemplar-storage with enabled: true. If the entry is absent, the flag is not enabled.

Check 4: Trace backend has the trace.

The trace backend must hold the trace referenced by the exemplar. The check is to take a trace ID from the exemplar and curl the trace backend.

# READ-ONLY
TRACE_ID=$(curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="1.0"}' \
  --data-urlencode 'start=2026-08-13T22:00:00Z' \
  --data-urlencode 'end=2026-08-13T22:30:00Z' \
  | jq -r '.data[0].exemplarLabels.trace_id')

curl -sf "http://tempo:3200/api/traces/$TRACE_ID" \
  | jq '.batches | length'

Expected: a non-zero count. If the count is zero, the trace backend is not ingesting or the trace is dropped.

Check 5: Grafana data source configured.

The Prometheus data source in Grafana must have an internal link to the trace backend. The check is the Grafana API.

# READ-ONLY
curl -sf -u admin:admin http://grafana:3000/api/datasources/uid/prom \
  | jq '.jsonData.internalLink'

Expected: an object with a tracing key, a valid dataSourceUid, and a label matching the exemplar label name. If the tracing key is missing, the internal link is not configured.

Check 6: Grafana panel options enabled.

The histogram panel must have the showExemplars option enabled. The check is the dashboard JSON.

# READ-ONLY
# Substitute your own value before running. DASHBOARD_UID is the id in
# the dashboard URL: /d/fdx8mn2kq1s0wb/checkout-latency
DASHBOARD_UID=fdx8mn2kq1s0wb

curl -sf -u admin:admin \
  "http://grafana:3000/api/dashboards/uid/$DASHBOARD_UID" \
  | jq '.dashboard.panels[] | select(.type=="histogram") | .options.showExemplars'

Expected: true for every histogram panel. If the value is false, the panel is suppressing the exemplars.

The fix

Each check has a fix. The fix is to repair the link in the chain. The fix is a configuration change; the fix is a restart; the fix is a re-run of the chain.

How to configure it

The configuration is the chain. The fixes for the six checks are the configuration knobs.

Fix 1: Producer active span context.

The producer must instrument the metric call inside an active span. The fix is to wrap the metric call in a span context.

// Go: OpenTelemetry span context for the metric call
ctx, span := tracer.Start(ctx, "checkout.handle")
defer span.End()

requestDuration.WithLabelValues("POST", "/checkout", "200").
    Observe(time.Since(start).Seconds())

# Python: OpenTelemetry span context for the metric call
from opentelemetry import trace

tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("checkout.handle"):
    REQUEST_LATENCY.labels("POST", "/checkout", "200").observe(
        time.time() - start
    )

Fix 2: Exporter trailer on /metrics.

The exporter must emit the trailer. The fix is to enable the exemplar reservoir.

// Go: prometheus/client_golang (verified on 1.20.x)
// Exemplars are recorded by default when an active
// span context is present. The exporter is the same.

# Python: prometheus_client (verified on 0.20.x)
from prometheus_client import Histogram

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'Time spent handling HTTP requests',
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10),
    labelnames=('method', 'route', 'status'),
    enable_exemplars=True,  # default since 0.18
)

Fix 3: Prometheus flag enabled.

The Prometheus server must be started with the flag. The fix is to add the flag and restart.

# CONFIGURATION — add to the systemd unit or pod spec
# Verified on prometheus 2.55.x
/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --enable-feature=exemplar-storage

Fix 4: Trace backend has the trace.

The trace backend must hold the trace. The fix is to raise the sampling rate or to fix the trace backend ingest.

# Python: OpenTelemetry SDK with explicit trace pipeline
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import TraceIDRatioBased

provider = TracerProvider(
    sampler=TraceIDRatioBased(0.01),
)

Fix 5: Grafana data source configured.

The Prometheus data source must have the internal link. The fix is to provision the data source.

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prom
    url: http://prometheus:9090
    jsonData:
      httpMethod: POST
      internalLink:
        tracing:
          dataSourceUid: tempo
          label: trace_id
          spanId: span_id

Fix 6: Grafana panel options enabled.

The histogram panel must have the toggle enabled. The fix is to update the dashboard JSON.

{
  "type": "histogram",
  "options": {
    "showExemplars": true
  }
}

How to validate it

The diagnostic order is the validation. The first check that fails is the broken link. The engineer fixes the link and re-runs the chain from the beginning.

The end-to-end check is the click. The engineer opens the dashboard, hovers the bar, clicks the diamond, and opens the trace. The click is the only end-to-end check that exercises the full chain.

# READ-ONLY
# End-to-end check via the Grafana API
# Substitute your own values before running. DASHBOARD_UID is the id in
# the dashboard URL: /d/fdx8mn2kq1s0wb/checkout-latency
DASHBOARD_UID=fdx8mn2kq1s0wb
HISTOGRAM_METRIC='http_request_duration_seconds_bucket{le="1.0"}'
START=2026-08-13T22:00:00Z
END=2026-08-13T22:30:00Z

# 1. Confirm the panel is a histogram with showExemplars=true
curl -sf -u admin:admin \
  "http://grafana:3000/api/dashboards/uid/$DASHBOARD_UID" \
  | jq '.dashboard.panels[] | select(.type=="histogram") | .options.showExemplars'

# 2. Confirm the Prometheus query returns exemplars
EXEMPLARS=$(curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode "query=$HISTOGRAM_METRIC" \
  --data-urlencode "start=$START" \
  --data-urlencode "end=$END")
echo "$EXEMPLARS" | jq '.data | length'

# 3. Confirm the trace backend has the trace
# Trace ID comes from the exemplar returned by step 2
TRACE_ID=$(echo "$EXEMPLARS" | jq -r '.data[0].exemplarLabels.trace_id')
curl -sf "http://tempo:3200/api/traces/$TRACE_ID" \
  | jq '.batches | length'

All three checks must return non-zero. If any check returns zero, the chain is broken at that link.

How it can fail

Six failure modes, mapped to the diagnostic order.

1. Producer lacks an active span context. The metric is instrumented but the request reaches the metric call outside a span. The bucket is incremented; the exemplar is empty. Check: Check 1 returns a line without the trailer. Fix: wrap the metric call in a span context.
2. Exporter reservoir is disabled. The client library is configured with enable_exemplars=False or the OTel filter is set to AlwaysOffExemplarFilter. The bucket is incremented; the exemplar is empty. Check: Check 2 returns a zero count. Fix: enable the reservoir.
3. Prometheus flag is missing. The flag is not in the systemd unit or the pod spec. The exemplar is parsed; the appender is not opened. Check: Check 3 returns no exemplar-storage entry. Fix: add the flag and restart.
4. Trace backend is not ingesting. The trace sampler is too aggressive; the trace backend is down; the trace ID is wrong. The exemplar is stored; the trace is empty. Check: Check 4 returns a zero count. Fix: raise the sampler rate or fix the trace backend.
5. Grafana data source is misconfigured. The internal link is missing or the UID is wrong. The diamond renders; the click is broken. Check: Check 5 returns no tracing key. Fix: provision the data source.
6. Grafana panel toggle is off. The panel options have showExemplars: false. The diamond is suppressed. Check: Check 6 returns false. Fix: update the panel options.

How to troubleshoot it

The troubleshooting procedure is the diagnostic order. The engineer walks the chain from Check 1 to Check 6. The first check that fails is the broken link.

The procedure

[1] Producer active span context
     curl /metrics | grep '^metric_bucket' | grep '# {trace_id'
     -> false: wrap the metric call in a span context
     -> true: continue to Check 2

[2] Exporter trailer on /metrics
     curl /metrics | grep -c '# {trace_id'
     -> false: enable the exemplar reservoir
     -> true: continue to Check 3

[3] Prometheus flag enabled
     curl /api/v1/status/runtimeinfo | jq '.data.featureFlags'
     -> false: add --enable-feature=exemplar-storage
     -> true: continue to Check 4

[4] Trace backend has the trace
     curl http://tempo:3200/api/traces/<trace_id>
     -> false: raise the sampler rate or fix the trace backend
     -> true: continue to Check 5

[5] Grafana data source configured
     curl /api/datasources/uid/prom | jq '.jsonData.internalLink'
     -> false: provision the data source
     -> true: continue to Check 6

[6] Grafana panel options enabled
     curl /api/dashboards/uid/<uid> | jq '.panels[].options.showExemplars'
     -> false: update the panel options
     -> true: chain is healthy

The diagnostic report

The diagnostic report is the output of the six checks. The team should document the diagnostic report in the runbook. The report is a single page with the six commands and the expected output.

# READ-ONLY
# Run the diagnostic report
cat > /tmp/exemplar-diag.sh <<'EOF'
#!/bin/bash
set -e

echo "=== Check 1: Producer active span context ==="
curl -sf http://checkout.svc:8080/metrics \
  | grep '^http_request_duration_seconds_bucket' \
  | grep '# {trace_id' | head -1

echo "=== Check 2: Exporter trailer on /metrics ==="
curl -sf http://checkout.svc:8080/metrics \
  | grep -c '# {trace_id'

echo "=== Check 3: Prometheus flag enabled ==="
curl -sf http://prometheus:9090/api/v1/status/runtimeinfo \
  | jq '.data.featureFlags'

echo "=== Check 4: Trace backend has the trace ==="
TRACE_ID=$(curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="1.0"}' \
  --data-urlencode 'start=2026-08-13T22:00:00Z' \
  --data-urlencode 'end=2026-08-13T22:30:00Z' \
  | jq -r '.data[0].exemplarLabels.trace_id')
curl -sf "http://tempo:3200/api/traces/$TRACE_ID" \
  | jq '.batches | length'

echo "=== Check 5: Grafana data source configured ==="
curl -sf -u admin:admin http://grafana:3000/api/datasources/uid/prom \
  | jq '.jsonData.internalLink'

echo "=== Check 6: Grafana panel options enabled ==="
curl -sf -u admin:admin \
  "http://grafana:3000/api/dashboards/uid/<uid>" \
  | jq '.dashboard.panels[] | select(.type=="histogram") | .options.showExemplars'
EOF
chmod +x /tmp/exemplar-diag.sh
/tmp/exemplar-diag.sh

The script is the runbook entry. The engineer runs the script; the script identifies the broken link; the engineer fixes the link.

Security implications

The diagnostic order does not change the privacy posture. The six checks read the same data the panels read; the privacy is the same.

The trace backend is the boundary. The trace ID is a handle; the trace is the payload. The trace backend must be authenticated; the Grafana user must have read access to the trace backend data source.

The diagnostic report is an information disclosure. The report contains trace IDs, timestamps, and label values. The report should not be sent to a public channel; the report should be retained in the team runbook.

Performance implications

The diagnostic order is cheap. Each check is one HTTP request; the total cost is six requests. The cost is paid once per incident. The cost is negligible.

The diagnostic order is also a regression test. The team that runs the diagnostic order on the production exemplar chain confirms the chain is healthy. The cost is one request per link per test run.

Production guidance

• Document the diagnostic order in the team runbook. The runbook entry should be reachable in five minutes from the on-call rotation. The entry should include the six checks and the six fixes.
• Run the diagnostic order on a known-good chain. The team should run the diagnostic order on a healthy exemplar chain to confirm the chain is working. The output is the baseline for the next failure.
• Run the diagnostic order on a known-broken chain. The team should run the diagnostic order on a broken exemplar chain to confirm the failure mode. The output is the regression test for the next failure.
• Automate the diagnostic order. The script is small enough to run as a scheduled job. The job should alert on any check that fails.
• Monitor the chain. The Prometheus server emits prometheus_target_scrape_pool_exemplar_appended_total. The Grafana audit log records the trace backend queries. The two metrics are the operational signal.
• Treat the chain as a single deployment. The chain has six links. The links are configured in different files and different tools. The team should treat the chain as a single deployment and validate the chain before the next incident.

Verification

You should now be able to answer:

• What is the diagnostic order for the missing-exemplar symptom?
• Which link in the chain is broken when the Prometheus API returns an empty array but the raw scrape has the trailer?
• Which link is broken when the diamond renders but the click opens a 404?
• How do you distinguish the producer-side failure from the wire-side failure from the server-side failure?
• What is the right runbook entry for the missing-exemplar symptom?

Quiz