Metric to Trace Workflow

The p99 latency for the checkout endpoint is 4.2 seconds. The panel shows the spike. The on-call engineer wants to know which trace produced the spike. There is no trace panel on the dashboard. The engineer has to click into one of the buckets, guess the trace ID, paste it into Tempo. The pivot does not exist on the panel. The investigation dies at the histogram.

The metric-to-trace pivot closes the gap. The pivot is the exemplar. An exemplar is a sample attached to a histogram bucket that carries the trace ID of the observation that produced the value. The panellist clicks the bucket dot in Grafana, the pivot opens Tempo filtered by the trace ID, and the trace is in front of them. The drill is one click.

What it is

The metric-to-trace pivot is a Grafana drill that opens a Tempo trace from a Prometheus histogram bucket. The mechanism is the OpenMetrics exemplar. The exemplar is a single observation attached to a histogram bucket, carrying the trace_id of the request that produced the sample.

  Prometheus histogram
  +--------------------------------------+
  | http_server_request_duration_seconds |
  |   bucket(le=5.0)  count=243          |
  |   bucket(le=1.0)  count=120          |
  |   bucket(le=0.5)  count=80           |
  |     # {trace_id="0af76519..."} 4.2   |
  |     +-- exemplar attached to the bucket
  +----------------------+---------------+
                         |
                         | click
                         v
  Tempo query
  +--------------------------------------+
  | trace_id = 0af7651916cd43dd8448eb211c80319c
  +--------------------------------------+

The OpenMetrics text format carries the exemplar as a suffix on the bucket line:

http_server_request_duration_seconds_bucket{le="5.0",job="checkout"} 243
  # {trace_id="0af7651916cd43dd8448eb211c80319c"} 4.2 0001-01-01T00:00:00Z

The # {trace_id="..."} value timestamp is the exemplar. Prometheus stores it. Grafana renders it as a dot on the histogram. The panellist clicks the dot and the pivot opens Tempo.

Why a sysadmin cares

The metric-to-trace pivot is the only drill that lands on a specific request from a histogram. Every other pivot lands on a service, an instance, or a label set. The exemplar lands on one request. The drill is the closest thing the metric side has to a database query.

Three operational payoffs.

1. The latency investigation starts from a real request. The exemplar is the actual observation that produced the spike. The trace is the same request. The drill is from real evidence to real evidence.
2. The floor of detail is bounded. The pivot always lands on a trace. The trace could be slow, the trace could be fast, the trace could be an error. The pivot is a fixed drill. The investigation is bounded.
3. The retry-shape and the timeout-shape are visible. The histogram bucket alone does not show why the request was slow. The trace shows the slow span. The pivot is the bridge between the aggregate and the detail.

The cost is the discipline of attaching an exemplar to every histogram observation. The investment is a single line of application code per histogram. The return is one drill that works on every latency investigation.

How it works — the exemplar lifecycle

The lifecycle has four steps.

1. Application observes. The instrumented code records an observation into a histogram. The OpenTelemetry SDK attaches the active span context to the observation.
2. Exporter carries the trace_id. The OpenMetrics exporter writes the bucket line followed by the exemplar line. The exemplar carries the trace_id and the observation value.
3. Prometheus stores the exemplar. Prometheus parses the OpenMetrics text, extracts the exemplar, and stores it alongside the bucket. The storage is bounded: one exemplar per bucket per series per scrape.
4. Grafana renders the dot. The histogram panel renders the exemplar as a clickable dot. The panellist clicks the dot and the pivot opens Tempo.

  application
     |
     | observe(value, ctx)
     v
  OpenTelemetry SDK
     |
     | exporter(s) sends OpenMetrics text
     v
  Prometheus scrape
     |
     | parse + store
     v
  Prometheus TSDB
     |
     | query: exemplar
     v
  Grafana histogram panel
     |
     | click on dot
     v
  Tempo trace

The pivot URL is the same shape as the trace-to-logs drill, opened in reverse. The URL uses the __value.raw substitution to carry the trace ID into the Tempo query.

/explore?schemaVersion=1&panes=%7B%22traces%22%3A%7B%22datasource%22%3A%22tempo%22%2C%22queries%22%3A%5B%7B%22query%22%3A%22$${__value.raw}%22%2C%22queryType%22%3A%22traceql%22%7D%5D%7D%7D

How to configure it

The Prometheus side — the feature flag is enabled and the storage is sized:

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

# The feature flag is on by default in Prometheus 2.40+.
# The storage limit must be set explicitly.
# --storage.exemplar-exemplars-limit=100000

The Grafana side — the Prometheus data source has the exemplar toggle and the data link. The Prometheus data source is provisioned with the exemplarTraceIdDestinations field:

# grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    url: http://prometheus:9090
    jsonData:
      httpMethod: POST
      # The exemplar trace ID is rendered as a clickable link.
      # The destination datasource is the Tempo trace explorer.
      exemplarTraceIdDestinations:
        - name: trace_id
          datasourceUid: tempo
          urlDisplayLabel: 'Open trace'

The histogram panel is configured to show exemplars. The relevant toggle in the panel editor is Exemplars → Show exemplars. The toggle is set to On in the panel JSON:

{
  "type": "histogram",
  "targets": [
    {
      "expr": "histogram_quantile(0.99, sum by(le, job, instance) (rate(http_server_request_duration_seconds_bucket{job=\"checkout\"}[1m])))",
      "legendFormat": "{{job}} {{instance}}"
    }
  ],
  "options": {
    "exemplars": true,
    "dataLinks": [
      {
        "title": "Trace for this exemplar",
        "url": "/explore?schemaVersion=1&panes=%7B%22traces%22%3A%7B%22datasource%22%3A%22tempo%22%2C%22queries%22%3A%5B%7B%22query%22%3A%22$${__value.raw}%22%2C%22queryType%22%3A%22traceql%22%7D%5D%7D%7D&orgId=1"
      }
    ]
  }
}

The application side — the OpenTelemetry SDK attaches the active span context to every observation. In Go:

import (
    "go.opentelemetry.io/otel/trace"
    "go.opentelemetry.io/otel/metric/instrument"
)

// The instrumented histogram observes a value while the
// active span context is on the goroutine. The exporter
// reads the context and attaches it as the exemplar.
httpDuration.Record(ctx, elapsed.Seconds())

In Python with the OTel SDK:

from opentelemetry import trace
from opentelemetry.metrics import get_meter

meter = get_meter("checkout")
histogram = meter.create_histogram(
    name="http.server.request.duration",
    unit="s",
)

# The active span context is captured by the SDK and
# attached as the exemplar on the bucket.
with trace.get_tracer("checkout").start_as_current_span("checkout"):
    histogram.record(elapsed, attributes={"http.method": "POST"})

How to validate it

# 1. The Prometheus exemplar storage is enabled.
curl -s http://prometheus:9090/api/v1/status/config | \
  jq '.data.yaml | test("exemplar-exemplars-limit")'
# true

# 2. The exemplar is present on the histogram bucket.
curl -s 'http://prometheus:9090/api/v1/query?query=http_server_request_duration_seconds_bucket' \
  | jq '.data.result[0].exemplar'
# {"labels":{"trace_id":"0af7651916cd43dd8448eb211c80319c"},"value":"4.2"}

# 3. The trace ID is present in Tempo.
tempo-cli query '{ trace = "0af7651916cd43dd8448eb211c80319c" }'
# Span: 0af7651916cd43dd8448eb211c80319c  service=checkout  duration=4.2s

# 4. The Grafana data source has the exemplar destination.
curl -s -u admin:admin http://grafana:3000/api/datasources/uid/prometheus \
  | jq '.jsonData.exemplarTraceIdDestinations'
# [{"name":"trace_id","datasourceUid":"tempo","urlDisplayLabel":"Open trace"}]

# 5. The dashboard panel has the exemplars toggle on.
curl -s -u admin:admin http://grafana:3000/api/dashboards/uid/checkout \
  | jq '.dashboard.panels[] | select(.type=="histogram") | .options.exemplars'
# true

How it can fail

Six recurring failure shapes.

1. The Prometheus exemplar storage is disabled. The feature flag is off or the storage limit is zero. Symptom: the /api/v1/query endpoint returns no exemplar field. The histogram panel shows no dots.
2. The application does not stamp the trace context. The active span context is missing on the goroutine that records the observation. Symptom: the exemplar is present but the trace_id value is empty. The pivot opens Tempo with an empty query.
3. The Grafana exemplar destination is wrong. The exemplarTraceIdDestinations field points to a data source UID that does not exist. Symptom: the panel renders the exemplar dots without clickable links.
4. The histogram is a summary, not a histogram. The Prometheus summary type does not carry exemplars. Only the histogram type does. Symptom: the # TYPE line in the OpenMetrics text says summary; the exemplar is never emitted.
5. The exemplar is overwritten faster than the operator clicks. The storage bound is one exemplar per bucket per scrape. At high scrape frequency, the exemplar the operator saw five minutes ago is gone. Symptom: the histogram shows a dot, the operator clicks, the trace is for a different request than the one on the panel.
6. The Tempo data source changes its query API. A Tempo upgrade moves the trace query from traceId to traceql. The data link still uses the old parameter. Symptom: the pivot opens Explore but the Tempo query returns no traces.

How to troubleshoot it

The diagnostic order is “is the exemplar on the wire?”, “is Prometheus storing it?”, “is Grafana rendering it?”, “is the pivot URL right?”.

1. Is the exemplar on the wire? curl -s http://app:8080/metrics | grep -A1 _bucket | grep trace_id. The OpenMetrics text format emits the exemplar line right after the bucket line.
2. Is Prometheus storing it? curl -s 'http://prometheus:9090/api/v1/query?query=..._bucket' | jq '.data.result[0].exemplar'. The exemplar field should be populated. If it is null, the storage limit is the suspect.
3. Is Grafana rendering it? Open the dashboard, hover over the histogram. The dots are visible. If the dots are missing, the panel’s exemplars toggle is off.
4. Is the pivot URL right? Click an exemplar dot. The URL should contain the trace ID after the query= parameter. If the URL is empty, the data link uses the wrong substitution.
5. Is the Tempo query accepting the trace ID? Open the Tempo query directly with the trace ID. The trace should be returned. If Tempo returns no trace, the trace ID is not in Tempo’s storage.

Security implications

The exemplar carries the trace_id, which is a 128-bit random value. The trace_id is opaque and has no semantic content. It is safe to log, safe to forward, and safe to store.

The risk is around the labels that the histogram carries. A histogram labelled with user_id, tenant_id, or account_id is a fingerprint that survives the metric. The mitigation is to keep the label set on the histogram limited to the operational labels (job, instance, status, method, path). The user-identifying labels belong on the trace, not on the metric.

The second-order risk is around the OTel SDK’s trace context extraction. The SDK reads the active span context from the goroutine. A misconfigured custom span propagator could extract a trace_id from a different request. The mitigation is to use the SDK’s default context propagation and to explicitly pass the context.Context to the observation function.

Performance implications

The exemplar is attached to a single observation per bucket per scrape. The cost is one text suffix per bucket per scrape. At 100 buckets per series and 10 000 series, the wire cost is roughly 1 MB per scrape. The Prometheus storage cost is one exemplar per bucket per series, bounded by the exemplar-exemplars-limit flag.

The OTel SDK cost is the cost of reading the active span context on the observation goroutine. The context is on the goroutine anyway; reading the trace_id from the context is sub-microsecond. The cost is negligible.

The cost to watch is the cardinality of the histogram. A histogram with 200 buckets × 50 label combinations × 10 000 series produces 100 million exemplars on the wire. The mitigation is to keep the bucket count reasonable and the label set small.

Production guidance

• Enable the Prometheus exemplar storage. The flag is off by default in older Prometheus builds. The flag is on by default in 2.55.x but the storage limit is zero. The limit must be set to a positive value.
• Standardise on the histogram type. The summary type does not carry exemplars. The Prometheus best practice is to use the histogram type for any latency metric that will be pivoted to a trace.
• Use the OTel SDK default span context. The default propagator reads the trace_id from the active context. A custom propagator that uses a different mechanism breaks the exemplar.
• Validate the pivot under load. A panel that shows exemplars at 1 RPS does not show exemplars at 10 000 RPS. The storage bound kicks in. The validation is a load test that asserts the exemplar is present on the bucket after the storage is saturated.

Verification

You should now be able to answer:

• What is an OpenMetrics exemplar and what does it carry?
• Which Prometheus CLI flag controls the exemplar storage limit?
• Why does the pivot use the __value.raw substitution instead of __series.labels?
• What is the failure shape of a histogram whose producer does not attach the span context?
• How do you validate the exemplar end to end with a synthetic request?

Quiz