Pivot from Metrics to Traces

A Grafana latency histogram for POST /checkout shows a flat line at 80 ms and a single diamond near the 1.4 s bucket at 02:47. The engineer clicks the diamond. Tempo opens. The trace shows a 1.3 s pause inside the Redis connection pool on cart-svc. The root cause is in front of them ninety seconds after the click.

The diamond is an exemplar: a small piece of metadata that the OpenTelemetry SDK attaches to the histogram bucket at the moment an observation is recorded. The exemplar is the join between “the metric says the system is slow” and “the trace shows why”. This lesson is about the pivot.

What it is

An exemplar is a reference from a single Prometheus histogram bucket to one specific trace. It is not the trace itself, not a log line, and not a full request UUID. It is the minimum metadata the trace backend needs to find the trace: the trace ID, the span ID that produced the observation, the value of the observation, and a timestamp.

In the OpenMetrics text format the exemplar rides on the bucket line after a # separator:

# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{method="POST",route="/checkout",status="200",le="1.0"} 18434 # {trace_id="aa11bb22cc33...",span_id="44556677"} 0.948 1715638801.456 1.0
http_request_duration_seconds_bucket{method="POST",route="/checkout",status="200",le="+Inf"} 18440 # {trace_id="ee99ff88dd77...",span_id="22334455"} 1.412 1715638802.044

Read the suffix left-to-right: trace_id, span_id, the observed value, the timestamp. The bucket is the join; the trace ID is the pointer; Tempo is the destination.

The pivot is the Grafana action that uses the exemplar. With a Tempo data source configured, clicking the diamond on a histogram panel opens the trace in Tempo with the right span centred. With the data source misconfigured, the diamond is decorative.

Why a sysadmin cares

The metric tells the operator the system is slow at 02:47. The trace tells the operator why the system is slow at 02:47. The exemplar is the join between the two. Without it, the operator has to:

1. Open the application log for the time window.
2. Grep for the slowest request.
3. Find the trace ID in the log line.
4. Paste the trace ID into the trace backend.
5. Hope the trace is still in retention.

That is a five-step manual workflow with three places to lose the thread. With an exemplar, the workflow is one click.

Three operational realities make the pivot important:

1. The metric is the early signal. The histogram spikes before the user reports anything. The exemplar is what makes the early signal actionable.
2. The trace is the late signal. The trace answers “why”. It is large, slow to read, and useless without a trace ID. The exemplar provides the trace ID.
3. The dashboard is the on-call surface. On-call engineers live in Grafana. The pivot from panel to trace happens in the same UI; the operator never leaves Grafana for the duration of the incident.

How it works

The pivot has four parts in three systems:

  Application
  (OTel SDK)
      |
      | records observation, attaches trace_id and span_id
      v
  OpenTelemetry Collector / Alloy
      |
      | forwards spans to Tempo, metrics to Prometheus
      v
  Prometheus                       Tempo
  (histogram with exemplar)        (trace by id)
      |                                 ^
      +-- Grafana data sources --------+
            |
            v
        Grafana panel
        click diamond --> open trace in Tempo

1. The SDK records the observation. When a request lands in a histogram bucket, the SDK captures the current trace ID and span ID and attaches them to the bucket.
2. The collector forwards both signals. Spans go to Tempo; metrics go to Prometheus. The trace ID is the join key; the SDK is the only place that knows it.
3. Prometheus stores the exemplar. The exemplar rides on the bucket line in the OpenMetrics scrape response. Prometheus requires --enable-feature=exemplar-storage to keep it; the default is to drop exemplars.
4. Grafana draws the diamond. With a Tempo data source configured, Grafana reads the exemplar from Prometheus and turns the trace ID into a click target that opens the trace.

A second pivot path goes the other way: the metrics-generator in Tempo derives RED metrics from spans and exports them to a Prometheus-compatible endpoint. This produces a Prometheus metric that is backed by span data, no application-side instrumentation required. The two paths are complementary, not redundant.

Under the hood

How to configure it

Three components must be configured for the pivot to work end-to-end.

Prometheus: enable exemplar storage

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

# Start Prometheus with the feature flag.
# prometheus \
#   --config.file=/etc/prometheus/prometheus.yml \
#   --enable-feature=exemplar-storage \
#   --storage.tsdb.path=/var/prometheus/data

Severity: SERVICE-IMPACT. The Prometheus process must be restarted to apply.

Grafana: Tempo data source with the right UID

# /etc/grafana/provisioning/datasources/tempo.yaml
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    uid: tempo
    access: proxy
    url: http://tempo.internal:3200
    isDefault: false
    jsonData:
      httpMethod: POST
      tracesToLogsV2:
        datasourceUid: loki
      serviceMap:
        datasourceUid: prometheus

The uid: tempo is referenced by the Prometheus data source configuration as the internal link target. Pick a stable UID; it becomes part of the URL operators bookmark.

Prometheus data source: link to Tempo

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    access: proxy
    url: http://prometheus.internal:9090
    jsonData:
      # Link exemplar diamonds to the Tempo data source.
      exemplarTraceIdDestinations:
        - datasourceUid: tempo
          name: traceID
          url: '$${__value.raw}'

The exemplarTraceIdDestinations block tells Grafana which data source holds the trace. The $${__value.raw} substitution becomes the trace ID from the exemplar.

How to validate it

Severity: READ-ONLY. Four checks confirm the pivot is wired end-to-end.

1. Prometheus has exemplar storage enabled and is collecting exemplars:

curl -s http://prometheus.internal:9090/api/v1/status/config \
  | jq '.data.yaml | contains("enable-feature")'

true

(The feature flag is set at the command line, not in the YAML; this check confirms the YAML reload is fine. The flag itself appears in ps.)

2. The histogram bucket has an exemplar:

curl -s 'http://prometheus.internal:9090/api/v1/query?query=
  http_request_duration_seconds_bucket{
    method="POST",
    route="/checkout",
    le="1.0"
  }' | jq '.data.result[].value'

# ["18434","1715638801.456"]

The bucket has a value. To see the exemplar, hit the exemplars endpoint:

curl -s 'http://prometheus.internal:9090/api/v1/query_exemplars?query=
  http_request_duration_seconds_bucket{
    method="POST",
    route="/checkout",
    le="1.0"
  }' | jq '.data[].exemplars[0]'

{
  "labels": [
    {"name":"trace_id","value":"ee99ff88dd77cc11aa22bb33..."},
    {"name":"span_id","value":"22334455"}
  ],
  "value": "1.412",
  "timestamp": "1715638802.044"
}

3. Tempo resolves the trace by the ID from the exemplar:

curl -s "http://tempo.internal:3200/api/traces/ee99ff88dd77cc11aa22bb33" \
  | jq '.resourceSpans | length'

1

The trace exists; the click will resolve.

4. The Grafana data source link is configured. Open a histogram panel, click a diamond, and confirm Tempo opens.

How it can fail

Six shapes appear when the pivot breaks:

1. The feature flag is not set. Prometheus drops exemplars by default. Symptom: query_exemplars returns empty for every bucket; the diamonds never appear on the panel.
2. The Tempo data source UID is wrong. Grafana resolves the click but the URL does not match any data source. Symptom: the click silently does nothing; the engineer assumes the exemplar is decorative.
3. The trace is not in Tempo. The SDK emits the exemplar but the trace was never sent to Tempo, or it aged out of retention. Symptom: clicking the diamond opens Tempo to a “trace not found” page; the engineer concludes the trace pipeline is broken.
4. The SDK does not set the trace context. A custom metric recorded without going through the OTel SDK has no trace context. Symptom: the exemplar’s trace_id is empty; the diamond still appears (because the bucket line is valid) but the click is a no-op.
5. Sampling dropped the trace. The application sampled 1% of traces; the bucket observation came from the 99% that were not exported. Symptom: the exemplar points at a trace ID that does not exist in Tempo; the click opens an empty page.
6. Exemplar storage is full. Prometheus caps exemplar storage per series. When the cap is hit, newer exemplars evict older ones. Symptom: recent diamonds are missing on busy panels; old diamonds still appear.

How to troubleshoot it

Ordered diagnostics, cheapest first:

1. Is the feature flag set? Check the Prometheus process command line (ps aux | grep prometheus). The --enable-feature=exemplar-storage flag must be present.
2. Does the bucket have an exemplar? Hit the /api/v1/query_exemplars endpoint on a known histogram. Empty result: the SDK is not setting the trace context or Prometheus is dropping the exemplar.
3. Does the trace resolve in Tempo? Take the trace ID from the exemplar and hit /api/traces/{id}. Empty result: the trace was not exported, was sampled out, or aged out of retention.
4. Is the data source UID correct? Open the Grafana data source settings and confirm the UID in the Prometheus exemplarTraceIdDestinations block matches the Tempo data source UID.
5. Is the click reaching Grafana? Open the browser developer tools on a histogram panel, click the diamond, and inspect the network tab. A 404 means the link is configured wrong; a 200 means the click worked but Tempo could not resolve the trace.

Security implications

• Trace IDs are not secrets. An exemplar exposes the trace ID, not the trace data. A user who clicks the diamond still needs authorisation to read the trace.
• Tenant boundary. A multi-tenant Grafana whose Tempo data source is misconfigured can let a user in one tenant click an exemplar from another tenant and read the trace. The X-Scope-OrgID header is the boundary; the click action must respect it.
• Exemplar storage cost. Exemplars add to Prometheus disk usage. A high-cardinality histogram with many buckets can accumulate exemplar storage fast; the cap on per-series exemplar count prevents unbounded growth but does not prevent noisy panels.

Performance implications

• Prometheus disk. Exemplar storage is bounded per series; the default is a small number of exemplars per series. The disk cost is in the order of tens of KiB per active series.
• Scrape cost. Exemplars ride on the existing scrape response; they do not add a round trip. The cost is in the parse step on the Prometheus side.
• Query cost. The /api/v1/query_exemplars endpoint is fast — it reads a small dedicated storage area and returns exemplars for the matched series.
• Grafana panel cost. A panel that renders a histogram with exemplars draws one diamond per exemplar. With dozens of series and dozens of exemplars per series, the panel render is bounded but the browser is busy.

Production guidance

• Enable --enable-feature=exemplar-storage on every Prometheus that scrapes a histogram with OTel instrumentation.
• Pick a stable Tempo data source UID; reference it from the Prometheus data source’s exemplarTraceIdDestinations. Avoid changing the UID after dashboards are built.
• Align tail-sampling decisions with metrics-generator observations so every recorded observation produces a retrievable trace.
• Use the metrics-generator to derive RED metrics from spans for services that cannot instrument Prometheus directly; the exemplars on those metrics point at traces in the same Tempo cluster.

Verification

You should now be able to answer:

• What three pieces of metadata does an exemplar carry?
• Which Prometheus feature flag enables exemplar storage?
• What is the role of the Tempo data source UID in the pivot?
• What is the most expensive exemplar misconfiguration in production?
• What is the first thing to check when a diamond click opens a “trace not found” page?

Quiz