Dashboard-to-Traces Overview · Observability

A Prometheus histogram shows p99 latency on the dashboard. The operator sees a bar, the bar is too tall, the alert is firing. The operator needs to know which request produced the bar. A trace answers that question. The problem is that the metric and the trace live in different backends and have no obvious join.

The dashboard-to-traces pivot is the mechanism that gives them one. The histogram bucket carries a small piece of metadata called an exemplar: the trace ID and span ID of the request that landed in that bucket at that scrape. Grafana reads the metadata, draws a diamond on the bar, and turns the click on the diamond into a navigation event to a Tempo trace. The operator goes from bar to trace in one click.

This lesson is the mental model that frames the next five. The workflow has four moving parts, and missing any one of them silently disables the join. The lesson also describes the order in which the parts must be built — because the failure where the diamond never appears has more than one cause and the diagnostic order matters.

What it is

The dashboard-to-traces pivot is the Grafana-side join between two telemetry backends: Prometheus (the metric) and Tempo (the trace). The join is the exemplar. An exemplar is a small reference from a single histogram datapoint to one specific trace. It is not full trace data; it is a pointer that says “this trace represents this observation.”

  Prometheus histogram bar (one scrape, one bucket)
        |
        |  # {trace_id="4bf92f3a...",span_id="00ff00..."} 0.421
        |       |
        |       +----- exemplar: a reference, not the trace
        |
  Grafana renders the bar and a diamond on top of it
        |
  Operator clicks the diamond
        |
  Grafana routes to /explore?ds=<tempo_uid>&traceID=<trace_id>
        |
  Tempo serves the trace back to the operator

The pivot is one-directional at the diagram level (metric points at trace). The opposite direction — from a span in a trace back to the logs that share its trace ID — exists and is the tracesToLogsV1 block in the Tempo data source. This module focuses on the metric-to-trace direction. The lessons in Part LIII cover the trace-to-logs direction.

What the pivot is not: it is not a query language relationship, it is not a label join, and it does not require the metric and trace to share a label taxonomy. The exemplar carries the trace ID directly. The label taxonomy only needs to be self-consistent inside each backend.

Why a sysadmin cares

Three operational payoffs depend on the pivot being correct when the alert fires at 03:00:

Time-to-root-cause collapses. The investigation no longer starts by reconstructing what happened. The metric panel points to the trace. The trace shows the dependency span, the database span, the cache miss, the slow DNS lookup. The operator is reading evidence, not guessing.
False-positive triage is cheap. A single tall bar across an otherwise flat curve is almost always a tail request, not a regression. Clicking the diamond shows a 4.2 s outlier that was an upstream client’s connection retry rather than a service bug. The operator silences the alert in minutes, not hours.
Post-incident review has a single artefact. The histogram timestamp, the alert URL, the exemplar trace ID, and the trace detail all fit in one incident document. The reviewer does not need to rebuild the narrative from logs and metric panels that have aged out of the dashboard.

The cost of getting the pivot wrong is paid in incident minutes. The cost of getting it right is paid once during setup. Setup is the lesson.

How it works

The pivot has four components, in the order an HTTP request walks through them:

  +------------------+        +----------------+        +----------------+
  | Producer         |        | Prometheus     |        | Grafana        |
  | application or   |  --->  | scrapes /metrics| --->  | renders panel  |
  | sidecar emits    |        | stores exemplars|       | draws diamond  |
  | histogram +      |        | alongside      |        | routes click   |
  | trace context    |        | bucket counter |        | to data source |
  +------------------+        +----------------+        +----------------+
                                                              |
                                                              v
                                                         +----------------+
                                                         | Tempo          |
                                                         | receives spans |
                                                         | indexes by     |
                                                         | trace ID and   |
                                                         | serves trace   |
                                                         +----------------+

Producer. The service exposes a Prometheus histogram over its /metrics endpoint. When the service records an observation, it attaches the active W3C trace context to the bucket. The OpenTelemetry SDK does this by default; the prometheus/client_golang library does this by default; the Python prometheus_client requires a small bridge.
Prometheus. The Prometheus 2.55.x server, started with --enable-feature=exemplar-storage, parses the bucket lines and the exemplar suffix and writes both into its append-only TSDB. Without the flag, the suffix is parsed and dropped.
Grafana. The Prometheus data source is configured with an Internal link that names the trace data source UID. When the panel renders a histogram, Grafana overlays a diamond on any bar that has an exemplar. Clicking the diamond navigates to /explore with the captured trace ID.
Tempo. The trace backend has received the spans via the OTLP pipeline (or Grafana Alloy in its 0.110.x configuration). When Grafana calls /api/traces/<trace_id>, Tempo returns the spans.

Missing any of the four disables the join. The diamond is absent. The investigation reverts to grep.

Under the hood: what the exemplar actually contains

The exemplar suffix on a bucket line is small and bounded. In the OpenMetrics text exposition format it looks like this:

# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{method="POST",route="/checkout",status="200",le="0.5"} 18432 # {trace_id="4bf92f3a7c1b09e0",span_id="00ff00a1b2c3"} 0.421 1715638800.123
http_request_duration_seconds_bucket{method="POST",route="/checkout",status="200",le="1.0"} 18434 # {trace_id="aa11bb22cc33dd44",span_id="44556677aabb"} 0.948 1715638801.456

The suffix carries four pieces of information:

trace_id — the W3C traceparent of the request that produced this observation. Tempo uses this to find the trace.
span_id — the span inside the trace that produced the observation. Useful when the trace has thousands of spans and the operator wants to land directly on the slow one.
The numeric value — the actual observation that produced this exemplar. Not a fraction of the bucket counter.
The timestamp — sub-second Unix time of the observation. Used to align the diamond with the bar on the chart.

The exemplar is bound to the bucket, not the count. A bucket that received 500 observations holds one exemplar (representative), chosen by the SDK’s reservoir. This is why a bar can have a diamond without exemplars covering every observation that contributed.

Only histograms support exemplars. Counters, gauges, and summaries do not. A counter tells you “how many”; a histogram tells you “how long and how often in each band”; only the latter has a representative observation worth pointing at.

How to configure it

Setup is in four steps. Each step has a single validation that catches its specific failure mode.

1. Instrument the histogram with a traced SDK

For OpenTelemetry SDKs, the default histogram recording already attaches the active trace context. For Go services using prometheus/client_golang, the histogram observer gets the exemplar attached automatically when a span is open. For Python, the OpenTelemetry Prometheus bridge is the path.

# Python excerpt — instrumenting a Flask handler with the
# OTel Prometheus bridge so exemplars carry the active trace.
# Verified on opentelemetry-sdk 1.27.x and prometheus_client 0.20.x.
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.prometheus import PrometheusInstrumentor
from prometheus_client import Histogram

REQUEST_LATENCY = Histogram(
    "http_request_duration_seconds",
    "Time spent handling HTTP requests",
    ["method", "route", "status"],
    buckets=(0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0),
)

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
# The bridge below is what attaches the active trace context
# to every histogram observation, producing the exemplar.
PrometheusInstrumentor().instrument(app=app, histogram_defs={"http_request_duration_seconds": REQUEST_LATENCY})

The key line is the PrometheusInstrumentor registration. Without it, the histogram is updated but the trace context is not attached, and the bucket line will not have the suffix.

2. Enable exemplar storage on the Prometheus server

The Prometheus 2.55.x server does not write exemplars by default. The flag must be added to the systemd unit or the pod specification.

# CONFIGURATION — add to the systemd unit or container args
/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --enable-feature=exemplar-storage

Severity: SERVICE-IMPACT. The flag change requires a server restart to take effect.

3. Configure the Grafana Internal link

The Prometheus data source in Grafana 11.x has a section called “Internal link” under the Exemplars heading. The data source UID must reference the Tempo data source configured elsewhere in the same Grafana instance.

# Grafana provisioning file (CONFIGURATION)
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prom
    url: http://prometheus:9090
    jsonData:
      httpMethod: POST
      internalLink:
        tracing:
          dataSourceUid: tempo
          # Optional override: the label name Tempo expects
          label: traceID
          # Optional override: which label holds the span ID
          spanId: spanID

4. Confirm the trace backend holds the spans

The pivot only works if the trace pointed at by the exemplar actually exists in Tempo. OpenTelemetry Collector 0.110.x (or Grafana Alloy in the same role) carries the spans over OTLP. The minimal pipeline is:

# OpenTelemetry Collector config (CONFIGURATION)
# Verified on opentelemetry-collector-contrib 0.110.x
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/tempo]

The exporter’s endpoint must be reachable from the collector and the Tempo ingester must be listening on the matching OTLP gRPC port.

How to validate it

Three validation layers, each catching a different missing component.

1. The producer exposes exemplars on `/metrics`

# READ-ONLY
curl -sf http://checkout.svc:8080/metrics \
  | grep '^http_request_duration_seconds_bucket' \
  | grep -c '# {trace_id'

Expected: a positive integer. Zero means the SDK is not attaching the trace context. The histogram is being recorded but the exemplar suffix is absent.

2. Prometheus is storing exemplars

# READ-ONLY
curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="1.0"}' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq '.data | length'

Expected: a positive integer — the count of exemplars returned for the time range. Zero means either the flag is not enabled or the producer is not emitting.

3. The trace ID resolves in Tempo

# READ-ONLY
TRACE_ID=$(curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="1.0"}' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq -r '.data[0].exemplarLabels.trace_id')

curl -sf "http://tempo:3200/api/traces/$TRACE_ID" | jq '.traces | length'

Expected: 1 or higher. Zero means the producer is emitting exemplars but the spans are not arriving at Tempo — likely an OTLP collector problem or a sampling decision at the producer.

How it can fail

Six failure modes, ordered by frequency in real environments.

Exemplar storage flag missing on Prometheus. The producer emits # {trace_id="..."} on /metrics. Prometheus parses the lines, drops the suffix. The query_exemplars API returns an empty array. Symptom: the diamond never appears on any panel; the API is empty for every timestamp.
Producer lacks an active span context at the point of the histogram observation. The bucket is incremented but no span is open, so the suffix is not written. Symptom: /metrics shows the histogram with no # {trace_id suffix on any line. The histogram is fine; the exemplar is absent on every line.
Head sampler rejects the trace, but the histogram is recorded. The OpenTelemetry head sampler makes the decision after the metric is updated. The metric is present, the trace is not in Tempo. Symptom: query_exemplars returns data but the trace ID resolves to nothing in Tempo.
Internal link UID does not match any data source. The Prometheus data source has the Exemplars section enabled, but the dataSourceUid references a UID that is not registered. Symptom: the diamond renders; the click opens an Explore page with the trace ID but no trace data.
Trace ID label name mismatch. Grafana 11.x defaults to traceID; the producer uses trace_id. Grafana builds the link with the wrong field. Symptom: the diamond renders; the click opens Tempo with an empty trace ID query.
Exemplar retention has expired. The appender file holds exemplars for a bounded period (default 15 minutes in Prometheus 2.55.x). Symptom: the diamond is present for recent activity; absent for anything older than the retention window.

How to troubleshoot it

Steps in order from cheapest to most expensive. Each step isolates one of the six failure modes.

Confirm the producer emits exemplars. curl /metrics and grep for # {trace_id. If the suffix is absent, the producer is the problem — not Prometheus, not Grafana.
Confirm the Prometheus flag is enabled. curl /api/v1/status/runtimeinfo and inspect featureFlags.exemplar-storage. If absent, the appender is not being written. Restart with the flag.
Confirm the API returns data. query_exemplars with the same query the panel uses. Empty array with a healthy producer is a Prometheus-side problem.
Confirm the data source link is configured. Open Prometheus data source settings in Grafana. Inspect the Internal link section. Cross-check the UID against the Tempo data source configuration; mismatches are the fourth failure mode.
Confirm the trace backend holds the trace. Take a trace ID from one exemplar and curl /api/traces/<id> on Tempo. If the trace is absent, the OTLP pipeline or sampling is the problem; not the pivot.
Confirm the panel has exemplars enabled. In Grafana 11.x, the panel’s Exemplars option must be enabled. This is independent of the data source configuration and is a common silent disable.

Security implications

The pivot does not introduce new authentication surfaces. The exemplar is a metadata field in Prometheus’s append-only TSDB; the trace ID is read inside an authenticated Grafana session and routed to Tempo via a configured data source. The authorisation question is “who can see this data source?”

Three things to check:

The Prometheus data source’s internal link UID references a Tempo data source that the same team can read. A misconfig here causes a trace to open for a user who has not been granted trace access. Verify in the data source’s permissions tab.
The trace ID is not, in itself, secret. A trace ID is not an authentication token. But the trace it points at may carry PII (request bodies, error messages, user IDs). Treat the trace’s data as you would logs of the same service.
The exemplar storage on Prometheus 2.55.x is local to the TSDB. If the TSDB is shipped off-host (remote write, Thanos), exemplars are not currently replicated downstream. Confirm what your remote-write receiver does with the exemplar suffix.

Performance implications

Performance cost has three parts, each small but real.

Producer overhead. Attaching the trace context to a histogram observation is two pointer reads and a struct write. On hot request paths the overhead is measurable but below 1% on a modern x86_64 core. Disable on producers that have a tight CPU budget and are not affected by tail-latency investigations.
Prometheus TSDB. The append-only exemplar file holds at most a few hundred bytes per exemplar. Default retention is 15 minutes. The cost is bounded by design. The bound is honoured by rate-limiting exemplar writes, not by retention; sustained high exemplar volume will be rejected at the appender.
Grafana panel render. The diamond is a small SVG overlay. Render cost is negligible per panel. Cost grows with the number of distinct label combinations visible in the histogram — but that is the cost of the histogram, not the cost of the pivot.

Production guidance

Build the four components in the order producer, flag, data source, trace backend. Validate each before moving to the next. Do not deploy all four simultaneously; a partial failure is then impossible to attribute.
Pin the bucket layout to one that aligns with the investigation patterns you expect. A p99 alert on a histogram with buckets (0.1, 0.25, 0.5) has no useful bucket to point at when the alert fires at 1.4 s; pick buckets that bracket the alert thresholds.
Treat the pivot as observable plumbing, not a one-time configuration. The producer SDK, the Prometheus flag, and the Grafana data source are each owned by a different team; the pivot breaks when any one of them ships a change without considering the others.

Verification

You should now be able to answer:

What are the four components that must be present for the pivot to function end-to-end?
Why do counters, gauges, and summaries not produce exemplars?
What is the role of --enable-feature=exemplar-storage and what happens when it is absent?
How does Grafana know which trace backend to route to when the diamond is clicked?