Grafana Exemplar Linking

At 09:38 a checkout service started returning 503s on 2% of requests. The error counter on the dashboard climbed. The latency histogram showed a thin tail of 1.5 s outliers. The on-call engineer hovered the tail bar in Grafana. The diamond was there. The engineer clicked. The Explore page opened. The trace from the dashboard was supposed to be from a failed POST to /checkout, but the trace that loaded was from a successful GET to /healthz. The diamond was correct; the route label was wrong.

The team had two dashboards pointing at the same Prometheus data source but with different exemplars configurations. One dashboard used the default traceID label; the other was configured to use trace_id. The exporter was emitting the trace ID under the label trace_id. Grafana was looking for traceID. The diamond rendered; the click opened the wrong trace.

The fix was to align the label name on the Grafana data source with the exporter convention. Total time to diagnose: 90 minutes. The lesson is about the configuration that makes the diamond render correctly and the click open the right trace.

What it is

Grafana 11.x renders exemplars on a histogram panel as a small diamond at the (bucket, timestamp) coordinate. The diamond is not a separate panel; it is an overlay on the histogram bar. The diamond is rendered only when the data source query returns exemplars alongside the buckets.

The diamond is a clickable element. The click handler is configured on the Prometheus data source under the “Internal link” section. The configuration is a single binding:

• The Prometheus data source UID.
• The trace backend data source UID.
• The label name on the Prometheus exemplar that holds the trace ID (default traceID).

When the click handler fires, Grafana builds a URL of the form /explore?schemaVersion=1&panes={"trace":{"datasource":"<uid>","queries":[{"query":"<trace_id>"}]}} and navigates the browser to it. The Explore page opens the trace backend data source with the trace ID as the query.

Why a sysadmin cares

The diamond is the only visual element in Grafana that connects an aggregate metric to a specific trace. Without the diamond, the on-call engineer is reduced to copying the timestamp from the panel into the trace backend’s search field and waiting for the result. With the diamond, the engineer is one click away from the trace.

The configuration is also the boundary between the metric namespace and the trace namespace. A team that has not configured the internal link has a metric dashboard that shows histograms but a trace backend that holds traces. The two never connect. The click handler is the wire.

How it works

The mental model is a histogram bar with an optional diamond, and a click handler that opens the trace:

Grafana renders histogram panel
  -> query Prometheus with histogram metric
  -> Prometheus returns buckets + exemplars
  -> Grafana plots buckets as bars
  -> Grafana plots exemplars as diamonds on the bars
  -> user clicks diamond
  -> Grafana reads the trace ID from the exemplar label
  -> Grafana builds the Explore URL
  -> Grafana navigates to the Explore page
  -> Tempo (or Jaeger) loads the trace
  -> user is on the trace page

The two requests are independent. The Prometheus query is the same; the exemplar is a side-channel of the response. The Tempo query is a new request, fired by the click.

The internal link configuration

The internal link is configured on the Prometheus data source. In Grafana 11.x, the configuration is in the data source settings under “Exemplars”:

# Grafana provisioning file (CONFIGURATION)
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prom
    url: http://prometheus:9090
    jsonData:
      httpMethod: POST
      internalLink:
        tracing:
          # The UID of the trace backend data source
          dataSourceUid: tempo
          # The label name on the Prometheus exemplar
          # that holds the trace ID. Default is traceID.
          label: traceID
          # Optional: the label name for the span ID.
          # Default is spanID.
          spanId: spanID

The configuration is per data source, not per panel. Every panel that queries the Prometheus data source inherits the configuration. The team that wants to use a different trace backend for different dashboards must configure multiple Prometheus data sources.

The panel options

The histogram panel has a “Show exemplars” toggle in the panel options. The toggle is on by default in Grafana 11.x for the histogram visualisation. The toggle is also per-panel; the data source setting is the default but the panel can override it.

The panel options also include:

• Max exemplars — the maximum number of exemplars rendered per bar. The default is 1. Increasing the value renders multiple diamonds on the same bar at the same timestamp; the visual cost is clutter.
• Exemplar colour — the colour of the diamond on the panel. The default is the same colour as the bar.
• Exemplar query range — the time range of the exemplar query. The default is the same range as the panel. The team can shorten the range to focus the exemplars on a sub-window.

The label-name mismatch

The most common production failure of the exemplar linking is the label-name mismatch. The exporter emits the trace ID under one label; Grafana looks for it under another. The diamond renders (because the exemplar is present) but the click opens the wrong trace (or no trace, because the trace ID is empty).

The convention is trace_id (snake case). The Grafana default is traceID (camel case). The mismatch is the default. The team must explicitly set the label name in the internal link configuration.

How to configure it

Two configurations: the Prometheus data source, and the Grafana panel.

1. Prometheus data source.

# Grafana provisioning file (CONFIGURATION)
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prom
    url: http://prometheus:9090
    jsonData:
      httpMethod: POST
      internalLink:
        tracing:
          dataSourceUid: tempo
          label: trace_id
          spanId: span_id

The dataSourceUid must match the UID of the trace backend data source. The label must match the label name on the exemplar. The spanId is optional.

2. Grafana panel.

The histogram panel options are accessible in the panel editor under “Visualisation options”. The relevant options:

{
  "type": "histogram",
  "options": {
    "showExemplars": true,
    "exemplarColor": "rgba(255, 0, 0, 0.7)",
    "maxExemplars": 1
  }
}

The showExemplars toggle is on by default. The maxExemplars is the limit per bar. The exemplarColor is the diamond colour.

3. Trace backend data source.

The Prometheus data source must point at a valid trace backend data source. The trace backend data source has its own configuration:

apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo:3200
    jsonData:
      httpMethod: GET
      # The trace backend must accept the trace ID as
      # a query parameter. Tempo accepts:
      #   /api/traces/<trace_id>
      tracesToLogsV2:
        datasourceUid: loki

The UID must match the dataSourceUid in the Prometheus internal link configuration. The two are independent configuration keys; the team must keep them aligned.

How to validate it

Three layers of validation, each catching a different failure mode.

1. The Prometheus data source returns exemplars.

# READ-ONLY
curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="1.0"}' \
  --data-urlencode 'start=2026-08-13T09:00:00Z' \
  --data-urlencode 'end=2026-08-13T09:30:00Z' \
  | jq '.data[0].exemplarLabels'

Expected: an object with trace_id and span_id keys. If the keys are absent, the data source is not configured or the Prometheus server is not storing exemplars.

2. The Grafana data source has the internal link.

# READ-ONLY
curl -sf -u admin:admin http://grafana:3000/api/datasources/uid/prom \
  | jq '.jsonData.internalLink'

Expected: an object with a tracing key, a valid dataSourceUid, and a label matching the exemplar label name. If the tracing key is missing, the internal link is not configured.

3. The trace backend holds the trace.

# READ-ONLY
# Take a trace ID from a real exemplar
TRACE_ID=$(curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="1.0"}' \
  --data-urlencode 'start=2026-08-13T09:00:00Z' \
  --data-urlencode 'end=2026-08-13T09:30:00Z' \
  | jq -r '.data[0].exemplarLabels.trace_id')

curl -sf "http://tempo:3200/api/traces/$TRACE_ID" \
  | jq '.batches | length'

Expected: a non-zero count. A zero count means the trace backend does not have the trace.

4. The end-to-end click works.

Open the dashboard in Grafana. Hover the bar. The diamond appears. Click the diamond. The Explore page opens with the trace loaded. The trace is the one represented by the exemplar; the route and status labels match the bar.

The click is the only end-to-end check. The API responses can be present without the click working; the panel options can be enabled without the trace backend containing the trace.

How it can fail

Six failure modes, ordered by frequency.

1. Label name mismatch. The exporter uses trace_id; Grafana looks for traceID. The diamond renders; the click opens the wrong trace or no trace. Symptom: the diamond is present; the click navigates to a blank Explore page or a 404.
2. dataSourceUid mismatch. The Prometheus data source points at a Tempo UID that does not exist. The diamond renders; the click opens an error toast. Symptom: the click handler fires; the data source lookup fails.
3. Trace backend does not contain the trace. The exemplar is from a request that the trace backend dropped. The diamond renders; the click opens a 404 from the trace backend. Symptom: the click navigates to the right page; the trace is empty.
4. Histogram visualisation not selected. The panel is a bar chart or a time series, not a histogram. The exemplars are not rendered at all. Symptom: the panel shows the bars; no diamond appears.
5. Panel toggle is off. The panel options have showExemplars: false. The diamonds are suppressed. Symptom: the same dashboard has diamonds on every other panel; this panel has none.
6. Authentication mismatch. The Grafana user can read the Prometheus data source but cannot read the Tempo data source. The click opens the Explore page; the trace query fails with a 403. Symptom: the click navigates to the right page; the trace is forbidden.

How to troubleshoot it

Steps in order from cheapest to most expensive.

1. Confirm the diamond renders. Hover the bar. If the diamond is absent, the panel options are wrong. Open the panel editor and verify showExemplars is enabled.
2. Confirm the data source is configured. The Prometheus data source must have an internal link. Open the data source settings and inspect the “Exemplars” section.
3. Confirm the label name matches. Read the data source configuration and the exporter convention. The label name on the internal link must match the label name on the exemplar. The exporter convention is trace_id; the Grafana default is traceID.
4. Confirm the trace backend has the trace. Take a trace ID from the exemplar and curl the trace backend directly. If the trace is absent, the trace sampler is too aggressive or the trace backend is not ingesting.
5. Confirm the trace backend data source UID matches. The Prometheus internal link dataSourceUid must match the UID of the trace backend data source. Mismatched UIDs are silent.
6. Confirm the user has access to both data sources. The Grafana user must have read access to the trace backend data source. Role-based access is configured per data source.

Security implications

The click handler navigates to a trace backend that holds production traces. The Grafana user must have read access to the trace backend data source. A user with read access to the Prometheus data source but not to the trace backend data source will see the diamond render but the click will fail with a 403.

The trace backend may hold PII. The exemplar trace_id is a handle; the trace is the payload. The trace backend should be authenticated; the Grafana role-based access should be configured to align with the trace backend’s data residency policy.

The internal link is a URL that crosses trust boundaries inside the same Grafana instance. The URL contains the trace ID; the trace ID is not a secret, but the URL is logged in the browser history and in the Grafana audit log. The audit log should be reviewed for exemplar-clicks on sensitive data.

Performance implications

The exemplar rendering is cheap. The diamonds are drawn on top of the bars; the rendering cost is negligible. The click handler is one navigation; the cost is the trace backend query.

The trace backend query is the cost. Each exempl ar click fires a query against the trace backend. The query is a single trace ID lookup; the trace backend returns the trace. The trace backend must be sized for the peak click rate, not the peak ingest rate.

A team that expects 1,000 clicks per hour on a panel should size the trace backend for 1,000 trace retrievals per hour. The trace storage is sized for the ingest rate, not the retrieval rate. The two are independent.

The Grafana query load is also a factor. Each panel refresh fires a Prometheus query for the buckets and the exemplars. The exemplar query is a sub-query of the bucket query; the cost is proportional to the panel count. A 50-panel dashboard with histograms produces 50 exemplar queries per refresh.

Production guidance

• Align the label name with the exporter convention. The exporter emits trace_id; the Grafana internal link must be configured to look for trace_id. The mismatch is silent.
• Use the same data source UID across provisioning. The Prometheus internal link dataSourceUid must match the trace backend data source UID. The two are configured in different YAML files; the team must keep them aligned by convention.
• Provision the data sources, do not click them. The internal link is a configuration setting; the team should not configure it through the Grafana UI. The UI configuration is invisible to the next maintainer.
• Monitor the click rate. The Grafana audit log records the trace backend query. The team should monitor the rate of exemplar clicks on each metric. A high click rate on a metric is a signal that the metric is the centre of investigation.
• Test the click in staging. The exporter convention is trace_id; the Grafana default is traceID. The mismatch is the default. The team should test the click in staging before the production rollout.

Verification

You should now be able to answer:

• What is the difference between the panel options and the data source configuration for exemplars?
• Why is the Grafana default traceID label name a common production failure?
• How does the click handler resolve the trace ID from the exemplar?
• What are the three fields the internal link configuration expects?
• How do you validate the click end-to-end without a browser?

Quiz