Exemplar Link to Trace

The diamond is on the bar. The operator clicks it. Tempo opens the trace. Between the click and the trace is a small piece of Grafana configuration called the Internal link: a block inside the Prometheus data source that names the trace backend, names the label, and tells Grafana what URL to build. This lesson is about that block — its four fields, what each one does on the wire, and the four ways it can silently fail.

The Internal link is the place where dashboards meet traces. It is small. It is easy to misconfigure. It is one of the last things a team wires because it is invisible until an alert fires. When the alert fires at 03:00 and the click does not work, the lesson is either already known or about to be.

What it is

The Internal link is a YAML block inside the Prometheus data source’s jsonData provisioning. It declares how the diamond on a histogram panel becomes a navigation event to the trace data source. The block has four fields:

internalLink:
  tracing:
    dataSourceUid: tempo           # which data source to open
    label: traceID                 # which Prometheus exemplar label holds the trace ID
    spanId: spanID                 # optional: which label holds the span ID
    urlDisplayLabel: Trace         # optional: the link text shown in tooltips

Of the four fields:

• dataSourceUid is required. It must match the UID of a Tempo (or other tracing) data source configured in the same Grafana instance.
• label is required. Grafana uses this to read the trace ID from the exemplar. Default is traceID; many OTel SDKs emit trace_id. The drift between the two is the most common silent failure.
• spanId is optional. When present, the URL Tempo receives includes both traceID and spanID, and Tempo scrolls to the matching span when the trace loads. When absent, Tempo opens the trace at the root.
• urlDisplayLabel is cosmetic; it controls the tooltip text on the diamond. Default is “Trace”.

The block sits inside jsonData rather than at the top level because the Internal link is a property of the Prometheus data source, not a panel property. Panel options control whether the diamond is rendered; the Internal link controls where the click goes.

Why a sysadmin cares

A correctly configured Internal link is invisible; the operator never thinks about it. A misconfigured Internal link is the single most common reason the pivot fails at runtime. Three failure shapes appear repeatedly:

1. The diamond renders but the click opens a blank Explore page. Most often a UID mismatch. The Prometheus data source has dataSourceUid: tempo, but the Tempo data source has uid: Tempo. UIDs are case-sensitive.
2. The diamond renders but the click opens Tempo with an empty query. Almost always a label-name drift. The producer emits trace_id; Grafana reads traceID. Default behaviour assumes OTel SDK conventions, but prometheus/client_golang exposes both names depending on version.
3. The link works but the trace returns no results. The Internal link is fine; Tempo does not have the spans. This looks like an Internal link failure but is a trace pipeline failure.

The Internal link is the seam where the dashboard story meets the tracing story. Both teams own part of it. When the seam breaks, neither team can tell where the seam is.

How it works

The click on a diamond goes through five steps on the wire:

  Operator clicks diamond on histogram bar
        |
        v
  Grafana reads the exemplar labels from the bar at the click
        |
        v
  Grafana builds /explore?ds=<dataSourceUid>&traceID=<label-value>
  Span ID appended if spanId label is configured
        |
        v
  Grafana routes the URL to the matching data source
  Tempo is reached via its configured URL
        |
        v
  Tempo resolves the trace and returns a trace detail panel

Step 1 happens in the browser; steps 2 to 4 are Grafana JavaScript; step 5 is the trace backend’s job. Failure modes 1 and 2 above are steps 2 to 4; failure mode 3 is step 5.

The Internal link is loaded when the data source is created. A change to the block is read on the next Grafana data source reload, which the data source UI does not always force; a provisioning change requires either a Grafana reload or a restart depending on the field.

How to configure it

Two configuration steps: the Tempo data source and the Internal link.

1. Provision the Tempo data source first

The Internal link references this data source by UID. The UID must exist before the Internal link can be validated. The UID in this configuration is tempo. Lock it in provisioning, do not let Grafana generate it on first save.

# Grafana provisioning file (CONFIGURATION)
# Place under /etc/grafana/provisioning/datasources/
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo:3200
    access: proxy
    jsonData:
      httpMethod: GET
      tracesToLogsV1:
        datasourceUid: loki
        tags: ['job', 'instance']
        mappedTags: [{ key: 'service.name', value: 'service' }]
        # The inverse link; this lesson focuses on the metric-to-trace side

uid: tempo is the value the Internal link will reference. Note that name, uid, and the data source type are three separate fields; only the UID is referenced by other data sources.

2. Configure the Internal link on the Prometheus data source

# Grafana provisioning file (CONFIGURATION)
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prom
    url: http://prometheus:9090
    jsonData:
      httpMethod: POST
      internalLink:
        tracing:
          dataSourceUid: tempo           # matches the Tempo UID above
          label: traceID                 # Grafana reads this label from the exemplar
          spanId: spanID                 # optional, scroll-into-span behaviour
          urlDisplayLabel: Trace         # tooltip text on the diamond

A complete Grafana provisioning directory looks like:

/etc/grafana/provisioning/datasources/
  ├── datasources.yaml          # Prometheus and Loki, one file
  └── tempo.yaml                # Tempo, separate to avoid merge-conflict churn

Each file is loaded once at startup and re-read on Grafana hot-reload, which is enabled by default in 11.x.

3. Verify the data source wiring via the API

The Grafana HTTP API reports the merged configuration, which is the most reliable way to confirm the Internal link is loaded.

# READ-ONLY
curl -sf -u admin:admin http://grafana:3000/api/datasources/uid/prom \
  | jq '.jsonData.internalLink'

Expected:

{
  "tracing": {
    "dataSourceUid": "tempo",
    "label": "traceID",
    "spanId": "spanID",
    "urlDisplayLabel": "Trace"
  }
}

Severity: CONFIGURATION when writing the file, SERVICE-IMPACT when the file is reloaded on a running Grafana instance. The hot-reload is fast; the data source update is asynchronous.

How to validate it

Three validation layers.

1. The Internal link block is present and well-formed

# READ-ONLY
curl -sf -u admin:admin http://grafana:3000/api/datasources/uid/prom \
  | jq '.jsonData.internalLink.tracing | keys'

Expected: [ "dataSourceUid", "label", "spanId", "urlDisplayLabel" ] (or a subset). If the array is empty, the block was never loaded.

2. The UID resolves to an actual data source

# READ-ONLY
UID=$(curl -sf -u admin:admin http://grafana:3000/api/datasources/uid/prom \
  | jq -r '.jsonData.internalLink.tracing.dataSourceUid')
curl -sf -u admin:admin "http://grafana:3000/api/datasources/uid/$UID" \
  | jq '.type'

Expected: "tempo". If the response is 404, the UID is pointing at something Grafana does not know about.

3. The click produces a trace

The end-to-end check still needs a real exemplar and a real trace in Tempo. Use the same query_exemplars and /api/traces/<id> calls from lesson 01.

# READ-ONLY — end-to-end
TRACE_ID=$(curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="1.0"}' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq -r '.data[0].exemplarLabels.trace_id')

curl -sf "http://tempo:3200/api/traces/$TRACE_ID" | jq '.traces | length'

Expected: at least 1. Zero means the trace is absent from Tempo even though the labels were forwarded — the Internal link is correct, the OTLP pipeline is the failure.

How it can fail

Four failure modes, ordered by frequency. Each has a single visible symptom.

1. UID mismatch. The Prometheus data source has dataSourceUid: tempo and the Tempo data source has uid: Tempo. UIDs are case-sensitive. Symptom: the diamond renders; the click opens an Explore page with no data source selected.
2. Label name drift. The producer emits trace_id; the Internal link has label: traceID. Grafana reads the exemplar label that does not exist and sends an empty trace ID. Symptom: the diamond renders; the click opens Tempo with traceID= empty.
3. Span ID label missing. spanId is configured to a label that the exemplar does not carry. Symptom: the link has the trace ID but spanID= is empty. Tempo opens the trace at the root span instead of scrolling to the matching one.
4. Provisioning file has not been reloaded. The Internal link block exists in YAML on disk but Grafana has not loaded the new file. Symptom: the API reports the Internal link as empty; the diamond renders but clicking it does nothing, or opens an Explore page that does not resolve.

How to troubleshoot it

Steps in order. Each step rules out one of the four failure modes.

1. Inspect the Internal link via the API. The first command from the validation section. If the block is empty, the provisioning file was not reloaded. Force a Grafana hot-reload and re-check.
2. Resolve the UID against the data sources. The second command from the validation section. If the response is 404, the UID does not match. Edit the Internal link to point at the actual UID; do not change the Tempo data source UID, which would invalidate dashboards, links, and permissions.
3. Cross-check the label names against an exemplar. Take any exemplar from query_exemplars and inspect the exemplarLabels field. The Internal link’s label value must be a key in that object.
4. Click the diamond with developer tools open. The URL fragment from the Under the hood block above appears in the address bar. Empty traceID= confirms a label drift. datasource.uid pointing at the wrong UID confirms a mismatch.

Security implications

The Internal link is a routing rule, not an authentication boundary. Three things still matter:

• Same-origin requirement. The Tempo data source URL (http://tempo:3200 in the example) must be reachable from the Grafana server. If Tempo is on a private network, the Grafana server must be on the same network. A misconfigured proxy that exposes Tempo on a public IP creates an unauthenticated trace read endpoint.
• Data source permissions. A user who can read the Prometheus data source can click the diamond and read the trace. Grafana does not enforce “can read histogram but not trace” as a separate rule. If only a subset of users should see traces, the Tempo data source must be on a separate organisation.
• Trace ID leakage. The trace ID is in the URL fragment. URL fragments are not typically logged by intermediaries, but they are stored in browser history, shared via shoulder-surfing, and sometimes captured by monitoring tooling. Treat trace IDs as low-grade but not zero identifiers.

Performance implications

The Internal link itself adds no runtime cost. The Tempo data source on the receiving side is the source of cost, in three shapes:

• Single-trace fetch. /api/traces/<id> is one Tempo query per click. Cost is bounded by the trace’s span count; a 1,000-span trace returns a 200 KB JSON, a 100,000-span trace returns 20 MB. Configure Tempo’s max_spans_per_trace to bound the second.
• Concurrent clicks. A busy incident response may produce dozens of clicks per minute from many operators. Tempo scales horizontally for trace queries; the question is query concurrency, not throughput.
• Network path. Grafana-to-Tempo is an internal network hop. A cross-region hop or a proxied connection adds latency. The recommended pattern is a Grafana-server-side proxy with access: proxy rather than access: browser.

Production guidance

• Provision both data sources from files. Never create the Tempo data source from the UI for an environment that has a Prometheus data source with an Internal link. The UID is a cross-data-source reference; one of the two going through a versioning system without the other is how production pivots break.
• Place the Internal link block in the same YAML file as the Prometheus data source. Two reasons: it makes the relationship between the Internal link UID and the Tempo data source UID obvious to the next reader, and it makes git log show pivot-relevant changes as a single diff.
• Add a smoke test in CI. The three validation commands can be a script that asserts the Internal link block is present, the UID resolves, and a click produces a non-empty trace response. Run it after every Grafana upgrade.

Verification

You should now be able to answer:

• Which four fields does the Internal link block have, and which is the optional one?
• What is the visible symptom when the data source UID referenced by the Internal link does not resolve?
• What does the URL fragment look like when the label name in the Internal link does not match the exemplar labels?
• How do you confirm the Internal link block is loaded on the live Grafana instance?

Quiz