Break/Fix: Metric/Log/Trace Correlation Broken

Reported symptoms

An engineer on the checkout rota raises a ticket titled “exemplars are broken”. Two days later the ticket has grown four comments and no cause.

The original report: the latency histogram still draws diamonds on the tail buckets, exactly as it always has. Clicking one opens an Explore pane that stays empty, with an error banner naming a data source the picker does not offer.

The second comment adds a symptom from a different workflow. The trace-ID link that Loki renders on every structured log line does the same thing - clickable, and it lands nowhere. Two features built by two different people, broken identically.

The third comment is the one that stalled the investigation. Selecting Tempo in Explore and pasting the same trace ID by hand returns the trace immediately, every time. The Tempo dashboards - span rate, service graph, error ratio - are all healthy. So Tempo is fine, and the team goes looking upstream.

The fourth comment records what upstream looked like. Prometheus runs with --enable-feature=exemplar-storage. The /api/v1/query_exemplars endpoint returns exemplars with trace_id labels. The exporter’s /metrics carries the # {trace_id="..."} suffix on the bucket lines. Every component in the chain passes its own test.

And one detail nobody has connected to anything. From inside a trace, the pivot out to Loki still works. Correlation is dead in one direction and healthy in the other.

The scheduled correlation test - the synthetic probe that fires a request with a known trace ID and asserts it lands in Prometheus, Loki and Tempo - has passed every run for three weeks.

Evidence provided

Start where the failure actually happened, not where its name points. Clicking the diamond navigated the browser somewhere; the address bar records where.

/explore?schemaVersion=1&panes=%7B%22trace%22%3A%7B%22datasource%22%3A%22tempo%22...

URL-decoded, the part that matters:

{"trace":{"datasource":"tempo","queries":[{"query":"4a1f9c2e8b7d6053a1c4e9f2b8d70a16"}]}}

Grafana tried to open a data source called tempo. The next question is whether one exists. In the commands below, GRAFANA is the instance URL and TOKEN is a viewer-scoped API token; both are read-only queries against the Grafana HTTP API.

$ curl -sf -H "Authorization: Bearer $TOKEN" "$GRAFANA/api/datasources" \
| jq -r '.[] | .uid + "  " + .type + "  " + .name'

prom        prometheus  Prometheus
loki-prod   loki        Loki
tempo-prod  tempo       Tempo

$ curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $TOKEN" "$GRAFANA/api/datasources/uid/tempo"

404

The two places that name it are the two features that broke.

$ curl -sf -H "Authorization: Bearer $TOKEN" "$GRAFANA/api/datasources/uid/prom" \
| jq '.jsonData.exemplarTraceIdDestinations'

[
{
  "name": "trace_id",
  "datasourceUid": "tempo"
}
]

$ curl -sf -H "Authorization: Bearer $TOKEN" "$GRAFANA/api/datasources/uid/loki-prod" \
| jq '.jsonData.derivedFields'

[
{
  "name": "TraceID",
  "matcherRegex": "\"trace_id\":\"([a-f0-9]{32})\"",
  "url": "${__value.raw}",
  "datasourceUid": "tempo"
}
]

And the change calendar, read after the fact: Tempo was redeployed three weeks ago, and its data source provisioning file was rewritten in the same change.

Work the evidence before reading on

Every backend holds the right data. Every component passes its own test. The pivot is dead.

1. query_exemplars returns a trace ID, and the Tempo API returns the trace for that ID. Given both are true, what part of the chain has not been tested by either check?
2. The correlation probe asserts the trace ID is present in Prometheus, Loki and Tempo. Name the thing it never exercises.
3. Pivots into Tempo are dead; the pivot out of Tempo works. What is configured on the Tempo data source that is not configured on the other two, and which way does it point?
4. Provisioning applied cleanly and logged nothing. What would Grafana have had to do, at provisioning time, to notice this?

Before continuing: the diamond still renders. Say why the fault cannot be anywhere upstream of Grafana, using only that fact.

Root cause

1. A UID is a foreign key with no referential integrity

Grafana identifies a data source by its uid, and other objects reference it by that string: an exemplar destination on the Prometheus data source, a derived field on the Loki data source, a panel’s data source binding, an alert rule’s query.

Nothing enforces that the string resolves. Provisioning reads each file, validates it in isolation, and applies it. A file that names a UID no other file declares is not an error; it is just a string that will fail to resolve later, in a browser, in front of whoever clicked.

Three weeks ago the Tempo data source was redeclared with uid: tempo-prod and the old tempo went away. The two files that referenced tempo were not part of that change, were not reviewed with it, and applied cleanly afterwards.

2. Every component test passed because none of them tests this

The team’s checks were good checks. They were also all on the wrong side of the fault.

Only the last row exercises the reference, and the last row is the only thing the team had not automated. The diamond rendering is itself the proof that everything upstream is healthy: Grafana cannot draw a diamond it did not receive.

3. The correlation test asserted the data, not the navigation

The scheduled probe fires a synthetic request with a known trace ID, waits for the telemetry to land, and asserts the ID appears in Prometheus, in Loki and in Tempo. All three assertions were true throughout. Every store held exactly what it should.

What the probe never did was follow the link. The join key was present at every signal; the pointer between two Grafana objects was dangling, and no query against a backend can see a Grafana configuration error. A black-box probe shaped around the contract “the trace ID reaches every store” cannot detect a break in the contract “the operator can get from one store to another”.

4. The direction asymmetry is the tell

Pivots into Tempo come from configuration held on the other data sources: the Prometheus exemplar destination and the Loki derived field, both naming the Tempo UID. The pivot out of Tempo comes from configuration held on the Tempo data source itself, naming the Loki UID - which did not change.

So one migration broke every inbound pivot and left every outbound one intact. Described that way the cause is obvious. It was not described that way for three weeks, because the inbound pivots and the outbound pivot belong to different people’s workflows and arrived as unrelated tickets.

Resolution

1. Enumerate every reference before editing anything. Pull each data source from the API, extract every datasourceUid in its jsonData, and compare that set against the set of declared UIDs. Fixing the two you already found and missing a third is a second incident.
2. Decide which UID is canonical. tempo-prod is declared in the provisioning file and is already referenced by anything repaired during the last three weeks, so it is usually the right answer - but decide, and write the decision down, rather than letting whichever file loads last win.
3. Edit the provisioning files, not the running instance. A change made through the UI survives until the next reload and no longer.
4. Read the whole provisioning directory before reloading. A reload re-applies every file in it, so a stale file you did not touch is about to be applied along with your fix.
5. Reload provisioning and confirm the reload was clean, then re-run the reference enumeration. The set of unresolved references must be empty.
6. Click through in both directions: diamond to trace, log line to trace, and trace back out to logs. The last one was the path that still worked and is the one your edit could have broken.
7. Add the reference check to CI in the same change. This fault is one comparison away from being impossible, and the migration that caused it will happen again with a different backend.

The check that closes the class, rather than the instance:

# Substitute your own values before running:
GRAFANA=https://grafana.internal.example.com
TOKEN=glsa_replace_with_a_viewer_token

curl -sf -H "Authorization: Bearer $TOKEN" "$GRAFANA/api/datasources" \
  | jq -r '.[].uid' | sort -u > /tmp/declared-uids

curl -sf -H "Authorization: Bearer $TOKEN" "$GRAFANA/api/datasources" \
  | jq -r '.[].jsonData // {}
           | (.exemplarTraceIdDestinations // []) + (.derivedFields // [])
           | .[].datasourceUid // empty' \
  | sort -u > /tmp/referenced-uids

# Anything printed here is a reference that will fail at click time.
comm -23 /tmp/referenced-uids /tmp/declared-uids

And the corrected provisioning, with the UID treated as a declared contract rather than a generated value:

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prom
    url: http://prometheus.internal.example.com:9090
    jsonData:
      exemplarTraceIdDestinations:
        # `name` is the exemplar label carrying the trace ID.
        # `datasourceUid` must match a uid declared in this directory.
        - name: trace_id
          datasourceUid: tempo-prod

Verification

1. The click works. Open the checkout latency histogram, click a diamond, and land on the trace. This is the only end-to-end check; everything else in this list was already passing while the pivot was broken.
2. The other inbound pivot works. Open a log line in Loki and click its trace-ID link. Two features reference the same UID, and confirming one says nothing about the other.
3. The outbound pivot still works. From inside a trace, pivot to logs. This path was healthy before the fix and must remain so afterwards.
4. The reference set resolves. The comparison above prints nothing. Run it against the live instance rather than against the files, so it reflects what Grafana actually loaded.
5. It survives a reload. Reload provisioning and re-run all of the above. The characteristic failure of this repair is a reload that quietly restores the broken state.
6. The guard can fail. In staging, point one reference at a deliberately bogus UID and confirm the check reports it and the CI job goes red. A check that has only ever printed nothing has not been shown to work.

Prevention

• Declare uid on every data source and treat it as a public identifier. Other files depend on it. Letting Grafana generate one produces a value nobody chose and everybody references.
• Treat a UID change as a breaking rename. It deserves the same review as renaming a metric that dashboards query: find the references first, change them in the same commit.
• Resolve every cross-file reference in CI. The comparison above runs in seconds and makes this entire class of fault impossible to merge.
• Make the correlation probe follow the navigation. Asserting that each store holds the trace ID tests the pipelines. Asserting that the link resolves tests the thing the operator actually does at 03:00.
• Add “what references this UID” to every backend migration checklist, next to the storage and retention questions that already get asked.
• Annotate provisioning reloads onto the dashboards, so a configuration change appears on the same timeline as the symptom it caused.
• Read the URL first. When a correlation feature breaks, the click produced an address, and the address names what it tried to open. It is the cheapest evidence in the entire investigation and it was available on day one.