medium riskservice affecting~30 min

Runbook: Investigate a Correlation Failure

1 · Prerequisites

Confirm every item is in place before any state change.

Correlation Anatomy
Grafana Correlations
Correlation Test
Log / Trace Link
Tempo Derived Fields
Exemplars Overview
Exemplar Troubleshooting
Provisioning Datasources
Grafana admin credential, because /api/datasources and /api/admin/provisioning both require it

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Name the pivot exactly: which panel, which data source, and which direction — metric to trace, log to trace, trace to logs, or dashboard to logs. "Correlation is broken" is four different faults with four different owners.
· Establish scope by trying the same pivot on a second service. One service broken means a producer changed; every service broken means the Grafana wiring or a backend changed.
· Establish when it last worked: git log -p --since=14.days -- /etc/grafana/provisioning/datasources/ and journalctl -u grafana-server --since=-14days | grep -i ProvisioningDataSource.
· Confirm each backend is healthy on its own before blaming the join: curl -fsS -u "admin:$GF_ADMIN" http://127.0.0.1:3000/api/datasources/uid/$UID/health for the Prometheus, Loki and Tempo UIDs. A red backend is a different runbook.
· Capture one concrete trace ID from the last 15 minutes that you have independently confirmed exists in Tempo. Every step below is an assertion about that one ID; without it you are testing a hypothesis against a moving target.
· Confirm the provisioning directory is under version control and that you can read the live config as well as the file — the two drift, and the difference is diagnostic.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Reproduce the pivot by hand in Explore and record the exact failure: no link rendered, link rendered but opens an empty result, or link rendered but opens an error. The three mean different things and only the first is a wiring fault at the source data source.
2Assert the join key on each signal independently for the captured trace ID: curl -fsS "http://tempo.example.com:3200/api/traces/$TRACE_ID" | jq ".batches | length", a LogQL query filtering trace_id="$TRACE_ID", and /api/v1/query_exemplars on the histogram metric. This single step halves the search space.
3If any signal does not carry the ID, stop: the fault is producer-side and the rest of this runbook does not apply. Hand it to the service owner with the specific signal named, and record which one.
4If Prometheus is the signal missing the ID, check the server before the application: curl -fsS http://prometheus.example.com:9090/api/v1/status/runtimeinfo | jq ".data.featureFlags" must show exemplar-storage enabled. A Prometheus restarted without --enable-feature=exemplar-storage drops every exemplar silently.
5If all three signals carry the ID, the fault is in Grafana. Read the LIVE data source config for the source side of the pivot: curl -fsS -u "admin:$GF_ADMIN" http://127.0.0.1:3000/api/datasources/uid/$UID | jq .jsonData.
6Check the direction-specific field: derivedFields[].matcherRegex on Loki, exemplarTraceIdDestinations[].datasourceUid on Prometheus, tracesToLogsV2.datasourceUid and .tags on Tempo. Then confirm the destination UID actually exists with GET /api/datasources/uid/ on it.
7Diff the live config against the file on disk. A match means the file is wrong. A mismatch means the file is right and the last reload did not take, or somebody edited the data source in the UI.
8Fix the smallest thing in the provisioning file, in a commit, not in the Grafana UI. A UI edit on a provisioned data source is reverted at the next poll and will make you re-debug the same fault tomorrow.
9Reload only the data source provisioning: curl -fsS -X POST -u "admin:$GF_ADMIN" http://127.0.0.1:3000/api/admin/provisioning/datasources/reload. Read the journal afterwards for the per-resource ProvisioningDataSource lines.
10Re-run the pivot with a FRESH trace ID generated after the reload, not the one captured in the pre-checks. The old ID proves the query works; only a new one proves the pipeline works.
11Record which of the six links in the chain was broken — producer, exporter, Prometheus flag, trace backend, data source, panel. That sentence is the only part of the incident note the next person can act on.

4 · Verification

Confirm the procedure actually fixed the problem.

✓A trace ID generated after the fix resolves in all three backends: non-zero batches from Tempo, at least one Loki line, and an exemplar carrying the same ID from /api/v1/query_exemplars.
✓Clicking the pivot in Explore opens the destination with a non-empty result, not merely with the correct URL. A link that opens an empty query is still a broken pivot.
✓GET /api/datasources/uid/$UID shows the corrected field, and the value matches the provisioning file byte for byte.
✓The scheduled correlation test (the four-assertion probe from the correlation-test lesson) exits 0 on its next run without being re-run by hand.
✓The pivot on a second, unrelated service still works. A reload reconciles every data source in the directory, so the regression risk is not limited to the one you edited.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Every step up to the file edit is read-only. There is nothing to roll back during diagnosis, which is why diagnosis comes first.
↶Revert the provisioning commit and reload again: git revert the change, then POST to /api/admin/provisioning/datasources/reload, then re-check the health endpoint of every data source in the directory, not just the one you touched.
↶If you edited a data source in the Grafana UI to test a hypothesis, that edit is temporary by design — the next provisioning poll reverts it. Treat the result as evidence, never as the fix, and put the change in the file.
↶If --enable-feature=exemplar-storage was added to the Prometheus unit, the rollback is to remove the flag and restart. Note that the restart itself drops in-memory exemplars, so verify against data scraped after the restart.
↶Anything changed by hand under pressure is temporary until it exists in the repository. A correlation fixed only on the running instance breaks again at the next rebuild, and nobody will remember why.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the service owner when a producer has stopped stamping trace_id on logs or exemplars. That is an application change, not a platform change, and no amount of Grafana configuration recovers it.
· Escalate to the platform team when a provisioning reload returns 200 but the live config does not change — that is a Grafana-side fault (file permissions, wrong directory, a parse error logged and skipped) rather than a content fault.
· Escalate to security immediately if a derived field is extracting something that is not an opaque ID — a session token, an email address, a customer identifier — and turning it into a cross-system link. That is a data-handling defect and it is live for every viewer of that panel.
· Escalate to the trace-pipeline owner when Tempo holds the trace but the sampling rate is low enough that most exemplars point at traces that were never stored. The pivot is then working correctly and the answer is a sampling decision.
· Escalate when the correlation has been broken for longer than one retention window on any signal. The historical join for that window is not recoverable, and any incident review covering it needs to know.

“The correlation is broken” is a report about a click, and a click crosses two independent systems. The data layer has to put the same trace_id on the metric, the log line and the span. The UI layer has to be told, per data source, how to turn that value into a link. Both can fail, they fail for different reasons, they have different owners, and the symptom at the panel is identical.

This runbook exists to make that split in the first five minutes instead of the last five. Everything before the split is read-only.

When to use this runbook

A “Logs for this span” or “Open trace” link that used to be there is no longer rendered.
A link is rendered, opens the destination, and the destination returns nothing.
Exemplar diamonds have disappeared from a latency histogram.
The scheduled correlation test started failing.

When not to use it

A whole data source is unreachable. Panels are empty rather than unlinked, and the health endpoint is red. Correlation is a downstream symptom; fix the data source first.
Nobody can name a pivot. “Observability feels worse” is not a correlation failure, and this runbook will not find anything.
The trace genuinely does not exist because sampling dropped it. That is a sampling decision working as configured, not a broken join.

The model: correlation is not pivoting

These two words get used interchangeably at 03:00 and they name different layers with different owners.

Correlation is the data-side property: the metric exemplar, the log line and the span all carry the same trace_id. It is created by the OpenTelemetry SDK, the log handler and the exemplars API. It is invisible to the user, and it lives in the application.

Pivoting is the UI-side navigation: the click in Grafana that turns that value into an open query in another data source. It is created by data links, derived fields and the trace-to-logs block. It is visible to the user, and it lives in the provisioning YAML.

   correlation  (application: SDK, log handler, exemplars)
        |
        |   produces the join key on every signal
        v
   pivoting     (Grafana: derived fields, exemplar destinations, traceToLogs)

A correlation with no pivot is invisible but recoverable: an operator who knows the trace ID can still write the query by hand. A pivot with no correlation is worse, because the button is there and it lies — it opens a query that returns nothing, and the operator concludes the backend lost the data.

Steps 1 to 4 of this runbook decide which layer you are in. Everything after step 4 assumes the answer was “pivoting”.

Blast radius

Steps 1 to 7 read state and change nothing. Step 8 edits a provisioning file, and step 9 reloads it — and that reload is the only part of this runbook with real cost. The data source loader reconciles the whole directory by UID, so a reload applies every uncommitted edit sitting in those files, not just yours. Read git status on the provisioning directory before you POST to the reload endpoint.

Step 1: Name the pivot, then name the failure

Three failures look the same from across the room and are not the same fault:

What the operator sees	Where the fault is
No link rendered at all	Source data source: the matcher never fired, or the field is absent
Link rendered, destination empty	The join key does not exist on the destination signal
Link rendered, destination errors	Destination UID wrong, or destination backend unhealthy

Establish which one you have before running anything. If the link is absent, the value was never extracted and nothing about the destination matters yet.

Step 2: Make one trace ID the subject of the whole investigation

Read-only / Safepick a subject and confirm it exists

# Substitute your own values before running:
TRACE_ID=0af7651916cd43dd8448eb211c80319c
TEMPO=http://tempo.example.com:3200

curl -fsS "$TEMPO/api/traces/$TRACE_ID" | jq '.batches | length'

A non-zero count means Tempo holds the trace and the ID is real. Every later assertion is about this value. Investigating “correlation” in the abstract is how an hour disappears; investigating one 32-character string is a bounded problem.

Pick an ID from the last fifteen minutes. Older IDs start to collide with retention differences between the three backends, and a missing log line then tells you about retention rather than about correlation.

Step 3: Assert the join key on all three signals

This is the step that halves the search space, and it is cheap.

Read-only / Safedoes each signal carry the id

# Substitute your own values before running:
TRACE_ID=0af7651916cd43dd8448eb211c80319c
PROM=http://prometheus.example.com:9090
LOKI=http://loki.example.com:3100

# Logs: does any line carry it?
logcli query --addr="$LOKI" --since=15m \
'{service="checkout"} | json | trace_id="'"$TRACE_ID"'"'

# Metrics: does any exemplar carry it?
curl -fsSG "$PROM/api/v1/query_exemplars" \
--data-urlencode 'query=http_request_duration_seconds_bucket' \
--data-urlencode 'start=2026-08-18T10:00:00Z' \
--data-urlencode 'end=2026-08-18T10:15:00Z' \
| jq '[.data[].exemplars[].labels.trace_id] | length'

Result	Fault is	Owner
All three carry the ID	Grafana wiring — continue to step 5	Platform
Logs missing the ID	The log handler is not reading the active span context	Service owner
Exemplars missing the ID	Producer has no active span at the observation, or Prometheus lost the flag	Service owner, then platform
Tempo missing the trace	Sampling, or trace ingest	Trace-pipeline owner

Check the Prometheus flag before you accuse the application

An exemplar that is present on the scrape response and absent from the API is not an instrumentation bug. Prometheus only stores exemplars when it was started with --enable-feature=exemplar-storage, and a server restarted from a unit file that lost the flag drops every exemplar without logging anything a dashboard would show.

curl -fsS http://prometheus.example.com:9090/api/v1/status/runtimeinfo \
  | jq '.data.featureFlags'

This is one HTTP request and it removes a whole class of wrong conclusions. Run it before you open a ticket against a service team.

Step 4: If a signal lost the ID, stop here

The remaining steps configure Grafana. Grafana cannot invent a join key that the telemetry does not carry. Hand it over with the specific signal named — “checkout logs no longer carry trace_id as of the 14:20 deploy” is actionable; “correlation is broken” is not.

Record the finding anyway. Half of all correlation incidents end at this step, and the ones that end here are the ones that recur.

Step 5: Read the live config, not the file

Read-only / Safewhat grafana is actually running

# Substitute your own values before running:
UID=loki-prod
GF=http://127.0.0.1:3000

curl -fsS -u "admin:$GF_ADMIN" "$GF/api/datasources/uid/$UID" | jq '.jsonData'

The field to read depends on the direction of the pivot:

Pivot	Data source	Field
Log to trace	Loki	`derivedFields[].matcherRegex`, `.datasourceUid`
Metric to trace	Prometheus	`exemplarTraceIdDestinations[].datasourceUid`
Trace to logs	Tempo	`tracesToLogsV2.datasourceUid`, `.tags`

Then confirm the destination exists. A datasourceUid that points at a Tempo instance somebody removed last quarter provisions cleanly, renders the link, and fails only at click time — which is to say, only during an incident.

Step 6: Diff the live config against the file

# The live view, normalised, next to the file that is supposed to produce it.
curl -fsS -u "admin:$GF_ADMIN" \
  http://127.0.0.1:3000/api/datasources/uid/loki-prod \
  | jq -S '.jsonData'

sudo cat /etc/grafana/provisioning/datasources/logs.yaml

The difference is diagnostic, and it is the step most people skip:

Live matches the file, and both are wrong. Ordinary content bug. Fix the file.
Live differs from the file. Either the last reload never ran, or somebody edited the data source in the UI. The UI edit is the interesting case: it works until the next provisioning poll reverts it, which is why “it broke again on its own” appears in so many of these incident notes.

The single most common content bug here is a regex that stopped matching because the log format changed. A matcherRegex written for "trace_id":"..." does not fire against a payload that now emits "traceId", and nothing anywhere reports it: the log line still renders, just without the link.

Step 7: Change the file, in a commit

Step 8: Reload, and know what the reload costs

Configuration changereload only the data source provisioning

git -C /etc/grafana status --short
curl -fsS -X POST -u "admin:$GF_ADMIN" \
http://127.0.0.1:3000/api/admin/provisioning/datasources/reload

journalctl -u grafana-server -n 50 --no-pager | grep -i ProvisioningDataSource

The git status line is not decoration. The reload reconciles every file in the directory, so a half-finished edit that a colleague left on the box goes live at the same moment your fix does. The journal lines afterwards name each resource that was inserted, updated or deleted — read them, because a deleted line for a data source you did not touch is the shape of an outage that started with a correlation ticket.

Step 9: Verify with a trace ID that did not exist yet

Re-testing with the ID from step 2 proves the query works. It does not prove the pipeline works, because that trace was produced before the change. Fire a new request, wait for the slowest of the three ingest paths, and assert again — the correlation-test lesson’s four-assertion probe is exactly this, and running the existing script beats reinventing it under pressure.

Common patterns

Symptom	Likely cause	First check
Link vanished, logs unchanged	`matcherRegex` no longer matches the log format	Live `derivedFields` against a real log line
Link opens an empty Loki query	Trace ID present in Tempo, absent in logs	Step 3, logs assertion
Exemplar diamonds gone after a Prometheus restart	Unit file lost `--enable-feature=exemplar-storage`	`/api/v1/status/runtimeinfo`
Link 404s on click	`datasourceUid` points at a removed data source	`GET /api/datasources/uid/` on the destination
Worked in staging, not in production	Different UIDs per environment, same dashboards	Compare both live configs
Fixed itself, then broke again hours later	The fix was a UI edit; the loader reverted it	`git log` on the provisioning directory
Trace-to-logs returns unrelated lines	`tracesToLogsV2.tags` carries a label the logs do not have	Live Tempo `jsonData`

Rollback

Diagnosis changes nothing. Only two actions in this runbook do, and both have a clean reverse:

Action	Reverse
Provisioning file edited	`git revert`, reload, re-check every data source health endpoint
Prometheus flag added	Remove the flag, restart, verify against data scraped afterwards
Data source edited in the UI	Nothing to do — the next poll reverts it. Put the change in the file.

Escalation

Escalate when the fault is not yours to fix: a producer that stopped stamping the ID belongs to the service owner, a reload that returns 200 without changing the live config belongs to the platform team, and a derived field extracting anything that is not an opaque identifier belongs to security today rather than at the next review.