LogQL Pivot to Trace

An on-call engineer is staring at a Loki panel. The line at 14:22:11 reads level=error msg="checkout failed" trace_id=4bf9.... The line is the only one for the failed checkout. The engineer highlights the trace_id= value, copies it, opens Tempo in a new tab, pastes the value into the search box, and waits. The trace appears; the engineer reads it; the engineer closes the Tempo tab and goes back to Loki.

The same investigation should take one click. The line in Loki should be clickable. The click should take the engineer straight to the trace in Tempo. The mechanism is the Loki data source’s derived fields: a regex matches a value inside the log line, the value is exposed as a clickable link, the link’s datasourceUid points at the Tempo data source, and Grafana does the rest. The shape of the rule, the format of the link, and the discipline of testing it are the lesson.

What it is

A LogQL pivot to trace is the Grafana-side configuration that turns a value in a Loki log line into a link to a Tempo trace. Two configurations produce the same effect from different starting points:

• Loki data source derivedFields — a regex that matches the trace_id value inside each Loki line and produces a link whose datasourceUid references the Tempo data source. This is the path from log to trace.
• Tempo data source tracesToLogsV1 — a link rendered on each Tempo span that opens Loki filtered by trace_id. This is the inverse path and is covered in lesson 04.

This lesson is the former: the Loki side. The Loki side’s derived-fields rule is what turns the 4bf92f... substring in the line into a click. The Tempo side’s tracesToLogsV1 block is the inverse and is conceptually symmetric; a correctly configured Grafana instance has both directions wired so an engineer can move freely between the two views.

The rule is a regex that captures a value from the log line and a template that substitutes that value into the link. The link’s internalLink.datasourceUid names the destination data source; Grafana handles the click by routing to that data source with the captured value passed as a query.

Why a sysadmin cares

A panel with a working pivot is the difference between a two-minute “open the trace and read it” investigation and a twenty-minute “copy the trace ID, paste it, hope Tempo recognises it” hunt. Three operational payoffs depend on the pivot being correct:

• Reduced MTTR. Every investigation that ends in a trace benefits from the pivot. A Grafana environment with 50 dashboards and 200 panels that all wire the pivot right is a 30% faster mean time to resolution compared to one that uses copy-paste.
• Reduced error. The on-call engineer at 03:00 should not be copy-pasting 32-character hex strings by hand. The pivot removes the transcription-error class of bug — the “Tempo returns no trace because the user pasted 31 characters” ticket.
• Self-service. A second engineer with read-only Grafana access can investigate without knowing the Tempo URL or the trace ID format. The click is the interface.

The cost is configuration discipline. The Loki data source’s derived-fields block is provisioned once, but the regex, the destination UID, and the link template must all match the state of the platform on the day the engineer clicks. A datasourceUid typo, a removed data source, a renamed field, or a Loki line format change all break the click silently.

How it works

The click path is entirely client-side; the Grafana server is not in the loop:

Loki panel renders log lines
   |
   |  Each line has trace_id="4bf92f..."
   v
derivedFields regex matches "trace_id":"([0-9a-f]{32})"
   |
   |  Captured value: 4bf92f3577b34da6a3ce929d0e0e4736
   v
Grafana renders the link
   url: "$${__value.raw}"
   datasourceUid: tempo-prod-eu
   |
   |  engineer clicks the link
   v
Grafana opens the Tempo data source with the value
   TraceQL implicit: { trace_id = "4bf92f..." }
   |
   v
Tempo trace detail panel renders the trace

Three observations:

1. The regex runs in the browser, not on the server. Grafana evaluates the derived-fields regex against the rendered text of each line. The match is purely textual; Loki has already returned the lines. A regex that fails to match means the line renders with no link, not an error.
2. The link is a client-side navigation. Grafana does not call Tempo’s HTTP API on the engineer’s behalf to resolve the trace. It opens the Tempo data source’s query panel pre-filled with the value. The Tempo data source itself issues the /api/traces/<id> call when the panel mounts.
3. The datasourceUid is the destination. A typo in the UID is the single most common cause of “the link is grey or returns nothing”. UIDs are case-sensitive and must match a data source in the same Grafana instance.

How to configure it

The Loki data source — provisioned via the /etc/grafana/provisioning/datasources/ directory:

# /etc/grafana/provisioning/datasources/loki.yml
apiVersion: 1

datasources:
  - name: loki-prod-eu
    uid: loki-prod-eu
    type: loki
    access: proxy
    orgId: 1
    url: https://loki-prod-eu.internal:3100
    isDefault: false
    editable: false

    jsonData:
      maxLines: 1000
      maxSeries: 100

      # Derived fields turn a log line into a pivot.
      # Each rule extracts a value via regex and produces a
      # link using a template that may reference ${__value.raw}.
      derivedFields:
        - name: traceID
          # Match the JSON-shaped trace_id field on the line.
          # The capture group is what ${__value.raw} resolves to.
          matcherRegex: '"trace_id":"([0-9a-f]{32})"'
          url: '$${__value.raw}'
          urlDisplayLabel: 'Open in Tempo'
          datasourceUid: tempo-prod-eu
          internalLink:
            expr: '${__value.raw}'
            datasourceUid: tempo-prod-eu

        # A second common rule: link the level field back to
        # Loki filtered by the same level.
        - name: level
          matcherRegex: '"level":"(debug|info|warn|error|fatal)"'
          url: '$${__value.raw}'
          urlDisplayLabel: 'Filter by level'
          datasourceUid: loki-prod-eu
          internalLink:
            query: '{service_name=~"$${__var.service_name:json}"}->level="${__value.raw}"'
            datasourceUid: loki-prod-eu

    secureJsonData:
      basicAuthPassword: ${LOKI_PASSWORD}

A few production notes on the options:

• type: loki. Anything else (including prometheus with a Loki URL) produces a data source that Grafana treats as Prometheus and breaks on the first derived-fields query.
• matcherRegex is the regex the browser evaluates against the rendered line. A regex with a literal " must escape the quote correctly. Double-escaping (the YAML parser and the JS regex both interpret the string) is a recurring cause of “the field is empty”. Validate by clicking the link in a panel.
• internalLink.datasourceUid is the UID of the destination data source. A typo produces a link that fails at click time, not at provisioning time. Reference the UID from the Tempo data source’s provisioning file to keep them in sync.
• ${__value.raw} is the captured substring (group 0). ${__var.<label_name>} is a stream label from the current line, used for cross-link queries (e.g. “filter Loki by the same level and service”).

The companion — the Tempo data source provisioning file should already exist (lesson 04 covers the inverse pivot in detail). The UID is what links the two configs.

How to validate it

# READ-ONLY: confirm the Loki data source is provisioned.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/uid/loki-prod-eu \
  | jq '.uid, .type, (.jsonData.derivedFields | length)'
# "loki-prod-eu"
# "loki"
# 2   (the traceID rule plus the level rule)

# READ-ONLY: confirm the Tempo data source UID is reachable.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/uid/tempo-prod-eu
# { "id": 3, "uid": "tempo-prod-eu", "type": "tempo", ... }
# (If this 404s, the derivedFields rule points at a
# non-existent UID; the link will be broken at click time.)

# READ-ONLY: confirm a Loki line matches the regex.
TRACE=4bf92f3577b34da6a3ce929d0e0e4736
logcli query --since 1h '{service_name="checkout-api"}' \
  | grep -c "$TRACE"
# 14
# (If 0, the line format has drifted from the regex, or the
# structured field is named something other than trace_id.)

# READ-ONLY: confirm the captured value resolves to a trace.
curl -fsS -u "$TEMPO_USER:$TEMPO_PASS" \
  "http://tempo.internal:3200/api/traces/$TRACE" \
  | jq '.batches | length'
# 4

# CONFIGURATION: make the click test programmatic. Open the
# derived-field URL directly and confirm Tempo returns 200.
URL='https://grafana.internal:3000/explore?left={"datasource":"tempo-prod-eu","queries":[{"refId":"tempo-trace","queryType":"traceql","query":""}]}'
echo "$URL"
# (Manual: open in browser, paste the trace_id into the
# search bar, confirm Tempo returns the trace. There is no
# good way to script the click-side of the pivot; the
# server-side rule load is the part the API can verify.)

If step 1 reports zero derived fields, the provisioning file has not been loaded (Grafana does not hot-reload data sources on file changes — restart or use the API). If step 2 returns 404, the Tempo data source UID was renamed or removed and the Loki side’s reference needs updating. If step 4 returns 0 matches, the field is now traceID instead of trace_id (camelCase), or the JSON spelling changed, or the structured metadata has been demoted back to the message text.

How it can fail

1. Destination UID typo. The derivedFields.datasourceUid references tempo-prod-eu but the Tempo data source is actually tempo or tempo-eu. Symptom: the link renders in the panel; clicking opens a new panel that says “Data source not found”.
2. Tempo data source was deleted. The data source was removed (admin’s UI cleanup, a provisioning mistake). Symptom: the link renders but the click 500s with “data source uid was not found” or silently opens an empty panel.
3. Regex does not match the line format. The application’s log library switched from JSON to logfmt, or the field is now traceID instead of trace_id, or the structured metadata path now stores the field as trace_id=4bf92f... instead of "trace_id":"4bf92f...". Symptom: the panel renders the line with no link; an empty field.
4. YAML quote escaping. The regex contains a literal quote that the YAML parser interprets differently from the JS regex parser. Symptom: the rule shows up in the data source JSON but the regex is malformed; the field never matches.
5. Authentication mismatch. The engineer has viewer role on Loki but no access to Tempo. Symptom: the panel renders, the click goes to a Tempo URL, but the response says “Data source is not authorised for the current organisation”.
6. Tempo has dropped the trace. The trace has aged out of Tempo’s trace block storage. Symptom: the link renders and the click opens an empty Tempo panel with the message “trace not found”. The Loki side is correct; the Tempo side is missing the data.

How to troubleshoot it

The diagnostic order for “the trace_id link does not work in the panel”:

1. Does the regex match the line? Copy a Loki line that is supposed to be linked; run the regex against it in grep -E. A failing match is the most common cause.
2. Does the UID resolve? /api/datasources/uid/<uid> in the Grafana API. A 404 means the UID is wrong; fix the derivedFields.datasourceUid value.
3. Does the destination data source work on its own? Open the Tempo data source directly via the explore view and run a query by trace_id. A broken Tempo data source is the cause if the standalone query also fails.
4. Is the data source provisioned? /api/datasources in the Grafana API. A missing entry means the provisioning file failed to load; check Grafana’s log for the parse error.
5. Does the engineer have permission? Grafana 11.x has per-data-source ACLs; a viewer on Loki may not have access to Tempo. Confirm the role assignment.

Security implications

The pivot is a client-side link. It does not, in itself, leak data — the engineer already has read access to the line and its trace_id. The risks are:

• Cross-data-source ACL bypass. A viewer role on Loki does not, by default, grant access to Tempo. Grafana 11.x enforces per-data-source ACLs; the pivot is correct only if both ACLs align. Review role assignments alongside the data source UID change.
• Regex DoS. A catastrophic backtracking regex on matcherRegex against every line of every panel can hang the browser. Avoid unbounded quantifiers; validate regexes with grep on representative lines before deploying.
• URL injection. matcherRegex that captures a substring used directly in url (not ${__value.raw} but ${__value.sub}) can produce URLs that resolve outside Tempo. The ${__value.raw} form is the safe choice; the team should be able to audit every URL it produces.

The datasourceUid itself is a Grafana-internal identifier; it is not exposed to the browser and is not user-injectable.

Performance implications

The derived-fields regex runs in the browser against every rendered line. The cost is small for a single panel; it grows linearly with the panel’s maxLines and with the number of panels on the dashboard. A panel with 1000 lines and five derived-fields rules is 5000 regex evaluations on every refresh. On a low-end laptop on a Grafana Cloud free tier, this is roughly 30 ms per refresh; on a 4K monitor with 20 panels open, the lag is visible.

Tune maxLines on the Loki data source down (500 is the pragmatic minimum for a useful pivot experience, 100 for log-summary dashboards). Reduce the number of derived-fields rules to the ones the team actually clicks. A regex that always matches is cheap; a regex with capture groups is the cost.

The Tempo side has its own performance profile (search and fetch). Cover it in lesson 05.

Production guidance

• Use a regex that captures only what is needed (32 lowercase hex chars for trace_id). Avoid greedy quantifiers.
• Provision every environment’s data source from a templating tool that shares the UID assignment. A typo on a UID is the most common breakage.
• Test the click as part of the change pipeline. The test in lesson 06 fails the deployment if the derived-field rule does not resolve a real trace.
• Roll the change back by reverting the provisioning file. Grafana reloads data sources on file change if the --webhook-provisioner is enabled, or on restart. The previous rule set is restored from the previous file revision.

Verification

You should now be able to answer:

• Which data source carries the derivedFields configuration for the Loki-to-Tempo pivot?
• What does the ${__value.raw} template variable resolve to?
• What is the observable symptom of a typo on the destination datasourceUid?
• What happens if the application switches the log line format from JSON to logfmt?
• What is the first diagnostic step when the link renders but the click returns nothing?

Quiz