Log to Trace Workflow

The on-call engineer is chasing a slow checkout. The metrics panel shows the spike. The engineer pivots to logs and finds the line:

2026-08-13T03:12:44.512Z service=checkout msg="payment failed" trace_id=0af7651916cd43dd8448eb211c80319c

The trace_id is right there. The engineer wants to click on it and jump to Tempo. The pivot does not work. The trace_id is displayed as plain text. The engineer has to copy it by hand, open Tempo, paste the ID, hit Enter. The investigation takes eight seconds instead of one click.

The log-to-trace pivot closes the gap. The pivot is a Loki derived field. A derived field is a regex that extracts a value from the log line and turns it into a clickable link. The panellist clicks the link, the drill opens Tempo, the trace is in front of them. The flow is one click.

What it is

The log-to-trace pivot is a Grafana drill that opens a Tempo trace from a trace_id value in a Loki log line. The mechanism is the Loki derived field. The derived field is a regex that runs over each log line at render time. When the regex matches, the captured value is turned into a clickable URL with a user-defined label.

  Loki log line
  +--------------------------------------------------+
  | 2026-08-13T03:12:44.512Z service=checkout        |
  |   msg="payment failed"                           |
  |   trace_id=0af7651916cd43dd8448eb211c80319c      |
  +----------------------+---------------------------+
                         |
                         | derived field regex
                         v
  Grafana render
  +--------------------------------------------------+
  | 2026-08-13T03:12:44.512Z service=checkout        |
  |   msg="payment failed"                           |
  |   trace_id=0af7651916cd43dd8448eb211c80319c      |
  |                  |                               |
  |                  +-- "Open trace" link            |
  +--------------------------------------------------+
                         |
                         | click
                         v
  Tempo query
  +--------------------------------------------------+
  | trace_id = 0af7651916cd43dd8448eb211c80319c      |
  +--------------------------------------------------+

The derived field is configured on the Loki data source, not on the panel. The reason is that the derived field is a property of the log line, not of the panel. Every panel that shows the log line picks up the derived field automatically.

Why a sysadmin cares

The log-to-trace pivot is the second-most-common pivot in an on-call runbook. It is the drill that the engineer runs when the metrics panel is green but the application is misbehaving. The log line is the evidence. The trace is the explanation.

Three operational payoffs.

1. Single click from error to trace. The on-call engineer does not need to copy the trace_id by hand. The click opens Tempo with the right query.
2. The pivot survives log retention. Loki retains logs for fourteen days by default; Tempo retains traces for shorter. The pivot does not change the retention. The pivot just makes the existing data easier to reach.
3. The pivot survives log parser changes. The derived field is a regex that runs over the raw log line. The regex does not depend on a Loki parser or on a structured metadata field. The pivot works on unstructured logs too.

The cost is the discipline of constructing a regex that is loose enough to match the production log lines but tight enough to avoid false matches. The regex is the failure shape.

How it works — the derived field regex

The derived field is a regex with a single capture group. The capture group is the value that becomes the clickable link. The URL is a template that runs at render time.

The Loki data source carries the configuration:

# grafana/provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    uid: loki
    url: http://loki:3100
    jsonData:
      derivedFields:
        - name: traceID
          matcherRegex: '"trace_id":"([a-f0-9]{32})"'
          url: '$${__value.raw}'
          urlDisplayLabel: 'Open trace'

The matcherRegex is the regex that runs over each log line. The capture group is ([a-f0-9]{32}). The regex requires the 32-hex value to be wrapped in double quotes, which is the shape of the JSON-formatted log line. The url is the template that becomes the clickable link. The ${__value.raw} substitution is replaced with the captured value.

The regex variants that matchable log lines produce:

• JSON structured log. matcherRegex: '"trace_id":"([a-f0-9]{32})"' matches the compact JSON form. The capture group is the 32-hex value.
• Logfmt structured log. matcherRegex: 'trace_id=([a-f0-9]{32})' matches the logfmt form. The capture group is the value up to the next whitespace or comma.
• Plain text log. matcherRegex: 'trace[_-]id[:=]["\s]*([a-f0-9]{32})' matches the human-readable form. The capture group is the 32-hex value.

The URL points to the Tempo data source. The substitution chain is ${__value.raw} → the trace ID → the Tempo query parameter query=${traceId}.

How to configure it

The Loki data source has the derived field. The provision is idempotent and reloadable.

# grafana/provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    uid: loki
    url: http://loki:3100
    jsonData:
      derivedFields:
        - name: traceID
          # The JSON form. The opening quote on the field name
          # prevents the matcher from drifting into a
          # substring that contains a hex string.
          matcherRegex: '"trace_id":"([a-f0-9]{32})"'
          # The URL is replaced with the captured value.
          # The $$ is the YAML escape for a literal $.
          url: '$${__value.raw}'
          urlDisplayLabel: 'Open trace'

        - name: traceID
          # The logfmt form. The capture group is the value
          # up to the next whitespace or comma.
          matcherRegex: 'trace_id=([a-f0-9]{32})'
          url: '$${__value.raw}'
          urlDisplayLabel: 'Open trace'

The application side — the log handler stamps the trace_id on every log line. In Go with slog:

import (
    "go.opentelemetry.io/otel/trace"
    "log/slog"
)

type traceHandler struct{ slog.Handler }

func (h *traceHandler) Handle(ctx context.Context, r slog.Record) error {
    span := trace.SpanFromContext(ctx)
    if span.SpanContext().IsValid() {
        r.AddAttrs(slog.String("trace_id", span.SpanContext().TraceID().String()))
    }
    return h.Handler.Handle(ctx, r)
}

The handler uses slog’s JSON handler, which emits the trace_id field in the compact JSON form ("trace_id":"0af7651916..."). The Loki parser indexes the field as a structured metadata field. The derived field regex matches the raw line.

The Loki pipeline — the OTel logs receiver and the relabel pipeline that ships the log lines to Loki:

# /etc/alloy/config.alloy
loki.relabel "checkout" {
  rule {
    source_labels = ["__meta_otel_logs_resource_attributes_service_name"]
    target_label  = "service"
  }
  rule {
    source_labels = ["__meta_otel_logs_resource_attributes_host_name"]
    target_label  = "instance"
  }
}

loki.write "checkout" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

How to validate it

# 1. The log line carries the trace_id in the right shape.
logcli query --since=10m \
  '{service="checkout"} | json | trace_id!=""' --tail=1
# 2026-08-13T03:12:44.512Z {service="checkout"} msg="payment failed" trace_id=0af7651916cd43dd8448eb211c80319c

# 2. The regex matches the raw line.
echo '2026-08-13T03:12:44.512Z service=checkout msg="payment failed" trace_id=0af7651916cd43dd8448eb211c80319c' \
  | grep -oE '"trace_id":"([a-f0-9]{32})"' | head -1
# "trace_id":"0af7651916cd43dd8448eb211c80319c"

# 3. The Tempo data source accepts the trace_id.
tempo-cli query '{ trace = "0af7651916cd43dd8448eb211c80319c" }'
# Span: 0af7651916cd43dd8448eb211c80319c  service=checkout  duration=4.2s

# 4. The Grafana data source has the derived field loaded.
curl -s -u admin:admin http://grafana:3000/api/datasources/uid/loki \
  | jq '.jsonData.derivedFields'
# [{"name":"traceID","matcherRegex":"\"trace_id\":\"([a-f0-9]{32})\"",
#   "url":"${__value.raw}","urlDisplayLabel":"Open trace"}]

# 5. The drill opens Tempo with the trace ID.
# (UI step) Open the Explore page, run the Loki query, click
# the "Open trace" link in the rendered log line. The link
# opens Tempo with the trace ID in the query parameter.

How it can fail

Six recurring failure shapes.

1. The regex does not match the production log line. The regex was written against a test fixture. The production log line has spaces or extra fields. Symptom: the log line displays without a clickable link. The derived field is silently no-op.
2. The trace_id is in a structured metadata field, not in the raw line. The Loki pipeline parses the JSON and indexes the field. The raw line still has the value, but the derived field regex is run on the raw line, not on the parsed field. Symptom: the structured metadata field is present, the derived field is not. The fix is to write the regex against the raw line, not the parsed field.
3. The url uses the wrong substitution. The URL is '{__value.raw}' instead of '$${__value.raw}'. The YAML parser collapses the $$ to a single $. The resulting URL is the literal string ${__value.raw}, which the Grafana template parser does not substitute. Symptom: the clickable link points to http://tempo:3200/${__value.raw}. The drill opens Tempo with a literal ${__value.raw} in the URL.
4. The Tempo data source UID is wrong. The URL points to datasource=tempo but the data source UID is tempo-prod. Symptom: the pivot opens Explore with the wrong data source, or with the default data source picker.
5. The trace_id is truncated. The log pipeline truncates strings over 32 characters. The truncated trace_id is short. The Tempo query for the full 32 returns no trace. Symptom: the clickable link opens Tempo, Tempo returns no traces, the operator concludes the trace is missing.
6. The derived field is configured on the wrong data source. The team has three Loki data sources (one per region). The derived field is configured on the wrong one. Symptom: the pivot works in one region and is silently no-op in the other two.

How to troubleshoot it

The diagnostic order is “is the trace_id in the log line?”, “does the regex match?”, “is the URL right?”, “is the Tempo data source right?”.

1. Is the trace_id in the log line? logcli query --since=10m '\{service="checkout"\} | json | trace_id!=""' --tail=1. The trace_id field should be populated. The shape is a 32-hex value.
2. Does the regex match? Run the regex against the raw log line: echo '<log line>' | grep -oE '<regex>'. The capture group should be the trace_id. If the regex misses, the regex is the suspect.
3. Is the URL right? Inspect the data source configuration in Grafana (Administration → Data sources → Loki → Derived fields). The URL should contain ${__value.raw}. The substitution is replaced by the Grafana template parser at render time.
4. Is the Tempo data source right? curl -s -u admin:admin http://grafana:3000/api/datasources/uid/tempo. The URL field is the Tempo query frontend. The drill must point to the right backend.
5. Is the clickable link rendered? Open the Explore page, run the Loki query, hover over the log line. The derived field label should appear. If it does not, the regex is the suspect.

Security implications

The trace_id is a 128-bit random value. It is opaque and carries no semantic content. The pivot URL does not introduce new attack surface. The URL is internal to the Grafana instance and does not leave the cluster.

The risk is around the regex. A regex that is too loose (for example, matcherRegex: '[a-f0-9]{32}') will match the wrong substring and produce a broken link. The convention is to require the trace_id to be wrapped in a separator (quote, brace, comma) and to require the exact 32-hex length.

The second-order risk is around the log line content. A log line that contains a credential, a session token, or a PII field will be parsed by the derived field regex if the value matches the regex shape. The mitigation is to keep the trace_id in a separate field with a separate name, and to ensure the log pipeline does not log sensitive values in the trace_id field.

Performance implications

The derived field runs at render time, once per log line. The cost is a single regex match per line. At 1 000 lines per panel, the cost is 1 000 regex matches, which is sub-millisecond on a modern browser. The cost is negligible.

The cost to watch is the regex complexity. A regex with backtracking or alternation (for example, '(trace_id|traceID|trace-id)[:=]...' ) is O(n*m) where n is the line length and m is the regex pattern. The discipline is to keep the regex simple and anchored.

The second cost is the Loki parser. The derived field runs on the raw line, not on the parsed field. The Loki parser runs on every query. The combined cost is the regex match plus the parsed-field lookup. Both are sub-millisecond.

Production guidance

• Anchor the regex. The convention is to require the trace_id to be wrapped in a separator (quote, brace, comma) and to require the exact 32-hex length. The regex '(a-f0-9]{32}' is too loose. The regex '"trace_id":"([a-f0-9]{32})"' is right.
• Test the regex on production log lines. The test fixture is not the production log line. The regex is validated against a sample of production log lines before it is deployed.
• Render the trace_id as a hidden field. The log line is rendered with the derived field inline. The label (“Open trace”) is the clickable affordance. The raw trace_id is shown as a tooltip on hover.
• Provision the derived field per data source. The convention is one derived field per Loki data source. The field is not shared across data sources. The audit is a single shell command that lists every data source and its derived fields.

Verification

You should now be able to answer:

• What is the Loki derived field and what does it match?
• What is the difference between the JSON form and the logfmt form of the trace_id in a log line?
• Why does the URL use $${__value.raw} in the YAML provisioning?
• What is the failure shape of a regex that is too loose?
• How do you validate the pivot end to end with a synthetic request?

Quiz