Tempo Derived Fields · Observability

An engineer opens a slow trace in Tempo. The critical-path span is POST /checkout from the checkout-api service. It lasted 1.4 seconds; the backend dependency lasted 1.1. The engineer clicks the span to see its attributes — order.id = 8821, customer.id = u_9182, service.name = checkout-api — and wants to find the matching log line. There is no “Logs for this span” button. The engineer copies order.id by hand, opens the order management tool in a new tab, and pastes. The pivot from span to log line was not configured.

The fix is on the Tempo data source. Tempo’s data source exposes two configurations for joining to other systems: the tracesToLogsV1 block (the canonical “Logs for this span” link, which sits beside every span and jumps to Loki filtered by trace_id) and the derivedFields block (a finer-grained mechanism that matches attribute values inside a span and renders them as links, with the destination and the target data source named explicitly). The two configurations solve related problems at different layers:

tracesToLogsV1 is the inverse of lesson 03’s pivot-from-Loki. It is rendered as a single button on the span; the destination is always Loki; the join key is always the trace_id.
derivedFields is per-attribute. It can match any span attribute (order.id, customer.id, http.route, a custom business key), produce a link whose URL is built from the captured value, and route it to any data source (Loki, Prometheus, an external API via a JSON datasource).

This lesson is the second shape — the attribute-level pivot that the simple tracesToLogsV1 cannot express.

What it is

The Tempo data source’s derivedFields block is an array of rules. Each rule says: “for every span, run this regex against the span’s attribute map; if it matches, render the captured value as a link to that data source”. The block serves two production use cases that the tracesToLogsV1 button does not:

High-cardinality lookup keys. A span carries an order.id attribute whose value the team wants to be clickable as a link to the order management tool. The tracesToLogsV1 button does not handle this; the link is a derived field with a target URL the team controls.
Any attribute, any destination. The team wants customer.id to link to a customer profile page, or db.statement to link to the Loki search for that SQL. The tracesToLogsV1 block cannot express these; the derivedFields block can.

The trade-off is operational. The tracesToLogsV1 block is auto-rendered once per span and points at Loki by trace_id; the cost is fixed. The derivedFields block runs a regex against every span attribute on every open trace; the cost is per-rule per-attribute. A block with five rules and a span with twenty attributes is one hundred regex evaluations per open trace, all client-side.

Why a sysadmin cares

Three production patterns depend on Tempo data source derived-fields being correct:

Trace-to-log via a custom key. The team’s spans carry a correlation.id rather than a W3C trace_id. The tracesToLogsV1 block has nothing to bind to; the derivedFields block extracts correlation.id from the span and links to Loki’s correlation.id="<value>" filter.
Trace-to-internal-system. A span attribute like customer.id becomes a link to a customer profile dashboard, or order.id becomes a link to the order management tool. This is the only way to bridge Tempo and systems that do not speak trace IDs.
Trace-to-CDN-log. A span attribute like cache.key becomes a link to the CDN’s log search. The link lives in the trace UI as a span-level pivot; the engineer moves freely.

The cost is configuration discipline and browser CPU. The rules run client-side; a regex that captures too much or backtracks catastrophically is a self-inflicted DoS. The team pays once at provisioning for the right regex and once per rule per span in browser time.

How it works

The click path mirrors lesson 03 in shape, with the data sources reversed:

Tempo trace panel renders a span
   |
   |  Span attributes: { order.id = "8821", ... }
   v
derivedFields regex matches "order.id":"([0-9a-z-]+)"
   |
   |  Captured value: 8821
   v
Grafana renders the link
   url: "https://orders.internal/${__value.raw}"
   urlDisplayLabel: "Open order in OMS"
   |
   |  engineer clicks
   v
Open the OMS (or jump to Loki query, or open an external
   JSON datasource filtered by the captured value)

Three observations:

The regex evaluates against the span’s attribute map, not against the raw OTLP payload. Each attribute value is passed to the regex matcher; the first matching rule wins.
The link is a URL, not a query string. The team can produce any URL the destination accepts: a deep link into an internal tool, a Loki query= parameter that opens Explore, a Prometheus expression string. There is no server-side enforcement that the URL matches the destination data source; the rule’s url is what matters.
The datasourceUid is optional. If the destination is a Grafana data source, set datasourceUid to route the click through the Explore proxy (and through auth). If the destination is external (a third-party tool), omit it and Grafana opens the URL directly in a new tab.

Why two configurations exist

Tempo’s data source exposes two ways to join to other systems because they solve different problems.

tracesToLogsV1 answers: “for the current trace, give me the log lines across every service.” It is rendered as one button per span. The destination is always Loki. The join key is always trace_id. The configuration is a query template plus a UID.
derivedFields answers: “for the current span, let me click on any attribute value and route it somewhere.” It is rendered as a small inline link next to each attribute. The destination can be any Grafana data source or any external URL. The join key is a regex match on the attribute value. The configuration is a regex plus a URL template plus an optional UID.

The two are independent. A Tempo data source with only tracesToLogsV1 works for “show me the logs for this span”. A Tempo data source with only derivedFields works for “click this attribute to jump to its destination”. A production data source usually has both, configured side by side.

How to configure it

The Tempo data source — provisioned alongside the Loki data source from lesson 03:

# /etc/grafana/provisioning/datasources/tempo.yml
apiVersion: 1

datasources:
  - name: tempo-prod-eu
    uid: tempo-prod-eu
    type: tempo
    access: proxy
    orgId: 1
    url: https://tempo-prod-eu.internal:3200
    isDefault: false
    editable: false

    jsonData:
      tlsAuth: false
      tlsAuthWithCACert: true
      tlsSkipVerify: false

      # The canonical "Logs for this span" link (lesson 04's
      # inverse). Span click opens Loki filtered by trace_id.
      tracesToLogsV1:
        datasourceUid: loki-prod-eu
        tags: ['job', 'service.name', 'service.namespace']
        spanStartTimeShift: '10m'
        spanEndTimeShift: '10m'
        query: 'method="${__span.tags.method}"'
        refId: 'tempo-traces-to-logs'

      # Derived fields: per-attribute pivots from span
      # attributes to other data sources.
      derivedFields:
        # Rule 1 — link the order.id attribute to the order
        # management tool (an external URL).
        - name: orderID
          matcherRegex: '"order\\.id":"([0-9a-z-]+)"'
          url: 'https://orders.internal/orders/$${__value.raw}'
          urlDisplayLabel: 'Open order in OMS'

        # Rule 2 — link the customer.id attribute to the
        # Loki queries for that customer, via Grafana Explore.
        - name: customerID
          matcherRegex: '"customer\\.id":"([A-Za-z0-9_]+)"'
          url: '$${__value.raw}'
          urlDisplayLabel: 'Logs by customer.id'
          datasourceUid: loki-prod-eu
          internalLink:
            query: '{service_name=~"checkout-api|payments-api"} |= "customer.id=$${__value.raw}"'
            datasourceUid: loki-prod-eu

        # Rule 3 — link a custom correlation key. Useful when
        # the team's instrumentation is older than W3C.
        - name: correlationID
          matcherRegex: '"correlation\\.id":"([0-9a-f-]{36})"'
          url: '$${__value.raw}'
          urlDisplayLabel: 'Logs by correlation.id'
          datasourceUid: loki-prod-eu
          internalLink:
            query: '{service_name=~".+"} |= "correlation.id=$${__value.raw}"'
            datasourceUid: loki-prod-eu

      # Search-recent-traces: cap on the time window.
      search:
        maxDuration: '1h'
        defaultLimit: 20

      streamingAvailable: true

    secureJsonData:
      tlsCACert: |
        -----BEGIN CERTIFICATE-----
        MIIDazCCAlOgAwIBAgIUJx...
        -----END CERTIFICATE-----
      basicAuthPassword: ${TEMPO_PASSWORD}

A few production notes on the options:

tracesToLogsV1 and derivedFields are independent blocks. Either is valid alone. Most production deployments configure both so engineers can move freely between the two panels.
matcherRegex is dot-quoted. A span attribute key contains a dot; the regex must escape the dot to match the literal character (order\\.id). An unescaped dot matches any character; the rule matches every span and the first match wins, which may produce wrong links.
url is what opens. A datasourceUid and internalLink block route the click through the destination data source’s Explore view; without them, Grafana opens the URL directly. The $$ (literal $) is the Grafana template escape; without it, ${__value.raw} is evaluated on the URL, which produces a literal ${...} in the rendered URL.
internalLink.query is a Loki query string. The template variable ${__value.raw} (single $) is the captured substring; ${__var.<label>} is a stream label the team wants to carry over. Combined, the team can pivot to “show me every log line with this customer.id in the same service”.

How to validate it

# READ-ONLY: confirm the Tempo data source is provisioned
# with the expected blocks.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/uid/tempo-prod-eu \
  | jq '(.jsonData.tracesToLogsV1.datasourceUid), (.jsonData.derivedFields | length)'
# "loki-prod-eu"
# 3   (three rules)

# READ-ONLY: confirm a real span carries the attribute the
# rule is matching.
TRACE=4bf92f3577b34da6a3ce929d0e0e4736
curl -fsS -u "$TEMPO_USER:$TEMPO_PASS" \
  "http://tempo.internal:3200/api/traces/$TRACE" \
  | jq '.batches[].scopeSpans[].spans[] | select(.attributes[]
           | .key == "order.id") | .attributes[] | select(.key == "order.id")'
# { "key": "order.id", "value": { "stringValue": "8821" } }

# READ-ONLY: confirm the regex matches the attribute shape
# (the dot must be escaped).
echo '"order.id":"8821"' | grep -E '"order\.id":"([0-9a-z-]+)"'
# "order.id":"8821"

# READ-ONLY: confirm the destination Loki data source exists.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/uid/loki-prod-eu
# (A 404 here means the derivedFields rule points at a
# non-existent UID; the click will fail.)

# READ-ONLY: confirm the Loki query template works against
# a real customer.id value.
CUSTID=u_9182
logcli query --since 1h \
  '{service_name=~"checkout-api|payments-api"} |~ "customer.id='"$CUSTID"'"' \
  | head -5
# 2026-08-13T14:22:11.000Z {} ... customer.id=u_9182 msg="..."

# CONFIGURATION: do the click test manually in a browser.
# Open the trace; click an attribute that has a derived field;
# confirm Grafana routes the click correctly. There is no
# good API for the click itself; the rule load is what the
# API can verify.

If step 2 returns no span attributes for order.id, the attribute was renamed or the order service was not instrumented. If step 5 returns zero lines, the Loki query template is wrong (the |= filter does not parse because customer.id=u_9182 is the wrong shape for the line). If the API in step 1 returns a derivedFields array of zero, the provisioning file failed to parse.

How it can fail

Unescaped dot in the attribute key. The regex is order.id="..." instead of order\\.id="...". Symptom: the rule matches every span attribute (because . matches any character), the first match wins, links go to the wrong destination.
Destination data source UID typo. datasourceUid in internalLink points at loki-prod-eu but the data source is actually loki or loki-eu. Symptom: the link renders; the click opens an “unknown data source” panel.
datasourceUid missing on a non-Grafana URL. The rule has no datasourceUid but the url points at an internal Grafana data source. Symptom: Grafana opens the URL in a new tab as a raw HTTP request; the engineer’s browser does not get the same Grafana session as a data source click would; auth fails.
Template variable escaping. url: '${__value.raw}' (single $) interpreted by Grafana as a template and the rendered URL becomes the literal ${__value.raw} — the click goes to https://orders.internal/orders/${...}. The ${...} template interpolation in Grafana does not happen because the internalLink shape requires $$.
Attribute not on the span. The rule’s regex matches customer.id but the application set the attribute as customerId or customer_id. Symptom: the rule never matches; the attribute renders as plain text.
Regex DoS. A catastrophic backtracking pattern on matcherRegex against every attribute of every span. Symptom: opening a trace with a wide number of spans freezes the browser tab.

How to troubleshoot it

The diagnostic order for “the Tempo derived-fields link does not appear or does not behave correctly”:

Does the span carry the attribute? curl /api/traces/<id> and inspect the span’s attributes. The rule is correct; the attribute is missing.
Does the regex match the attribute shape? Run the regex against the attribute value with grep -E or a regex tool. The attribute is correct; the regex is wrong.
Is the dot escaped? A regex key with a dot in the attribute name must escape the dot. Use a regex tester to confirm.
Does the destination UID resolve? Use the Grafana API to confirm the UID. A renamed or removed destination breaks the click silently.
Is the $$ escape present in url? A url that contains ${__value.raw} (single $) inside an internalLink is being evaluated as a template and the captured value is replaced. The correct shape is $${__value.raw} (double $).

What if every link on a span goes to the wrong destination?

The most common cause is regex anchor mismatch. A derivedFields block whose matcherRegex uses . (any character) instead of \\. (literal dot) matches every attribute key in the span. The first matching rule wins; if the first rule was intended for order.id but matches xx id, the captured value is wrong, and the URL is rendered with the wrong value pointing at the wrong destination.

The second most common cause is rule order. A team that adds rules over time ends up with five rules; the first that matches wins. If a generic rule for id runs before the specific rule for order.id, the latter never fires.

The fix is to anchor and order. Start each rule with a specific key (order\\.id, customer\\.id), then order specific rules before generic ones, and validate by clicking each link on a real span.

Security implications

The Tempo data source’s derivedFields rules produce URLs the engineer’s browser opens. Two risks:

Open redirect. A rule whose matcherRegex captures a value from an attacker-controlled attribute and uses it as the URL host or path can produce a URL that opens an attacker-controlled system. Pin the scheme and the host; treat matcherRegex as if it could be triggered by attacker-controlled data.
ACL bypass. A datasourceUid that routes through a Grafana data source inherits the destination’s ACLs. Confirm that the destination data source has the same ACLs as Tempo; otherwise the engineer sees Loki lines they cannot see directly.

The rule is client-side; nothing on the server checks it. The review is a peer review of the provisioning file.

Performance implications

The derivedFields block runs in the browser against every attribute on every open span. The cost grows linearly with the number of rules and the number of attributes per span. A trace with 200 spans, 20 attributes per span, and 5 derived rules is 20 000 regex evaluations on every trace open. On a modern laptop, this is roughly 100 ms on a busy trace.

The browser cost is bounded by:

The number of derivedFields rules (prefer one or two over five).
The matcherRegex efficiency (no catastrophic backtracking).
The number of attributes per span (a span with 100 custom attributes is the costliest case; review the high-cardinality attribute policy in lesson 02).

The server cost is zero — the rules are evaluated in the browser. The team’s only server-side cost is the destination data source fetch, which is the same cost whether the link was provisioned or not.

Production guidance

Anchor every regex with the attribute key (order\\.id, not order.id).
Prefer one or two rules over five. The browser cost is per-rule per-attribute; ten rules doubles the open time of every trace.
Validate the link on a real span before rolling the provisioning file out. The capture-and-click test in lesson 06 is the production answer.
Roll back by reverting the data source provisioning file and restarting Grafana (or using the API to reload). Grafana does not hot-reload data sources on a file change unless explicitly enabled.

Verification

You should now be able to answer:

What is the operational difference between tracesToLogsV1 and derivedFields?
Why must order\\.id have an escaped dot in matcherRegex?
When would a derived-field link to a non-Grafana destination not need a datasourceUid?
What is the first diagnostic step when every derived-field link goes to the wrong destination?
Why is ${__value.raw} written with double $ inside a derivedFields.url?