Structured Logs in Grafana · Observability

The query returns twenty log lines. The on-call engineer squints at them in Grafana and reads the same JSON blob twenty times — {"ts":"...","level":"error","request_id":"7f4a...","msg":"payment_intent_failed","latency_ms":1240}. The fields are right there. The columns are not. The query did its job; the rendering did not.

Structured logs in Grafana is the discipline that closes the loop between a parsed query and a panel the on-call engineer can actually read.

What “structured logs in Grafana” means

The phrase covers three pieces of behaviour, each a different part of the stack:

Parsed-field rendering. The Grafana Loki panel turns each parsed field into a column. | json produces columns named ts, level, request_id, msg, latency_ms. The wall of JSON becomes a table.
Auto-detected JSON. When a log line is itself a JSON object and the panel has the “Display” option set, Grafana renders the JSON as columns without any pipeline configuration. The auto-detection is a fallback; the right answer is to parse at the pipeline.
Derived fields. A Grafana data-source configuration that says “this parsed field is a URL or an ID, link it to another system”. The canonical example is trace_id linking to Tempo and runbook_url linking to the team’s runbook.

These three pieces are independent. A panel can render columns without derived fields, or it can link a column to another system without rendering columns. A production panel does both.

Why a sysadmin cares

A panel that renders a wall of JSON is a panel the on-call engineer reads by eye. A panel that renders columns is a panel the on-call engineer scans. The difference between “tolerable during an incident” and “actively hostile to the on-call team” is the difference between these two.

Time to first pivot. A request_id column with a derived field that links to Tempo means the on-call engineer clicks the value and lands on the trace. The same panel without derived fields means the engineer copies the value, opens a Tempo tab, pastes, and clicks. The friction is real.
Cross-signal correlation. A derived field that links level to the alert rule and trace_id to Tempo turns the log panel into a control surface. The engineer stops reading text and starts clicking.
Documentation surface. A runbook_url derived field is the canonical place to surface the runbook from inside the panel. The on-call engineer does not need to remember the URL; the panel carries it.

How it works — the mental model

Loki query result
   |
   v
parsed JSON entries (or raw, if the pipeline did not parse)
   |
   v
Grafana Loki data source
   |- inspects each entry for a JSON shape (auto-detect)
   |- applies derived fields (URL / ID extraction)
   |
   v
panel renderer
   |- Statistics panel: counts per parsed field value
   |- Logs panel: rows with columns for each parsed field
   |- Table panel: rows with columns
   |
   v
on-call engineer reads the panel

Three pieces of configuration matter: the data source, the panel, and the panel’s display options.

Data source. Configured once per Grafana instance. Settings for derived fields, default max lines, and whether JSON auto-detection is on.
Panel. The LogQL query plus the visualisation type. A “Logs” panel renders rows; a “Table” panel renders columns; a “Statistics” panel renders counts.
Display options. Per-panel switches for “wrap lines”, “deduplicate”, “show labels”, “show time”. The right combination depends on the question.

How derived fields work end-to-end

A derived field has four properties:

Name. The label of the link in the panel.
Matcher. A regex against the log line that captures the value. (?P<trace_id>[a-f0-9]{32}) captures a 32-character hex string into trace_id.
URL or internal link. Where the captured value points. For Tempo, the URL is ${__value} substituted into the Tempo datasource URL. For a runbook, it is a static base URL with ${__value} appended.
Open in new tab. Whether clicking the link opens a new browser tab or replaces the current one.

When the panel renders, Grafana applies the matcher against each log line. A successful match produces a clickable value in the named column. The link uses the captured value as the substitution.

Three production details:

The matcher is RE2. A pathological pattern is bounded but still costs CPU on every line.
The URL substitution is ${__value}. A typo in the substitution is a silent failure — the link renders but points to the wrong place.
Derived fields are data-source-scoped. A derived field defined on the Loki data source is available to every Loki panel that uses that data source. To target one panel, scope the matcher narrowly.

How to configure it

The data-source configuration in the Grafana UI maps to a YAML that the provisioning system can apply:

# grafana datasources provisioning
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    jsonData:
      maxLines: 1000
      derivedFields:
        - name: trace
          matcherRegex: '"trace_id":"([a-f0-9]{32})"'
          url: '${__value}'
          urlDisplayLabel: 'View trace in Tempo'
          internalLink:
            datasourceName: Tempo
            query: '${__value}'
        - name: runbook
          matcherRegex: 'runbook_url=(https?://\S+)'
          url: '${__value}'
          urlDisplayLabel: 'Open runbook'

The panel-side configuration. The query produces the parsed fields; the panel renders them.

# Query: parse the JSON and filter to error lines.
{service="checkout"} | json | level="error"

In the panel editor, the “Display” section offers:

Show labels — render the stream labels as a header on each row. Production default is “on”.
Wrap lines — for long lines, wrap. Production default is “off” for tables, “on” for raw logs.
Deduplicate — collapse identical consecutive lines. Production default is “on” for noisy services.
Order by time — descending or ascending. Production default is “descending”.

A panel that displays a parsed-field column needs the query to include | json or the equivalent parser. The “Display” option called “Auto-detect JSON” does the same thing at render time; the pipeline parse is the better answer because the index can filter on the parsed field at ingest.

How to validate it

# 1. The query returns parsed entries.
logcli --addr=http://loki:3100 query --since=5m --limit=1 \
  '{service="checkout"} | json'
# {service="checkout"} 2026-08-14T14:02:11Z
#   level=error request_id=7f4a1c8e9b1d4e2a
#   msg=payment_intent_failed latency_ms=1240

# 2. The same query through Grafana returns columns.
#    (Open the panel; visually confirm columns: ts, level,
#    request_id, msg, latency_ms.)

# 3. The derived field is detected.
#    (Open the panel; click the trace_id value; confirm the
#    link to Tempo opens.)

# 4. The auto-detected JSON path renders the same columns.
#    (Open a panel whose query has no | json; confirm the JSON
#    is detected and rendered.)

# 5. The data-source configuration is loaded.
curl -s -u admin:admin http://grafana:3000/api/datasources/name/Loki | jq .
# {
#   "name": "Loki",
#   "type": "loki",
#   "jsonData": {
#     "derivedFields": [
#       { "name": "trace", "matcherRegex": "..." }
#     ]
#   }
# }

The single most useful validation is visual. Open the panel. Are the columns there? Does the derived-field link work? The query side has a way to fail silently (the columns do not appear); the visual side surfaces it immediately.

How it can fail

Six recurring failure modes. Each maps to an observable symptom.

The query has no parser. {...} |= "error" returns the raw line. The panel renders the raw line as a wall of text. Symptom: no columns; every row is the raw JSON blob.
The pipeline did not parse at ingest. The Loki entry is the raw string. | json parses at query time, which works, but the index cannot filter on the parsed fields. Symptom: queries that should be index-backed scan chunks.
The derived-field matcher is wrong. A typo in the regex captures nothing. The link does not appear. Symptom: the column renders as text, not as a link.
The derived-field URL substitution is wrong. A typo in ${__value} or a missing base URL means the link is malformed. Symptom: clicking the link opens a 404 or an empty Tempo panel.
The data-source configuration was never loaded. A new Grafana instance, a new derived field, but the provisioning file was not re-applied. Symptom: every Loki panel has no derived fields; the link never appears.
The display option “Show labels” is off. The stream labels are the canonical context. With them off, the engineer reads a row without knowing which service or environment it came from. Symptom: the panel is context-free; the on-call team reads wrong lines into the wrong service.

How to troubleshoot it

The diagnostic order for “the columns are missing” or “the derived field does not work”:

Confirm the parser ran. logcli query '{...} | json' — is the output parsed?
Confirm the data source has derived fields. curl /api/datasources/name/Loki | jq .jsonData.derivedFields — is the configuration loaded?
Confirm the matcher regex against a sample line. Take one log line and apply the regex. Does it match? Does it capture the right value?
Confirm the URL substitution. Open the link in a new tab. Does it land on the right system?
Inspect the panel display options. Are columns enabled? Are stream labels visible?
Reload the data-source configuration. A change in the provisioning file is not picked up until the file is re-applied or the data source is reloaded.

Security implications

Trace ID leakage. A trace_id derived field that links to Tempo exposes the trace. The trace may carry sensitive data (request bodies, headers, credentials). The link is doing its job; the trace is the problem. The right place to scrub is the trace pipeline.
Runbook URL injection. A runbook_url derived field that captures from log content can be poisoned if the log content is attacker-controlled. The matcher should match only known runbook URL shapes, not arbitrary URLs.
Cross-tenant access. A derived field that links to a Tempo datasource visible to another tenant exposes traces across tenants. RBAC on the underlying datasources is the authoritative control.

Performance implications

Render cost. The panel renders every line in the result. A thousand-line result is a thousand parse-and-render cycles in the browser. The right answer is a smaller query or a more selective filter.
Derived-field matcher cost. The matcher runs against every line. A pathological pattern is bounded but still costs CPU per line. An anchored regex is the cheapest pattern.
Auto-detect JSON. The auto-detect path inspects every line for a JSON shape. It is cheaper than parsing every line but not free. Disabling it when the pipeline parses is the right answer.

Production guidance

Configure the data source via provisioning, not by hand. The YAML is the source of truth; the UI is a transient view.
Keep derived fields to the small set the team actually uses: trace_id, runbook_url, request_id, alert_id. Every additional derived field is matcher CPU on every line.
Use the pipeline parse at ingest and the panel render at query. The pipeline parse is indexable; the panel parse is not.
Surface runbooks from inside the panel via derived fields. The on-call engineer does not need to remember the URL.

Verification

You should now be able to answer:

What is the difference between a parsed field and a derived field?
Where is the right place to parse JSON for an indexable field?
What configuration must be present for a derived-field link to render?
Why is “Show labels” a panel setting that should default to on?