Grafana Correlations · Observability

A dashboard has three panels: a metric panel for the error rate, a log panel for the application, and a trace panel for the checkout journey. The panels look correlated. The on-call engineer clicks the metric, sees the logs, sees the trace. The drill is real. The dashboards for the other services have none of it. The on-call engineer has to open three tabs, copy labels by hand, paste them into Loki, find the trace_id in the log line, paste it into Tempo.

The difference between the two views is the Grafana correlation. The correlation is the wiring that connects one panel to another. The wiring is a data link on the panel and a derived field on the data source. The wiring is explicit. The wiring is per-dashboard. The wiring is what the rest of the lesson turns into a discipline.

What it is

A Grafana correlation is the wiring that turns a click in one panel into a query in another data source. The wiring has two halves.

Data link — a URL template on the panel that is substituted at click time with the values of the clicked datapoint. The template opens a new Explore page or navigates to another dashboard.
Derived field — a regex on the data source that extracts a value from a log line and turns it into a clickable link. The link is rendered at panel render time.

The two halves compose into a multi-signal correlation. The metric panel has a data link that opens Loki with the same labels. The Loki data source has a derived field that opens Tempo with the trace_id. The Tempo data source has a traceToLogs block that opens Loki with the same tags. The on-call engineer can click in any direction and land on the right signal.

  Metric panel      Loki data source     Tempo data source
  +------------+    +---------------+    +---------------+
  | data link  |--> | derived field |--> | traceToLogs   |
  +-----+------+    +-------+-------+    +-------+-------+
        |                   |                    |
        v                   v                    v
  Loki Explore       Tempo trace          Loki Explore
  (same labels)      (trace_id)           (same tags)

Why a sysadmin cares

The correlation is the difference between a dashboard that displays three signals and a dashboard that lets the operator investigate. The correlation turns a wall of green panels into a navigation graph.

Three operational payoffs.

Investigation speed. The correlation removes the label copy step. The on-call engineer clicks, the drill lands on the right signal. The drill is one second, not one minute.
Investigation correctness. The correlation preserves the time range, the label set, and the join key. The operator is looking at the same window as the source panel, not the dashboard’s default range.
Cross-signal navigation. The correlation covers all three signals. The operator can move from metric to log to trace to log to metric without leaving the dashboard.

The cost is the discipline of configuring the correlation on every panel and every data source. The investment is YAML in the provisioning. The return is paid on every incident.

How it works — the URL template

The Grafana data link is a URL template. The template is substituted at click time with the values of the clicked datapoint. The substitution variables:

${__series.labels} — the full label set of the metric series, formatted as {job="x", instance="y"}.
${__series.name} — the metric name.
${__value.time} — the timestamp of the datapoint.
${__value.raw} — the raw value of the datapoint (for a histogram exemplar, this is the trace_id).
${__url_time_range} — the dashboard’s current time range.
${__dashboard} — the dashboard name.
${__dashboardUid} — the dashboard UID.
${__data.fields.<name>} — a specific field value from a table panel.

The variables are substituted at click time. The result is a concrete URL that Grafana opens in a new tab.

The two halves of the correlation:

Half 1 — the data link on the panel. The data link template is per-panel. The template can target a different data source, a different dashboard, or a different URL.

# In the dashboard JSON, on the metric panel.
options:
  dataLinks:
    - title: 'Logs for {{job}} {{instance}}'
      url: '/explore?schemaVersion=1&panes=%7B%22logs%22%3A%7B%22datasource%22%3A%22loki%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22%7B${__series.labels}%7D%22%7D%5D%7D%7D&orgId=1'

Half 2 — the derived field on the data source. The derived field is per-data source. The derived field extracts a value from the log line and renders it as a clickable link.

# In the Loki data source provisioning.
jsonData:
  derivedFields:
    - name: traceID
      matcherRegex: '"trace_id":"([a-f0-9]{32})"'
      url: '$${__value.raw}'
      urlDisplayLabel: 'Open trace'

The two halves compose. A click on the metric panel opens Loki with the same labels. A click on the trace_id in the log line opens Tempo with the trace_id. The composition is explicit. The composition is per-dashboard.

The variable substitution pipeline

The Grafana template parser runs the substitution in two passes.

Pass 1 — the panel rendering. The data link template is parsed. The ${__series.labels} is flagged for substitution at click time. The $${__value.raw} is parsed as a literal ${__value.raw} (the $$ is the YAML escape for a literal $). The literal is the template for the click target.

Pass 2 — the click handler. The panellist clicks the datapoint. The click handler reads the panel state (the labels, the value, the time), substitutes the variables in the template, and produces the final URL. The URL is opened in a new tab.

The substitution is order-sensitive. The panel state must be populated before the click handler runs. The most common failure mode is the substitution producing an empty value because the panel state was not populated.

The YAML escape is non-negotiable. A data link with ${__value.raw} (no $$) is parsed as a YAML scalar that contains the literal characters ${__value.raw}. The click handler does not substitute the value because the variable is not in the template. The URL is the literal string.

How to configure it

The correlation is configured per data source. The provisioning YAML is the source of truth.

# grafana/provisioning/datasources/observability.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    url: http://prometheus:9090
    jsonData:
      httpMethod: POST
      # The exemplar destination is a hidden data link on the
      # histogram bucket. The trace_id is rendered as a
      # clickable link that opens Tempo.
      exemplarTraceIdDestinations:
        - name: trace_id
          datasourceUid: tempo
          urlDisplayLabel: 'Open trace'

  - name: Loki
    type: loki
    uid: loki
    url: http://loki:3100
    jsonData:
      # The derived field turns a trace_id in the log line
      # into a clickable link that opens Tempo.
      derivedFields:
        - name: traceID
          matcherRegex: '"trace_id":"([a-f0-9]{32})"'
          url: '$${__value.raw}'
          urlDisplayLabel: 'Open trace'

  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo:3200
    jsonData:
      httpMethod: GET
      # The trace-to-logs pivot opens Loki with the same
      # tags carried by the trace and a 500ms lookback.
      traceToLogs:
        datasourceUid: loki
        tags: ['job', 'instance', 'status']
        query: '{${__tags}} | json | trace_id="$${__trace_id}"'
        spanStartTimeShift: -500ms
        spanEndTimeShift: 0

The data links on the panels are configured per panel. The two patterns are common.

Pattern A — link on the metric panel that opens Loki:

{
  "type": "timeseries",
  "targets": [
    {
      "expr": "sum by(job, instance, status) (rate(http_server_requests_total{job=\"checkout\"}[1m]))",
      "legendFormat": "{{status}} on {{instance}}"
    }
  ],
  "options": {
    "dataLinks": [
      {
        "title": "Logs for {{job}} {{instance}} {{status}}",
        "url": "/explore?schemaVersion=1&panes=%7B%22logs%22%3A%7B%22datasource%22%3A%22loki%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22%7B${__series.labels}%7D%22%7D%5D%7D%7D&orgId=1"
      }
    ]
  }
}

Pattern B — link on the histogram panel that opens Tempo:

{
  "type": "histogram",
  "targets": [
    {
      "expr": "histogram_quantile(0.99, sum by(le, job, instance) (rate(http_server_request_duration_seconds_bucket{job=\"checkout\"}[1m])))",
      "legendFormat": "{{job}} {{instance}}"
    }
  ],
  "options": {
    "exemplars": true,
    "dataLinks": [
      {
        "title": "Trace for this exemplar",
        "url": "/explore?schemaVersion=1&panes=%7B%22traces%22%3A%7B%22datasource%22%3A%22tempo%22%2C%22queries%22%3A%5B%7B%22query%22%3A%22$${__value.raw}%22%2C%22queryType%22%3A%22traceql%22%7D%5D%7D%7D&orgId=1"
      }
    ]
  }
}

The variable substitution is the join key. The ${__series.labels} carries the Prometheus label set into the Loki query. The ${__value.raw} carries the exemplar trace_id into the Tempo query. The ${__trace_id} carries the Tempo trace_id into the Loki query. The variables are the join.

How to validate it

# 1. The Prometheus data source has the exemplar destination.
curl -s -u admin:admin http://grafana:3000/api/datasources/uid/prometheus \
  | jq '.jsonData.exemplarTraceIdDestinations'
# [{"name":"trace_id","datasourceUid":"tempo","urlDisplayLabel":"Open trace"}]

# 2. The Loki data source has the derived field.
curl -s -u admin:admin http://grafana:3000/api/datasources/uid/loki \
  | jq '.jsonData.derivedFields'
# [{"name":"traceID","matcherRegex":"\"trace_id\":\"([a-f0-9]{32})\"",
#   "url":"${__value.raw}","urlDisplayLabel":"Open trace"}]

# 3. The Tempo data source has the traceToLogs block.
curl -s -u admin:admin http://grafana:3000/api/datasources/uid/tempo \
  | jq '.jsonData.traceToLogs'
# {"datasourceUid":"loki","tags":["job","instance","status"],
#  "query":"{${__tags}} | json | trace_id=\"${__trace_id}\"",
#  "spanStartTimeShift":"-500ms","spanEndTimeShift":"0"}

# 4. The dashboard has the data links on the panels.
curl -s -u admin:admin http://grafana:3000/api/dashboards/uid/checkout \
  | jq '.dashboard.panels[] | select(.options.dataLinks) | .title'
# "HTTP request rate by status"
# "HTTP request duration p99"

# 5. The drill opens the right data source.
# (UI step) Click the metric panel. The URL opens Loki
# Explore. Click the trace_id in the log line. The URL opens
# Tempo. Click a span in the trace. The URL opens Loki.

How it can fail

Six recurring failure shapes.

The data link uses ${__field.labels} instead of ${__series.labels}. The field substitution is only populated on table panels. On a time series panel, the field is empty. Symptom: the metric-to-log drill opens Loki with {job=""}, the query is a parse error.
The derived field URL uses ${__value.raw} without the YAML escape. The YAML parser collapses the value to a literal string. The Grafana template parser does not substitute. Symptom: the clickable link points to http://tempo:3200/${__value.raw}. The drill opens Tempo with a literal ${__value.raw} in the URL.
The data source UID is wrong. The data link uses datasource=tempo but the data source UID is tempo-prod. Symptom: the drill opens Explore with the wrong data source, or with the default data source picker.
The traceToLogs query uses ${__trace_id} without the YAML escape. The same YAML escape issue. Symptom: the trace-to-logs drill opens Loki with a literal ${__trace_id} in the query.
The labels are not aligned between Prometheus and Loki. The Prometheus series is labelled with instance="10.0.1.5:8080". The Loki stream is labelled with instance="checkout-7d4b8". Symptom: the metric-to-log drill opens Loki with a query that returns no rows.
The trace_id is not in Tempo. The trace was never sampled, or the sampling rate was changed. Symptom: the log-to-trace drill opens Tempo with the trace_id, Tempo returns no traces.

How to troubleshoot it

The diagnostic order is “is the data source configured?”, “is the data link on the panel?”, “does the substitution expand?”, “does the resulting query return rows?”.

Is the data source configured? curl -s -u admin:admin http://grafana:3000/api/datasources/uid/loki. The derivedFields field should be populated. The exemplarTraceIdDestinations field on the Prometheus data source should be populated. The traceToLogs field on the Tempo data source should be populated.
Is the data link on the panel? Open the panel editor. The data link field should be populated. The template should contain the right substitution.
Does the substitution expand? Open the dashboard. Click the panel. Observe the URL. The substitution should be replaced with the actual value. If the URL contains ${__series.labels}, the template parser is not running.
Does the resulting query return rows? Run the expanded query in logcli or in the Explore query bar. The query should return the rows for the label set. If it returns zero rows, the labels are misaligned or the trace_id is missing.
Is the Tempo data source right? curl -s -u admin:admin http://grafana:3000/api/datasources/uid/tempo. The URL field should point to the Tempo query frontend.

What if the variable substitution is misconfigured?

The on-call engineer clicks the metric panel. The URL opens Explore with the right data source, but the query is empty. The engineer concludes the Loki data source is broken. The engineer pages the platform team. The platform team finds the data link uses ${__field.labels} instead of ${__series.labels}. The fix is to update the panel JSON and re-import the dashboard.

The cost is fifteen minutes of paging. The fix is at the dashboard, not the data source. The mitigation is to add a validation step that asserts the data link template uses the right substitution. The validation is a curl-based check that runs in the CI pipeline and fails the build if the template is wrong.

The discipline is to lint the dashboard JSON in the CI pipeline. The linter asserts that every data link template uses ${__series.labels} or ${__value.raw}, never ${__field.labels}. The linter catches the regression before the on-call engineer does.

Security implications

The Grafana correlation does not introduce new attack surface. The URLs are internal to the Grafana instance and do not leave the cluster. The substitution variables are derived from the panel state, not from user input.

The risk is around the data source UID. A data link that points to a data source UID that the user does not have access to will produce a permission error. The mitigation is to verify the data source permissions before deploying the dashboard.

The second-order risk is around the URL encoding. A label value that contains a special character (for example, &) can break the URL. The mitigation is to URL-encode the label values in the substitution template. The Grafana template parser does this automatically for the ${__series.labels} substitution.

Performance implications

The correlation is a single Explore page load per click. The cost is the cost of the Loki or Tempo query, not the cost of the correlation.

The cost to watch is the cardinality of the substitution. A data link that uses ${__series.labels} on a high-cardinality metric series produces a Loki query that scans every stream in the Loki index. The discipline is to pivot on labels with low cardinality (job, instance, status) and to filter on the high-cardinality labels (user_id) inside the query.

Production guidance

Lint the dashboard JSON in CI. The linter asserts that every data link template uses the right substitution. The linter catches the regression before the dashboard is deployed.
Use the bare ${__series.labels} for metric-to-log pivots. The field substitution is fragile. The series substitution works on every panel type.
Use $${__value.raw} in the YAML provisioning. The $$ is the YAML escape for a literal $. Without the escape, the YAML parser collapses the value.
Validate the correlation under load. A dashboard that shows labels at 1 RPS does not show labels at 10 000 RPS. The validation is a load test that asserts the labels align between Prometheus and Loki.

Verification

You should now be able to answer:

What is the difference between a data link and a derived field?
Which URL template variable carries the trace_id from the histogram to Tempo?
Which YAML escape is required for the derived field URL?
What is the failure shape of a misaligned label set between Prometheus and Loki?
How do you validate the correlation end to end with a synthetic request?