Correlation Configuration

A panel titled “5xx rate by service and route” has a data link with a title that says View logs, a url that contains a Loki Explore URL, and a targetBlank: true flag. An engineer clicks it during an incident and lands in the wrong Loki cluster. They do not know why. The URL template references loki-prod-eu; the panel comes from prom-prod-us. The link was copied from a dashboard the team inherited six months ago, and the UID was never updated when the team added the US Loki data source.

A different engineer opens the panel editor, copies the existing link, and adds a second one to “View traces” — but the second link’s url references the dashboard template variable ${ds_logs} instead of the data source UID. Grafana silently substitutes the variable and the link opens the default Loki data source, which is the staging cluster. The traces link looks identical to the logs link on the panel. Both are broken in production.

The data link is four fields and one discipline. The discipline is “every link targets a data source by UID and every value in the URL was deliberately substituted”. The author who treats the link as a sentence they have to type once gets a brittle link; the author who treats it as a small piece of code with variables gets a link that survives renames, additions, and team handoffs.

What it is

A Grafana 11.x data link is a panel-level configuration that turns a click on a panel point into an HTTP request to another Grafana location. The HTTP request is most often a URL into Explore (Loki, Prometheus, Tempo), into another dashboard, or into an external system.

The link is defined as an entry in the panel’s fieldConfig.defaults.links array. There are four fields:

{
  "title": "View logs for ${__series.labels.service}",
  "url": "/explore?schemaVersion=1&panes=...&from=...&to=...",
  "targetBlank": true,
  "includeVars": true
}

title is the link’s display label, rendered next to the panel point. url is the destination, with template variables substituted at click time. targetBlank controls whether the link opens in a new tab or replaces the current one. includeVars controls whether dashboard template variables are forwarded to the destination.

The same shape appears, dashboard-wide, in datasource_correlations. The dashboard-level form replaces the panel’s links with a single block per (metric, logs) pair. Grafana renders the same click behaviour for every panel that targets the metric data source.

Why a sysadmin cares

The link is the only piece of correlation an author writes that the engineer will trust on a bad day. Three production shapes appear when the correlation config is sloppy:

• The right link, the wrong data source. The URL targets a Loki data source that exists but is the wrong one — the staging cluster, the EU cluster, the legacy ingestor. The link opens; the lines are not there. The on-call concludes the service is silent. The real outage is somewhere else.
• The right data source, the wrong variable. The URL template references ${env} (a dashboard template variable), which forwards env=staging because that is the variable’s value in the engineer’s saved view. The link opens against staging; the production lines are not there.
• The right variable, the wrong substitution rule. The URL references ${service} but the metric panel exposes the value as service_name after a label rename. The substitution is empty; the LogQL returns no lines.

How it works

The mental model is “the link is a small function”. The inputs are the clicked series, the panel’s point time, and the dashboard’s variable state. The output is an HTTP URL. The function body is the URL template; the variables are the parameters.

   data link config            inputs              output
   ----------------            ------              ------
   title: ${service}-logs  +  series labels   =   rendered title
   url:   /explore?        +  point time      =   rendered URL
         ds=${LOKI_UID}    +  dashboard vars
         query={svc=...}
         from=${__value.time:date-seconds}-120
         to=${__value.time:date-seconds}+120
         &includeVars=true

Grafana resolves the template in two stages. First it substitutes the data-link template variables (${__value.time}, ${__value.time:date-seconds}, ${__series.labels.<name>}, ${__data.fields.<name>}, ${__value.raw}). Then it substitutes the dashboard template variables (${env}, ${service}, ${region}) from the dashboard’s current state. The order is fixed; the two namespaces do not collide because data-link variables begin with __ and dashboard variables do not.

The resulting URL is parsed by Grafana’s router. URLs that begin with /explore, /d/, or /dashboard/ are routed to the corresponding in-app view. URLs that begin with http:// or https:// are opened as external links. Everything else is treated as a relative path inside Grafana.

How to configure it

The minimum useful correlation config has three parts: a title the engineer can scan, a URL that targets a specific data source by UID, and a time window centred on the clicked point.

{
  "type": "timeseries",
  "title": "5xx rate by service and route",
  "datasource": { "type": "prometheus", "uid": "prom-prod-us" },
  "targets": [
    {
      "expr": "sum by(service, route) (rate(http_requests_total{status=~\"5..\"}[5m]))",
      "refId": "A"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "links": [
        {
          "title": "Logs: ${__series.labels.service} ${__series.labels.route}",
          "url": "/explore?schemaVersion=1&panes=%7B%22logs%22%3A%7B%22datasource%22%3A%22loki-prod-us%22%2C%22queries%22%3A%5B%7B%22expr%22%3A%22%7Bservice%3D%5C%22%24%7B__series.labels.service%7D%5C%22%2Croute%3D%5C%22%24%7B__series.labels.route%7D%5C%22%2Cstatus%3D%5C%225%5C%22%7D%22%7D%5D%7D%7D&from=${__value.time:date-seconds}-120&to=${__value.time:date-seconds}+120",
          "targetBlank": true,
          "includeVars": true
        }
      ]
    }
  }
}

The relevant options, walked through:

• title — the link’s display label. The ${__series.labels.<name>} syntax substitutes the value of the named label on the clicked series. A title that includes the service and route helps the engineer decide whether to click. A title that says “Logs” only is useless; the engineer does not know which service they are about to investigate.
• url — the destination URL with template variables. The panes parameter is the URL-encoded JSON that Grafana 11.x uses to restore Explore state. The expr field inside the JSON is the LogQL stream selector. ${__value.time:date-seconds} resolves to the clicked point’s timestamp in seconds since the epoch. -120 and +120 are the window offsets in seconds.
• targetBlank: true — opens the link in a new tab. The on-call engineer is investigating the dashboard; the pivot opens alongside it. targetBlank: false replaces the dashboard in the same tab and is almost always wrong for correlation links.
• includeVars: true — forwards dashboard template variables to the destination. Useful when the dashboard has ${env} or ${region} and the destination uses the same scheme. includeVars: false is the right choice when the destination data source has its own variable state.

The dashboard-level equivalent is datasource_correlations:

{
  "datasource_correlations": [
    {
      "uid": "prom-prod-us-loki-prod-us",
      "sourceUID": "prom-prod-us",
      "targetUID": "loki-prod-us",
      "label": "Logs",
      "description": "Pivot from any 5xx rate series to the underlying log lines",
      "config": {
        "field": "service",
        "target": {
          "expr": "{service=\"${__field.labels.service}\", route=\"${__field.labels.route}\", status=\"5\"}"
        },
        "type": "logs"
      }
    }
  ]
}

The dashboard-level form replaces ${__series.labels.<name>} with ${__field.labels.<name>} for the same value. The template namespace is different but the discipline is the same: target a data source by UID, pass the labels, pass the filter.

How to validate it

# READ-ONLY: list the links on a panel.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  "http://grafana.internal:3000/api/dashboards/uid/${DASH_UID}" \
  | jq '.dashboard.panels[]
        | select(.title=="5xx rate by service and route")
        | .fieldConfig.defaults.links'

# READ-ONLY: confirm the target data source exists.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/uid/loki-prod-us
# {"id":2,"uid":"loki-prod-us","name":"loki-prod-us","type":"loki",...}

# READ-ONLY: render the template with one set of label
# values and confirm the resulting URL returns 200.
SVC=checkout-svc
ROUTE=/v2/cart
T=1755100320
URL="http://grafana.internal:3000/explore?schemaVersion=1\
&panes=%7B%22logs%22%3A%7B%22datasource%22%3A%22loki-prod-us%22%2C%22queries%22%3A%5B%7B%22expr%22%3A%22%7Bservice%3D%5C%22${SVC}%5C%22%2Croute%3D%5C%22${ROUTE}%5C%22%2Cstatus%3D%5C%225%5C%22%7D%22%7D%5D%7D%7D\
&from=$((T-120))&to=$((T+120))"
curl -fsS -u grafana-admin:$GRAFANA_ADMIN "$URL" \
  -o /dev/null -w '%{http_code}\n'
# 200

# CONFIGURATION: reload dashboard provisioning.
sudo systemctl reload grafana-server

# READ-ONLY: in the UI, click the link from a panel point.
# Confirm the URL bar shows the expected /explore URL, the
# data source dropdown is the one you targeted, and the
# time range is the expected two-minute window.

A clean validation: the link renders in the panel, the target data source exists, the rendered URL returns 200, and the resulting Explore view shows the expected query, data source, and time range.

How it can fail

The most expensive correlation-config failure modes from real production incidents.

1. Wrong target UID. The URL references loki-prod-eu or loki (a UID that has been deleted). Grafana either opens a default data source (which may be a different cluster) or refuses to open the link. The symptom is “the link does nothing” or “the link opens against the wrong cluster”.
2. Dashboard variable used where a label is needed. The URL references ${service} and forwards it via includeVars: true. The dashboard’s service variable is a dropdown with a single value, “all”. The substitution is “all”; the LogQL {service="all", route="..."} returns nothing. The symptom is a link that opens to “no results” with no error message.
3. Label name drift between metric and URL. The metric legend exposes handler; the URL references ${__series.labels.handler}. The substitution is empty because the renamed metric uses route. The symptom is the same as above: link renders, click goes to Explore, query returns nothing.
4. Hand-authored URL with one level of JSON encoding. The panes parameter should be JSON URL-encoded into the URL; the author pasted a once-encoded JSON. Grafana parses it as malformed JSON; the link opens to a blank Explore panel. The symptom is “the link is there but the destination is empty”.
5. includeVars: false on a multi-cluster dashboard. The dashboard has ${region} set to us; the data source UID is fixed to loki-prod-us; the variable value is not forwarded to anything. The link opens to the right data source but the dashboard’s variable state is lost. The symptom is “the link opened to the wrong region”.
6. No targetBlank set on an incident dashboard. The default in Grafana 11.x is false. The link replaces the dashboard the engineer is investigating. They have to use the back button to return. The symptom is a small but persistent interruption cost during every incident.

How to troubleshoot it

The diagnostic order is “does the link render?”, “is the target data source present?”, “does the rendered URL return what is expected?”, “does the destination LogQL return lines?”.

1. Confirm the link renders. Open the dashboard in a browser, hover the panel, click the link. If nothing happens, the link is misconfigured at the panel level. Check the fieldConfig.defaults.links array via the API.
2. Confirm the target data source exists. Run /api/datasources/uid/<uid>. A 404 means the UID is stale; the data source was renamed or deleted. Replace with the correct UID.
3. Render the template by hand. Substitute the labels and the time into the URL template manually. Open the rendered URL in a private browser tab. If the rendered URL parses, the template is fine; the problem is at the destination.
4. Probe the destination LogQL. Run the resulting LogQL through the Loki proxy directly. If the proxy returns streams but the link returns nothing, the problem is in the URL encoding or the panes JSON.
5. Inspect the dashboard variable state. Open the dashboard’s variable dropdowns and confirm the values match what the link expects. A service=all is a common cause of “the link opened but the query returned nothing”.
6. Inspect Grafana’s logs. /var/log/grafana/grafana.log records every data link resolution. A failed URL parse shows up as a warning; a missing data source UID shows up as a routing error.

Security implications

• The link forwards dashboard variables. A variable that contains a customer ID, an email, or a session token is forwarded to the destination URL and lands in the destination access log. Treat dashboard variable values as values that flow to whatever the link targets.
• The link targets a data source. A link that targets a Loki data source with broader read permissions than the source Prometheus is a privilege escalation. Review the pair as a pair.
• The targetBlank: true flag opens a new tab. A link that points at an external URL (not /explore or /d/) opens a phishing risk if the URL is not reviewed. Audit external destinations; the link’s URL is the link’s authority.

Performance implications

• A wide link is a slow destination. A link that opens Loki with no time range and no status filter asks Loki to return every line for the dashboard’s window. The cost is paid in Loki CPU and browser memory.
• A link that re-renders every panel refresh. Grafana re-resolves the template on every panel render. A link that references many labels is cheap; a link that references an aggregation is not. Keep templates simple.
• A correct link is a fast lookup. A link that passes a narrow time window, a status filter, and a service label asks Loki for tens of lines. The cost is paid once and the engineer sees the answer.

Production guidance

• Pin every link by UID, not by display name. A rename of the data source should not break every pivot.
• Use the panes JSON generated by the Explore UI, not one you hand-author. The escaping is hard to get right twice.
• Set targetBlank: true for every incident-correlation link. The engineer should keep the dashboard open.
• Set includeVars deliberately. true forwards the dashboard’s variable state; false does not. The wrong choice is the one the author did not make.
• Name dashboard template variables so they cannot collide with metric labels. A variable called service that forwards service=all will silently shadow a metric label called service.
• Review the link config in code review, not in the panel editor. The link is JSON; the JSON has rules; the rules catch the common mistakes.

Verification

You should now be able to answer:

• What are the four fields of a Grafana 11.x panel-level data link, and which one controls whether the link opens in a new tab?
• What is the difference between ${__series.labels.<name>} and ${service} in a data link URL, and which is the right choice for passing a clicked-series label value?
• Why is targeting a data source by UID rather than by display name the right discipline for a pivot?
• What is the failure shape when a link’s panes JSON is hand-authored with one level of URL encoding rather than two?

Quiz