Variable Interpolation

An engineer shares a Slack link to a panel pinned to var-cluster=eu-west-1. The recipient opens the link. The panel says “No data”. The engineer double-checks: the cluster exists, the panel renders correctly on their laptop, the URL they sent is byte-identical. The recipient is looking at an empty panel because the variable the URL pins is cluster, but the panel query references $clustr. The typo is in the JSON, not the URL. The Slack link is correct; the panel is wrong.

This is what variable interpolation is about: the contract between a name in the URL, a name in the JSON, and a name in the data source’s query language. The contract is exact. A typo breaks it silently.

What it is

Variable interpolation is the substitution of a variable’s current value into a string that the data source or Grafana consumes. The string can be a query expression, a panel title, a panel description, or a legend format. The substitution happens before the query is sent to the data source.

Grafana 11 recognises four forms:

• $var — the most common form. Substitutes the current value of var into the string.
• ${var} — the same substitution, with explicit braces. Useful when the variable name is followed by a character that is part of the variable name (e.g., ${env}-prod).
• ${var:regex} — substitutes the value, then applies the regex to filter which labels match. Used inside the __values and __value_string mechanism in derived queries.
• [[var]] — the legacy bracket syntax. Still supported; equivalent to $var. Some integrations and exported dashboards prefer it because it survives a round trip through tools that strip dollar signs.

The choice between $var, ${var}, and [[var]] is stylistic. The data source does not see the syntax; it sees the substituted string.

Why a sysadmin cares

Three operational pains map directly to the interpolation contract:

1. Shared state. A panel query that uses $cluster automatically renders against whichever cluster the URL pins. The same panel is correct for eu-west-1 and us-east-2. The interpolation is the contract that makes the dashboard shareable.
2. Refactor-by-rename. A label rename in Prometheus (cluster -> k8s_cluster) is a single edit at the variable query. Every panel that references the variable updates at the same time; the dashboard does not drift.
3. Default state. The URL is the default state. The panel query reads $cluster. Without a var-cluster in the URL, the variable uses its current value. The interpolation is what makes “no URL parameter” mean something well-defined.

The wrong contract shows up as silent breakage: a panel that used $clustr returns nothing; a panel that uses ${cluster}-prod returns the literal cluster-prod because the braces were missing.

Panel-query versus panel-title interpolation

A panel has two distinct places where variables are substituted:

  Panel
    |
    +-- targets[].expr        (data source query)
    |      substitution:  server-side, before the
    |      query is sent. The data source sees the
    |      substituted string; it never sees $var.
    |
    +-- title, description    (Grafana UI)
           substitution:  client-side (in the browser)
           when the panel renders. The data source is
           not involved; the substitution is purely for
           human display.

A panel title with $cluster CPU reads as eu-west-1 CPU in the browser. The data source sees only the expr. The two substitutions do not interact; a typo in the title does not affect the query, and vice versa.

The legendFormat is a third place: it is interpolated client-side against the series labels the data source returned. {{cluster}} in the legend format reads as eu-west-1 for a series whose labels include cluster: eu-west-1. This is label interpolation, not variable interpolation.

The URL as the carrier of state

When the viewer picks a value from a dropdown, Grafana writes the value into the URL as var-name=value. The next panel query that asks for $name reads from the variable service, which has already parsed the URL. The URL is the bus; the variable service is the receiver; the panel query is the consumer.

A panel query that says cluster=~"$cluster" does not read the URL directly. It asks the variable service for the current value of cluster. The variable service reads the URL once per dashboard load and serves every consumer from the resolved map.

How it works

The substitution pipeline:

  panel JSON (expr: cluster=~"$cluster")
        |
        v
  +--------------------+   +--------------------+
  |  TemplatingSrv     |-->|  read URL          |
  |  resolve($cluster) |   |  parse var-cluster |
  +--------------------+   +--------------------+
        |
        v
  +--------------------+
  |  substitute the    |
  |  resolved value    |
  |  into the string   |
  +--------------------+
        |
        v
  cluster=~"eu-west-1"
        |
        v
  +--------------------+
  |  send to data      |
  |  source plugin     |
  +--------------------+
        |
        v
  PromQL / LogQL / TraceQL parser

Three details about the pipeline that matter in production:

• Substitution happens before parsing. The data source parser sees cluster=~"eu-west-1", not cluster=~"$cluster". A data source error message will reference the substituted value.
• $var is greedy. The parser reads the longest identifier it can after the $. $envprod is the variable envprod, not env followed by prod. Use ${env}prod to disambiguate.
• Multi-value substitution is comma-list, not regex. cluster=~"$cluster" where $cluster is eu-west-1, us-east-2 becomes cluster=~"eu-west-1, us-east-2", which is a single value with a comma, not a regex. Wrap in a regex transformation: cluster=~"(eu-west-1| us-east-2)" via $\{cluster:regex\}, or use the allValue convention.

How to configure it

The canonical place to see interpolation is a panel query that filters on a variable, plus a panel title that names the filter:

{
  "panels": [
    {
      "type":  "timeseries",
      "title": "Request rate [$cluster / $namespace]",
      "datasource": { "type": "prometheus", "uid": "prom-prod" },
      "targets": [
        {
          "refId":      "A",
          "datasource": { "type": "prometheus", "uid": "prom-prod" },
          "expr":       "sum by (service) (rate(http_requests_total{cluster=~\"$cluster\", namespace=~\"$namespace\"}[5m]))",
          "legendFormat": "{{service}}"
        }
      ]
    }
  ]
}

Three details to notice:

• The expr references both variables. The data source receives the following expression when both are at All:

  sum by (service) (rate(http_requests_total{cluster=~"eu-west-1|.*", namespace=~"kube-system|.*"}[5m]))

• The title reads Request rate [eu-west-1 / kube-system] in the browser. The square brackets are literal; the $cluster and $namespace are interpolated client-side.

• The legendFormat uses {{service}}, which is label interpolation, not variable interpolation. The data source returns series with a service label; the browser renders the legend from those labels.

For a Loki query that filters by a textbox variable, the pattern is the same:

{
  "expr": "{cluster=~\"$cluster\"} |= \"$search\"",
  "refId": "A"
}

The search textbox value is interpolated as a raw string. A viewer who types error|warn is asking for the literal string error|warn, which LogQL’s pipe-equals treats as OR. A viewer who types a regex (e.g., .*timeout.*) gets the regex interpretation. The interpolation is verbatim.

How to validate it

Three checks: one on the wire, one in the URL, one in the panel.

Severity: READ-ONLY.

# 1. Run the same expression the panel would run, with
#    the variable substituted by hand. This catches
#    typo-class bugs (e.g., $clustr).
curl -G -s http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=sum by (service) (rate(http_requests_total{cluster=~"eu-west-1",namespace=~"kube-system"}[5m]))' \
  --data-urlencode 'time='$(date +%s) \
  | jq '.data.result | length'
# 12

# 2. Decode the URL back into variable state. Use jq
#    against the dashboard URL, or just read it.
echo "https://grafana.example.com/d/svc-overview?var-cluster=eu-west-1&var-namespace=kube-system&from=now-1h&to=now"
# Three variable pin operations + a time range.

# 3. Inside the panel, open Inspect > Data. The
#    "Query" tab shows the substituted expression
#    the data source received, not the raw $var form.
#    This is the ground truth for what Grafana sent.
xdg-open "https://grafana.example.com/d/svc-overview?var-cluster=eu-west-1&var-namespace=kube-system"

The Inspect > Data view shows the substituted string the plugin forwarded. It is the only place in Grafana that shows the post-substitution query.

How it can fail

Six failure shapes appear repeatedly with interpolation:

1. Typo in the variable name. A panel query says $clustr (sic). The variable service does not have a variable called clustr. Grafana substitutes an empty string. The panel becomes cluster=~"", which matches nothing. Symptom: the panel is empty, no error.
2. Missing braces around an alphanumeric suffix. A panel title says cluster=$clusterprod. The variable service substitutes clusterprod (an empty string if the variable does not exist). Symptom: the title reads cluster= with no value, or cluster= followed by an unwanted suffix.
3. Multi-value substitution without =~. A panel query says cluster="$cluster" with multi: true. The data source receives cluster="eu-west-1,us-east-2", which is a single value with a comma. Symptom: one series matches; the rest do not.
4. $var in a panel description, not title. The description is markdown and is rendered by the browser; $cluster is interpolated only if the description template explicitly supports it. Symptom: the description shows the literal $cluster text.
5. Literal dollar sign. A panel title with a literal dollar sign (e.g., Cost in $USD) is interpreted as a variable reference. Symptom: the title reads Cost in followed by an empty string.
6. Legacy [[var]] in a copy-pasted dashboard. Grafana supports both $var and [[var]]; some exported dashboards mix them. Symptom: visual inconsistency in the JSON; no runtime error.

How to troubleshoot it

The diagnostic order:

1. Open Inspect > Data on the failing panel. The “Query” tab shows the substituted expression. If the expression has ="" where a value should be, the variable resolved to an empty string; the variable definition is the bug.
2. Decode the URL. ?var-cluster= means the variable is pinned to the empty string. ?var-cluster=$__all means the variable is pinned to All. The panel renders whatever the URL pins.
3. Run the substituted expression against the data source directly. Take the expression from Inspect > Data, paste it into the data source’s own query interface, and confirm the result. The data source knows the truth; Grafana is just the messenger.
4. Switch to Explore with the same expression. Explore has the same interpolation logic; if a panel fails and Explore succeeds with the same $cluster pin, the panel configuration is the bug (visualisation, not interpolation).
5. Check the variable name in JSON. jq '.dashboard.templating.list[].name' against the dashboard. A variable named Cluster (capital C) is different from cluster; variable names are case-sensitive.

Security implications

• Interpolation is server-side. A variable value is substituted by the Grafana server before the query reaches the data source plugin. The plugin receives a string; it does not receive a template that the data source has to expand. This is the principal defence against a textbox value containing PromQL injection.
• ${var:regex} is the operator’s lever. A variable value fed through ${var:regex} is matched against a regex the author wrote; the viewer cannot escape the regex to inject arbitrary PromQL. The author controls the regex; the viewer controls the input.
• $var in panel titles is purely client-side. It cannot leak data the data source has not already returned; the worst case is a misleading title.

Performance implications

• Interpolation is O(length of string). It is not a performance concern on its own.
• The cost is paid by the data source. A panel query with $cluster resolves to a regex that the data source must evaluate against every series in the time range. The cost is bounded by the value set the variable returned, not by the substitution itself.
• A refresh: 1 variable re-runs every dashboard load. A refresh: 2 variable re-runs every time-range change. Either way, the substitution cost is amortised across panels that share the variable.

Production guidance

• Standardise on one syntax. Pick $var for new work and migrate old [[var]] as you touch the dashboards. The two are equivalent; the inconsistency is the cost.
• Use ${var:regex} for any $var that interpolates into a regex matcher (=~). Multi-value selections depend on the regex wrapping the variable service provides.
• Use legendFormat for label interpolation; use $var for variable interpolation. Mixing them produces a legend that does not change when the variable changes.
• Document the variable names in the dashboard description. The cost of a typo is invisible; the cost of documentation is one paragraph.

Verification

You should now be able to answer:

• What is the difference between $var and ${var} in a panel query?
• Where in the rendering pipeline is the substitution performed for panel queries versus panel titles?
• How does a multi-value variable get into a regex matcher?
• What does ?var-cluster=eu-west-1&from=now-1h mean to a panel that references $cluster?
• What is the failure shape of a typo in a variable name?

Quiz