Template Variable Purpose

A production incident lands at 02:40. The on-call engineer opens the cluster overview dashboard, glances at the wrong panel, and pages the wrong team. The post-incident review asks the obvious question: why does a dashboard that is meant to show the cluster only show one of the twelve running? The answer is that someone edited the URL the day before and forgot. The variable dropdown says eu-west-1; the URL says var-cluster=eu-west-1; every panel is bound to the variable; the page is technically correct and operationally misleading.

This lesson is about the thing at the top of every reusable dashboard: the template variable. A template variable is a named, label-driven, runtime parameter. It looks like a dropdown. It behaves like a query parameter. It is encoded into the URL. It is shared by every panel on the dashboard.

What it is

A template variable in Grafana 11 is a dashboard-scoped, named parameter whose value is selected by the viewer at view time and substituted into panel queries, panel titles, and panel descriptions. The substitution is performed by the Grafana server before the query is sent to the data source.

A dashboard can declare any number of template variables. Each one has a name, a type (query, custom, constant, datasource, textbox, interval, or system), a definition, a refresh policy (on dashboard load, on time-range change, never), a current value (which the viewer can override), and an optional list of display and hide controls.

The simplest model is: a variable is a labelled query that becomes a dropdown, and the dropdown changes the panel query at view time.

Why a sysadmin cares

Five operational pains map directly to the decision “use a variable” rather than “duplicate the dashboard”:

1. One dashboard, N views. A cluster overview that shows eu-west-1 for the EU engineer and us-east-2 for the US engineer is one URL away from being two views. Without variables, the answer is N dashboards that drift apart.
2. Bookmarkable, shareable state. The URL encodes every variable value plus the time range. A Slack link to a specific panel of a specific service at a specific time is a single URL. Without variables, the recipient must repeat the filter.
3. Consistent filtering across panels. A service-name variable on the dashboard means every panel filters on that service name. A copy-paste mistake on a single panel is impossible.
4. Driven by the data, not by the author. A query-type variable reads label_values(up, cluster) from Prometheus every dashboard load. New clusters appear in the dropdown the moment they are scraped. The dashboard author does not edit a list.
5. Cheap to fix a typo. The same variable name is used everywhere. Renaming the metric or the label is a single edit at the dashboard root.

The cost is paid in query blast and variable-query latency (covered in the lesson on query cost) and in the discipline of keeping variable chains short.

How it works

The lifecycle of a variable on dashboard load runs through six stages:

   dashboard load
        |
        v
   +-------------------+   +-------------------+
   |  resolve chain    |-->|  for each var,    |
   |  order by         |   |  run the variable |
   |  dependencies     |   |  query against    |
   |                   |   |  the bound data   |
   +-------------------+   |  source           |
                           +-------------------+
                                    |
                                    v
                           +-------------------+
                           |  merge values     |
                           |  with the         |
                           |  "all" / "any"    |
                           |  pseudo-value     |
                           +-------------------+
                                    |
                                    v
                           +-------------------+
                           |  build the URL    |
                           |  ?var-x=a         |
                           |  &var-y=b         |
                           |  &from=...        |
                           |  &to=...          |
                           +-------------------+
                                    |
                                    v
                           +-------------------+
                           |  every panel      |
                           |  query is re-run  |
                           |  with $x and $y   |
                           |  substituted      |
                           +-------------------+
                                    |
                                    v
                                 pixels

Two facts about the model that shape the design:

• Variables are resolved before panels run. A variable query is one of the first things Grafana does on load. If the variable query is slow, the dashboard is slow. The data sources backing variable queries must be fast; an __metrics Prometheus is the usual answer for production.
• The URL is the source of truth for “current values”. A viewer can land on a dashboard from any link; the variables in the URL are what the panels bind to. Selecting a value from a dropdown writes the URL; refreshing the page re-reads it.

The chaining pattern

A variable can reference another variable. The pattern looks like:

  $cluster        (no dependency; the root)
       |
       v
  $namespace      (depends on $cluster)
       |
       v
  $service        (depends on $namespace)
       |
       v
  $instance       (depends on $service)

Each variable’s query filters on the previous variable’s current value. The dependency is declared by writing $cluster inside the $namespace query. The chain is topological; a cycle depends on the order Grafana loads them in and breaks silently.

A correct chain means the dropdown narrows as the operator drills down. A wrong chain (say $namespace before $cluster) means the namespace list is unfiltered, the dropdown is unbounded, and the dashboard takes minutes to load.

Ad-hoc versus curated variables

Grafana 11 distinguishes two scopes:

  +------------------------+   +------------------------+
  |  Curated variable      |   |  Ad-hoc variable       |
  |  (dashboard-level)     |   |  (panel-scoped)        |
  +------------------------+   +------------------------+
  |  Declared in           |   |  Declared at query     |
  |  Settings > Variables  |   |  time via the          |
  |                        |   |  variable edit menu    |
  |  Visible to every      |   |  Visible only in the   |
  |  panel on dashboard    |   |  panel that uses it    |
  |                        |   |                        |
  |  URL-encoded with      |   |  NOT URL-encoded;      |
  |  var-name=value        |   |  ephemeral             |
  |                        |   |                        |
  |  Refreshing chain      |   |  Re-runs on every      |
  |  cost is amortised     |   |  panel refresh;        |
  |  across panels         |   |  cost is per-panel     |
  +------------------------+   +------------------------+

The default and the production default is curated. Ad-hoc variables are a panel-edit convenience that turn into operational debt the moment another team member opens the dashboard. Promote an ad-hoc variable to a curated one the moment it is reused across panels.

How to configure it

A production-shaped templating.list for a Kubernetes service-overview dashboard looks like:

{
  "templating": {
    "list": [
      {
        "name":     "cluster",
        "type":     "query",
        "datasource": { "type": "prometheus", "uid": "prom-prod" },
        "query":    "label_values(up{job=\"kube-state\"}, cluster)",
        "refresh":  1,
        "includeAll": true,
        "multi":      true,
        "allValue":   ".*",
        "sort":       1,
        "current":    { "selected": true, "text": "All", "value": "$__all" }
      },
      {
        "name":     "namespace",
        "type":     "query",
        "datasource": { "type": "prometheus", "uid": "prom-prod" },
        "query":    "label_values(up{job=\"kube-state\",cluster=~\"$cluster\"}, namespace)",
        "refresh":  1,
        "includeAll": true,
        "multi":      true,
        "allValue":   ".*",
        "sort":       1,
        "current":    { "selected": true, "text": "All", "value": "$__all" }
      },
      {
        "name":      "service",
        "type":      "query",
        "datasource": { "type": "prometheus", "uid": "prom-prod" },
        "query":     "label_values(up{job=\"kube-state\",cluster=~\"$cluster\",namespace=~\"$namespace\"}, service)",
        "refresh":   1,
        "includeAll": true,
        "multi":      true,
        "allValue":   ".*",
        "sort":       1,
        "current":    { "selected": true, "text": "All", "value": "$__all" }
      }
    ]
  }
}

The three variables form a chain: cluster is the root, namespace references cluster, service references both. Three details are production defaults:

• refresh: 1 — refresh on dashboard load. Cost is amortised across every panel that reads the variable.
• includeAll: true plus allValue: ".*" — the All selection expands to a regex .* that bypasses filtering. This is how the dropdown says “do not filter on this dimension”.
• multi: true — selecting multiple clusters adds them to the URL as a comma-separated value and the regex =~ matches them.

How to validate it

Three checks confirm the chain is wired correctly.

Severity: READ-ONLY.

# 1. The dashboard JSON has the variables in the order
#    the dependency graph expects. Use jq to dump and read.
curl -s -u admin:$ADMIN \
  https://grafana.example.com/api/dashboards/uid/svc-overview \
  | jq '.dashboard.templating.list[] | {name, refresh, query}'

{
  "name":    "cluster",
  "refresh": 1,
  "query":   "label_values(up{job=\"kube-state\"}, cluster)"
}
{
  "name":    "namespace",
  "refresh": 1,
  "query":   "label_values(up{job=\"kube-state\",cluster=~\"$cluster\"}, namespace)"
}
{
  "name":    "service",
  "refresh": 1,
  "query":   "label_values(up{job=\"kube-state\",cluster=~\"$cluster\",namespace=~\"$namespace\"}, service)"
}

# 2. Each variable's query runs in Prometheus and returns
#    the expected value set. The variable query should
#    never return empty in production.
curl -G -s http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=label_values(up{job="kube-state"}, cluster)' \
  --data-urlencode 'time='$(date +%s) | jq '.data.result[].metric'

{
  "cluster": "eu-west-1"
}
{
  "cluster": "eu-west-2"
}
{
  "cluster": "us-east-1"
}
{
  "cluster": "us-east-2"
}

# 3. The URL encodes the chain. Open the dashboard with one
#    cluster pinned, copy it, and confirm var-cluster is in
#    the URL string.
xdg-open "https://grafana.example.com/d/svc-overview?var-cluster=eu-west-1"

The URL is the source of truth for the next person who opens the dashboard. If a value is missing from the URL, the dashboard defaults to All.

How it can fail

Five failure shapes appear repeatedly in dashboards that rely on variables:

1. Stale variable list. The variable query returns label_values(up, cluster) and up is being scraped, but the new cluster has not yet scraped. The variable dropdown does not show the new cluster. Symptom: an operator creates a new cluster, opens the dashboard, and the cluster is missing.
2. Unbounded variable query. A query variable that selects on label_values(... , cluster) without a metric selector that scopes to known scrape targets returns every label value Prometheus has ever seen, including relabeling artefacts. The dropdown becomes unscrollable. Symptom: the dashboard takes seconds to open and the dropdown is huge.
3. Cycle in the dependency graph. $a references $b and $b references $a. Grafana logs TemplatingSrv: circular dependency detected and one of the two variables ends up empty. Symptom: a blank dropdown with no error in the panel.
4. Drifted URL state. A viewer bookmarks a dashboard with var-cluster=eu-west-1; the cluster is later renamed to eu-west-1a; the URL still pins the old name; the dropdown shows the new name but every panel query uses the old name and returns nothing.
5. Multi-value without =~. A variable is set to multi: true and the panel query uses cluster="$cluster", which only matches a single value. When the viewer selects two clusters, the panel goes blank. Symptom: selecting the second cluster blanks a panel that was working with one cluster.

How to troubleshoot it

The diagnostic order is the same every time:

1. Inspect the URL. Look at the var-name=value pairs. If the value is wrong, the panel is right to show wrong data.
2. Open the variable editor. Settings > Variables. For each variable, click Run query (the refresh icon at the bottom of the preview list). A failed variable query is the dashboard’s load-time problem.
3. Run the variable query directly against the data source. Take the JSON query field, paste it into the data source’s own query interface (Prometheus’s graph page, Loki’s LogQL), and confirm the value list. If Prometheus has the data, the variable definition is the bug.
4. Inspect the chain order. Topological-sort the query fields by \$ references. The chain must run from broadest to narrowest.
5. Disable the variable temporarily. Set the variable to hide: variable and type: custom with a single value. The dashboard renders without the chain. If it renders correctly, the variable is the bug.
6. Look at the server logs. grafana-server logs the variable resolution at debug level. Filter for TemplatingSrv to see the per-variable round-trip time.

Security implications

• Variables are URL-controlled. Anyone with the URL can pin a variable to any value the variable definition permits. A reader with view-only access cannot pin a value the variable definition has not offered; a viewer with edit access can replace the variable definition entirely. Treat dashboard edit access as equivalent to data-source read access.
• The variable query runs against the data source with the data source’s permissions. A variable that queries Loki queries it as the data-source plugin credential, not as the viewer. This is normally fine but means a viewer can drive a Loki query that the viewer would not be allowed to author in Explore.
• allValue is a regex. A query variable with allValue: ".*" and multi: true lets the viewer match every value the data source has. Against Prometheus, this is bounded by what the data source itself exposes; against a non-paged query, this can be unbounded.

Performance implications

• The dashboard load is dominated by variable queries, not by panel queries. Each variable is one round trip to the data source; with refresh: 2 (on time-range change) it re-runs on every time-range change.
• A chain of three variables is three round trips. The total load is t_chain + t_panel. The chain is the silent cost.
• includeAll and multi are not free on the data source side. A panel with cluster=~"$cluster" where $cluster is eu-west-1|eu-west-2 is a multi-match query. The panel is bounded by the regex and the matcher; a value set of 1,000 cluster names is a thousand-match regex the data source has to evaluate.
• Long chains are expensive. A chain of seven variables, each with refresh: 1, is seven calls before a single panel runs.

Production guidance

• Keep variable chains short. Three variables is a reasonable upper bound; five is a smell.
• Default to curated (dashboard-level) variables. Promote any ad-hoc variable that more than one panel uses.
• Use multi: true and includeAll: true for filtering dimensions the viewer usually wants to expand. Use multi: false for dimensions where picking multiple has no operational meaning.
• Document the chain order in the dashboard description (Settings > Description). “Top to bottom: cluster narrows to namespace narrows to service.”
• Verify the URL after every variable rename. A bookmarked URL with a stale var-name is a hidden lie.

Verification

You should now be able to answer:

• What does a template variable do at view time?
• Where in the dashboard load lifecycle do variable queries run?
• How is the URL related to the variable state?
• What is the difference between a curated and an ad-hoc variable, and which should a production dashboard use?
• Why does the chain order matter?

Quiz