Variable Validation · Observability

The on-call engineer opened the cluster overview during an outage. The cluster dropdown at the top read All. The panel for “request rate by service” was empty. They switched the dropdown from All to prod, and the panel rendered instantly. The All value had expanded to a regex .* that did not match what the dashboard’s panels expected; the panel expr had a label matcher that .* silenced.

This is what variable validation is for. A Grafana 11 dashboard has one or more variables in templating.list[]. Each variable has a value list, a regex, an All-value, a multi flag, and a refresh policy. Each panel interpolates the variable’s current value into the panel expr. The dashboard renders correctly only if every variable, in every combination, interpolates into an expr that returns the right data.

Variable validation is the per-variable and per-combination contract that the variable’s value list is bounded, the regex matches the right subset, the All-value expands to a regex the panel expr accepts, and every combination of multi-value selections produces a non-empty result.

What it is

A variable is a dropdown (or a hidden field) at the top of the dashboard whose current value is interpolated into every panel expr that references it. The variable’s definition lives in templating.list[]:

  +----------------+----------------------------------------------+
  | Field          | Purpose                                      |
  +----------------+----------------------------------------------+
  |  name          | The identifier; $name in panel exprs         |
  +----------------+----------------------------------------------+
  |  type          | query / custom / constant / datasource /    |
  |                | interval / textbox / system                  |
  +----------------+----------------------------------------------+
  |  query         | The expression (query kind) or the literal   |
  |                | list (custom kind) or the type (datasource  |
  |                | kind)                                        |
  +----------------+----------------------------------------------+
  |  regex         | Optional filter applied to the value list;  |
  |                | values not matching the regex are hidden    |
  +----------------+----------------------------------------------+
  |  refresh       | 0 (never) / 1 (on load) / 2 (on time change) |
  +----------------+----------------------------------------------+
  |  includeAll    | Show an "All" pseudo-value                  |
  +----------------+----------------------------------------------+
  |  allValue      | The string the "All" value expands to       |
  +----------------+----------------------------------------------+
  |  multi         | Allow more than one selection               |
  +----------------+----------------------------------------------+
  |  current       | Persisted selection; survives reloads       |
  +----------------+----------------------------------------------+
  |  hide          | 0 (visible) / variable / label (hidden)    |
  +----------------+----------------------------------------------+
  |  skipUrlSync   | Do not write the variable to the URL        |
  +----------------+----------------------------------------------+
  |  sort          | 0 (none) / 1 (asc) / 2 (desc) / 3 (alpha)  |
  +----------------+----------------------------------------------+

Variable validation has four checks:

Value list shape. The variable’s resolved value list is bounded (typically < 200 values) and contains only values the panel exprs can match against.
Regex correctness. If the variable has a regex field, the regex matches the values the panel exprs expect; it does not silently filter out every value.
All-value expansion. If includeAll: true, the allValue field expands to a string the panel exprs accept. The default empty string breaks regex matchers; .* is the canonical expansion.
Multi-value handling. If multi: true, every panel expr that uses the variable uses =~ (regex matcher) rather than = (equality matcher); equality matcher matches exactly one value.

A variable that fails check 1 is unbounded. A variable that fails check 2 is filtered into emptiness. A variable that fails check 3 has a broken All value. A variable that fails check 4 has panels that go blank whenever the viewer selects more than one value.

Why a sysadmin cares

Three operational pains map directly to variable validation:

The unbounded variable. A query variable without a regex on a label that has 5,000 values produces 5,000 options in the dropdown. The dashboard does not break, but the dropdown becomes unusable; the viewer cannot scroll to find the value they want.
The empty regex. A variable with regex: "^prod$" when the data source returns prod, production, and production-east produces a dropdown with only prod. The viewer thinks there are three production environments; the dashboard shows one. The mismatch is silent.
The All-value that does not match. A variable with includeAll: true and allValue: "" (the default) interpolated into cluster=~"$cluster" produces cluster=~"", which matches nothing. The All selection produces an empty dashboard.

The wrong shape shows up as a dashboard that renders for one value but breaks for another, or as a dashboard that renders correctly when the variable is at the default but breaks when the viewer picks a different value.

The regex boundary

The regex field is the most failure-prone field in a variable definition. It is a Go regex applied to the value list before the dropdown is shown. The failure shapes:

  +----------------+----------------------------------------+
  |  regex         |  Effect on a list prod,production,dev |
  +----------------+----------------------------------------+
  |  (empty)       |  All three values in the dropdown      |
  |  "^prod$"      |  Only prod; production is hidden       |
  |  "prod.*"      |  prod and production; dev is hidden    |
  |  ".*"          |  All three values                      |
  |  "[invalid"    |  Compile error; variable fails to load |
  +----------------+----------------------------------------+

The fifth row is the silent-failure shape: the regex does not compile, the variable fails to load, the dashboard shows a “Templating init failed” badge. The fix is a CI check that compiles every variable’s regex against a small fixture list.

How it works

The variable-validation pipeline:

  dashboard JSON
        |
        v
  +-------------------------+
  |  parse templating.list[]|   jq '.templating.list[]'
  +-------------------------+
        |
        v
  +-------------------------+
  |  for each variable:     |
  |  resolve value list     |   label_values(...) for query kind;
  |                         |   split query field for custom kind
  +-------------------------+
        |
        v
  +-------------------------+
  |  apply regex to value   |   re2 match per value; count survivors
  |  list; count survivors  |
  +-------------------------+
        |
        v
  +-------------------------+
  |  verify All-value       |   for includeAll: true, confirm
  |  expansion              |   allValue matches panel expr regex
  +-------------------------+
        |
        v
  +-------------------------+
  |  walk panel exprs       |   for each $var reference, confirm
  |                         |   the variable uses =~ or = as intended
  +-------------------------+
        |
        v
  +-------------------------+
  |  cross-product check    |   for multi: true variables,
  |                         |   confirm the panel expr uses =~
  +-------------------------+
        |
        v
  +-------------------------+
  |  report                 |   list of empty regexes, broken
  |                         |   All-value expansions, multi/= mismatches
  +-------------------------+

Under the hood

The TemplatingSrv module in Grafana 11 resolves each variable in templating.list[] in two phases:

List resolution. For a query variable, the resolver calls the variable’s data source with the variable’s query expression and converts the response to a list of {text, value} pairs. For a custom variable, the resolver splits the query field on commas. For a constant variable, the resolver wraps the query field in a single {text, value} pair.
List filtering. The resolver applies the regex field to the value list using Go’s regexp package. Values that do not match are removed from the dropdown. A regex that fails to compile raises an error and the variable shows a “Templating init failed” badge.

The current field is then read from the URL (or the cookie) and the value is bound to $var. The panel expr interpolates $var with the bound value before the panel query runs.

The All pseudo-value, when selected, is bound to $__all. When the variable is interpolated, $__all expands to the allValue field. If allValue is empty, the interpolation produces an empty string; a regex matcher with an empty pattern matches nothing.

How to configure it

The canonical pattern: a CI script that walks every variable, resolves its value list, applies the regex, counts survivors, and fails the merge when the survivor count is unexpected.

#!/usr/bin/env bash
# scripts/check-variables.sh
# Severity: READ-ONLY against staging.
set -euo pipefail

PROM_URL=${PROM_URL:-http://prometheus-staging:9090}
MAX_VALUES=${MAX_VALUES:-200}
MIN_VALUES=${MIN_VALUES:-1}

check_query_variable() {
  local json_file=$1
  local var_name=$2
  local var_query=$3
  local var_regex=$4
  local var_all=$5
  local all_value=$6

  # Resolve the value list against the staging Prometheus.
  values=$(curl -G -s "$PROM_URL/api/v1/query" \
    --data-urlencode "query=$var_query" \
    --data-urlencode "time=$(date +%s)" \
    | jq -r '.data.result[].metric | to_entries
              | map(select(.key != "__name__"))
              | .[0].value')

  count=$(echo "$values" | wc -l)
  if (( count > MAX_VALUES )); then
    echo "FAIL: $json_file variable $var_name has $count values (max $MAX_VALUES)"
    return 1
  fi
  if (( count < MIN_VALUES )); then
    echo "FAIL: $json_file variable $var_name has $count values (min $MIN_VALUES)"
    return 1
  fi

  # Apply the regex (if present) and recount.
  if [[ -n "$var_regex" ]]; then
    survivors=$(echo "$values" | grep -E "$var_regex" | wc -l)
    if (( survivors == 0 )); then
      echo "FAIL: $json_file variable $var_name regex $var_regex matches nothing"
      return 1
    fi
  fi

  # Verify All-value expansion. An empty allValue with a
  # regex matcher is the canonical All-value failure shape.
  if [[ "$var_all" == "true" && -z "$all_value" ]]; then
    echo "WARN: $json_file variable $var_name has includeAll but empty allValue"
  fi
}

for json_file in dashboards/*.json; do
  jq -c '.templating.list[]? | select(.type == "query")
         | {name, query, regex, includeAll, allValue}' "$json_file" \
    | while read -r var; do
        name=$(echo "$var" | jq -r .name)
        query=$(echo "$var" | jq -r .query)
        regex=$(echo "$var" | jq -r .regex // "")
        includeAll=$(echo "$var" | jq -r .includeAll)
        allValue=$(echo "$var" | jq -r .allValue // "")
        check_query_variable "$json_file" "$name" "$query" \
          "$regex" "$includeAll" "$allValue"
      done
done

A minimal templating.list that the script validates:

{
  "templating": {
    "list": [
      {
        "name":       "cluster",
        "label":      "Cluster",
        "type":       "query",
        "datasource": { "type": "prometheus", "uid": "prom-prod" },
        "query":      "label_values(up{job=\"kube-state\"}, cluster)",
        "regex":      "",
        "refresh":    1,
        "includeAll": true,
        "allValue":   ".*",
        "multi":      true,
        "sort":       1,
        "current":    { "selected": true, "text": "All",
                         "value": "$__all" }
      }
    ]
  }
}

A panel expr that uses the variable correctly:

  sum by (service) (rate(http_requests_total{cluster=~"$cluster"}[5m]))

The =~"$cluster" is the regex matcher. With multi: true and $cluster = "prod,staging", the interpolation produces cluster=~"prod,staging", which matches series whose cluster label is either prod or staging. With All selected, the interpolation produces cluster=~".*", which matches every cluster.

How to validate it

Five checks confirm the variable discipline is live.

Severity: READ-ONLY.

# 1. Every variable in the dashboard declares a kind and a
#    query (or literal) that resolves.
curl -s -u admin:$ADMIN \
  https://grafana.example.com/api/dashboards/uid/svc-overview \
  | jq '.dashboard.templating.list[]
        | {name, type, query, regex, refresh, includeAll, allValue}'

{
  "name":        "cluster",
  "type":        "query",
  "query":       "label_values(up{job=\"kube-state\"}, cluster)",
  "regex":       "",
  "refresh":     1,
  "includeAll":  true,
  "allValue":    ".*"
}

# 2. The value list resolves against the live data source
#    and contains only the expected values.
curl -G -s http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=label_values(up{job="kube-state"}, cluster)' \
  --data-urlencode 'time='$(date +%s) \
  | jq '.data.result | map(.metric.cluster)'
# ["prod","staging","dev"]

# 3. The regex (if any) matches at least one value in the
#    resolved list. A regex that matches nothing is the
#    canonical empty-variable failure.
curl -G -s http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=label_values(up{job="kube-state"}, cluster)' \
  --data-urlencode 'time='$(date +%s) \
  | jq -r '.data.result[].metric.cluster' \
  | grep -E '^prod$' || echo "no exact match for ^prod$"

# 4. The All-value expansion is correct. includeAll: true
#    with allValue: "" is the canonical All-value failure.
curl -s -u admin:$ADMIN \
  https://grafana.example.com/api/dashboards/uid/svc-overview \
  | jq '.dashboard.templating.list[]
        | select(.includeAll == true) | {name, allValue}'

# 5. Multi-value variables use =~ in panel exprs, not =.
#    A panel expr with multi: true variable and = is the
#    canonical multi-value matcher failure.
for expr in $(curl -s -u admin:$ADMIN \
  https://grafana.example.com/api/dashboards/uid/svc-overview \
  | jq -r '.dashboard.panels[].targets[].expr'); do
  echo "$expr" | grep -E '\b\w+="\$' && echo "WARN: equality matcher on variable in: $expr"
done

How it can fail

Six failure shapes appear repeatedly with variable validation:

Regex drift. The variable’s regex field references a value pattern that no longer exists in the data source. Symptom: the dropdown is empty; the dashboard cannot render; the “Templating init failed” badge appears.
Multi-value with equality matcher. A multi: true variable interpolated into cluster="$cluster" matches exactly one value. Symptom: the panel renders one series when two are selected; the legend is missing one value.
All-value with empty expansion. A variable with includeAll: true and allValue: "" interpolated into cluster=~"$cluster" produces cluster=~"", which matches nothing. Symptom: the All selection produces an empty dashboard; switching to a specific value renders correctly.
Unbounded value list. A query variable on label_values(up, instance) returns one value per running instance; the value list grows as the fleet grows. Symptom: the dropdown becomes unusable; the repeat against the variable produces hundreds of panels.
Refresh policy mismatch. A refresh: 0 (never) variable on a label that changes frequently shows stale values. Symptom: a new cluster does not appear in the dropdown until the dashboard is reloaded.
Variable chain broken. A variable depends on another variable’s value; the parent variable’s resolution has changed. Symptom: the child variable’s value list does not match what the parent variable produced last time the dashboard was opened.

How to troubleshoot it

The diagnostic order:

Open Settings > Variables. Click Run query on the variable. The response should be a non-empty list.
Inspect the regex. Apply the regex to a fixture value list. If the regex matches nothing, the regex is wrong.
Click the dropdown. Confirm the All-value is present (when includeAll: true); confirm the multi checkbox works.
Switch the variable value. Pick a specific value, not All. The dashboard should render correctly.
Switch back to All. The dashboard should render correctly with the All-value expansion.
Pick two multi-values. The dashboard should render correctly with both selected.
Inspect the panel expr. Confirm the matcher is =~ for multi-value variables and = for single-value variables.

Security implications

Variable values are not auth-scoped. A query variable on label_values(up, cluster) returns every cluster the data source has, regardless of the viewer’s permissions. A viewer with limited cluster access still sees the full list.
Variable expansion is not escaped for LogQL/TraceQL. Grafana interpolates the variable value into the query text before sending it to the data source. A viewer who can pin a URL with a malicious value can inject LogQL/TraceQL syntax. The data source plugin is responsible for rejecting the injection.
The dashboard JSON exposes variable definitions. Treat the variable definitions as an artefact whose disclosure is bounded by the dashboard’s own permissions.

Performance implications

query variables dominate load time. Each refresh: 1 query is one round trip to the data source; each refresh: 2 query is one per time-range change.
custom and constant variables are essentially free. No data source call; no I/O.
datasource variables are a single in-memory call. Effectively free.
interval and system variables are computed locally. Effectively free.
The value list size bounds the per-refresh cost. A query variable with 200 values multiplied by 20 panels is 4,000 panel queries per refresh.

Production guidance

Bound every query variable with a regex or with a topk(N, ...) in the query itself. The value list should not grow with the fleet.
Set allValue: ".*" on every query variable with includeAll: true and a regex matcher. The default empty string is the canonical All-value failure.
Set multi: true together with =~ in panel exprs. Pair the variable flag with the matcher in the panel; an = matcher with a multi variable is a silent failure.
Set refresh: 1 on every query variable. The refresh: 2 knob is rare and operationally expensive; most variables do not need to refresh on time-range change.
Document the All-value and multi-value behaviour in the dashboard description. The viewer should know what All expands to before they click it.

Verification

You should now be able to answer:

What are the four checks that define variable validation?
Where do variables live in the dashboard JSON, and what fields does each variable declare?
How does the regex field silently filter the value list, and what is the canonical empty-variable failure shape?
Why is allValue: "" the wrong default for a query variable with a regex matcher?
What is the cost of a variable whose value list grows from 12 to 200?