Variable Query Cost

The dashboard load starts. Three seconds pass. Five. Eight. The on-call engineer hits reload; the same wait. The panels, when they appear, render in under 200 milliseconds each. The variable dropdowns at the top say “loading” for the first five seconds and only then resolve. The post-incident review asks the obvious question: why is the dashboard slow when every panel is fast?

The answer is that variable queries are not free, they dominate the load, and the dashboard author never measured them. A dashboard with five variables and thirty panels is thirty-five round trips on every load. A chained-variable storm pushes that number higher; a regex refresh cost on a wide label set pushes it higher still. This lesson is about the cost shape and how to control it.

What it is

Variable query cost is the round-trip count and latency that variable queries contribute to a dashboard load. It is a separate cost from panel-query cost, because variable queries run before panels and block the panel queries until they resolve.

The cost has three dimensions:

  +-----------------+   +-----------------+   +------------------+
  |  Variable       |   |  Chain          |   |  Refresh policy  |
  |  count          |   |  resolution     |   |                  |
  +-----------------+   +-----------------+   +------------------+
  |  N query        |   |  topological    |   |  refresh: 1      |
  |  variables =    |   |  order, parent  |   |  re-runs on load |
  |  N round trips  |   |  blocks child   |   |                  |
  +-----------------+   +-----------------+   +------------------+
                                                   | refresh: 2      |
                                                   | re-runs on      |
                                                   | time-range      |
                                                   | change          |
                                                   +------------------+

The cost is the sum of: (N round trips for variables) plus (M round trips for panels) plus (chain-induced repeats) plus (regex refresh cost on wide label sets).

Why a sysadmin cares

Three operational pains map directly to variable-query cost:

1. Slow dashboards are usually variable-bound, not panel-bound. A panel that takes 200 ms is fine. A variable query that takes 2 seconds for ten chain resolutions is a 20-second load. The fix is not at the panel; it is at the variable.
2. The N-by-M blast is hidden in plain sight. A dashboard with five variables of ten values each and three panels is 150 backend requests per refresh. The dashboard JSON does not show this number; only the wire does.
3. Variable state in cookies survives reloads. A viewer who picked a single cluster yesterday sees the same cluster today. The cookie is the source of truth for tab state; the URL is the source of truth for shared state.

The wrong shape shows up as a dashboard that worked in staging (one cluster, three services) and times out in production (twelve clusters, eighty services).

The N-by-M query blast

The per-refresh request count is:

  total requests
      = N_variables + sum(panel_requests per combination)

  per combination
      = M_panels  x  combinations(N_variables)

  combinations(N_variables)
      = product(value_counts)
        e.g., 5 vars at 10 values each = 10^5 = 100,000

For the dashboard described above:

• 5 variables -> 5 round trips for the variables themselves.
• 3 panels at 100,000 combinations -> 300,000 round trips if every panel runs at every combination.

In practice, Grafana parallelises panel queries and many combinations produce zero series, but the blast is real. A topk(10, ...) panel reduces the combinations to the top-10 series, not the cross-product.

The regex refresh cost

A query variable whose query field is label_values(some_metric, some_label) returns every distinct label value Prometheus knows. If some_label has 100,000 distinct values, the variable returns 100,000 values. The variable’s regex narrows the list; the cost is paid at the data source, before the regex.

A regex on the variable is client-side; it does not reduce the data-source cost. The data source still returns 100,000 values. The narrow-by-regex pattern is a UX improvement, not a performance optimisation.

The chained-variable storm

A chain of five variables, each refresh: 1, is five round trips in topological order. The chain is serial; a child cannot start until its parent has resolved. A chain that takes 200 ms per variable takes one second for the chain alone.

A chain that uses =~"$parent" and the parent is All adds another round trip: the child variable queries against an unbounded label set. The chain storm is the failure shape where each variable expands to the full label list and the next variable queries against that full list.

The variable-cookie state

Grafana persists variable state in a cookie scoped to the Grafana domain. The cookie contains:

• the dashboard UID;
• the current value of every curated variable that is not skipUrlSync: true;
• the time-range picker;
• the active data source theme.

A viewer who reloads the page reads the cookie first; the URL overrides only the variables it pins. A viewer who opens the dashboard in an incognito window has no cookie; the dashboard uses the variable’s current default.

The cookie is per-browser, not per-dashboard. A viewer who opens two tabs to the same dashboard sees the same variable state in both. A viewer who opens two tabs to two different dashboards sees two distinct state sets because the cookie is keyed by dashboard UID.

How it works

The variable-query lifecycle on dashboard load:

  dashboard load
        |
        v
  +-------------------+
  |  parse URL        |
  |  parse cookie     |
  +-------------------+
        |
        v
  +-------------------+
  |  topological      |
  |  sort variables   |
  |  by dependency    |
  +-------------------+
        |
        v
  +-------------------+    +-------------------+
  |  resolve var 1    |--> |  data source call |
  +-------------------+    +-------------------+
        |
        v
  +-------------------+    +-------------------+
  |  resolve var 2    |--> |  data source call |
  |  (depends on 1)   |    |  (parent sub)     |
  +-------------------+    +-------------------+
        |
        v
  ...
        |
        v
  +-------------------+    +-------------------+
  |  apply regex      |    |  client-side      |
  |  filters          |    |  JavaScript       |
  +-------------------+    +-------------------+
        |
        v
  +-------------------+
  |  run panel queries|
  |  with substituted |
  |  variables        |
  +-------------------+

Three details to notice:

• The chain is serial. A child variable cannot start until its parent has resolved. A chain of five variables is at least five round trips in series.
• The regex is client-side. It runs in the browser after the data source has returned. It does not reduce the data-source cost.
• Panels run after variables. Every panel query is blocked until the chain has resolved. A variable query that takes two seconds is a two-second delay before any panel runs.

How to configure it

Three knobs reduce variable-query cost:

{
  "name":        "cluster",
  "type":        "query",
  "datasource":  { "type": "prometheus", "uid": "prom-meta" },
  "query":       "label_values(kube_cluster_info, cluster)",
  "refresh":     1,
  "includeAll":  true,
  "allValue":    ".*",
  "multi":       false
}

Three configurations to compare:

// 1. The cheap chain: a recording-rule-based label_values
//    The recording rule is computed once per scrape; the
//    variable query is a label_values against the rule's
//    metric and returns in milliseconds.
{
  "query": "label_values(kube_cluster_info, cluster)"
}

// 2. The expensive chain: an unbounded regex on a wide label
//    The variable query runs against a wide label set; the
//    regex narrows the dropdown but the cost is paid at
//    the data source.
{
  "query": "label_values(up{job!=\"\"}, instance)",
  "regex": "/^prod-.*$/"
}

// 3. The bounded chain: skipUrlSync on a per-tab variable
//    The variable is not URL-pinned; the cookie holds its
//    state; reloads preserve it.
{
  "name":          "compare_namespace",
  "skipUrlSync":   true,
  "query":         "label_values(up{job=\"kube-state\"}, namespace)"
}

For a Prometheus-side optimisation, a recording rule that precomputes the value list:

groups:
  - name: variable_value_lists
    interval: 1m
    rules:
      - record: kube_cluster_info
        expr: |
          label_replace(
            label_replace(
              label_replace(
                label_replace(
                  up{job="kube-state"},
                  "cluster", "$1", "instance", "(.*)-.*"
                ),
                "region", "$1", "instance", "(.*)-.*"
              ),
              "environment", "$1", "instance", "(.*)-.*"
            ),
            "version", "v1.0.0", "", ""
          )

The recording rule runs once per scrape interval. The label_values variable query against kube_cluster_info returns in tens of milliseconds because the rule has precomputed the label set.

How to validate it

Three checks confirm the cost shape.

Severity: READ-ONLY.

# 1. The Grafana server exposes per-variable resolve
#    time. Hit /metrics and grep for variable metrics.
curl -s -u admin:$ADMIN http://grafana:3000/metrics \
  | grep -E '^grafana_templating_variable_resolve_time_seconds_count'

grafana_templating_variable_resolve_time_seconds_count{
    dashboard_uid="k8s-svc",
    name="cluster"}  142
grafana_templating_variable_resolve_time_seconds_count{
    dashboard_uid="k8s-svc",
    name="namespace"} 142
grafana_templating_variable_resolve_time_seconds_count{
    dashboard_uid="k8s-svc",
    name="service"} 142

# 2. The Prometheus side exposes the data-source cost
#    of each variable query. Run the same query and
#    measure latency.
time curl -G -s http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=label_values(up{job="kube-state"}, cluster)' \
  --data-urlencode 'time='$(date +%s) > /dev/null
# real    0m0.038s

# 3. Inspect the cookie state in the browser DevTools.
#    Application > Cookies > grafana_session. The
#    variable state is encoded into the cookie payload.
xdg-open "https://grafana.example.com/d/k8s-svc?var-cluster=prod"
# DevTools > Application > Cookies > click grafana_session
# JSON contains var-cluster: prod and the time range.

The three checks together give the cost shape: variable resolve time on the Grafana server, data-source latency at Prometheus, and the cookie state on the client.

How it can fail

Five failure shapes appear repeatedly with variable-query cost:

1. Wide-label label_values. A variable with label_values(up, instance) where instance has 50,000 distinct values. The variable returns 50,000 values; the regex narrows the dropdown but the data source still returns all of them. Symptom: dashboard load takes minutes.
2. Chain storm on All. Every variable in the chain queries against =~".*" when the parent is All. Each child returns the full label set. Symptom: a five-second chain when one cluster is picked, a thirty-second chain when All is picked.
3. refresh: 2 on a chain. A refresh: 2 on $cluster re-runs the chain on every time-range change. Symptom: long-range dashboards take forever to load because the chain re-resolves.
4. Recurring refresh policy. A variable with refresh: 2 plus multi: true plus allValue: ".*" plus skipUrlSync: false produces a query the data source evaluates against every series on every refresh. Symptom: the data source CPU is dominated by variable queries, not panel queries.
5. Cookie state pinned to a stale cluster. A viewer opens a dashboard, picks prod-eu, bookmarks the URL, closes the browser. The cluster is renamed to prod-eu-1. The bookmarked URL still pins prod-eu. The cookie also persists prod-eu for that dashboard UID. Symptom: the dashboard renders empty for that viewer, even after re-login.

How to troubleshoot it

The diagnostic order:

1. Open Inspect > Data on the failing panel. The “Stats” tab shows the total data-source latency for that panel. A panel that took 200 ms is not the problem.
2. Hit /metrics on the Grafana server. Filter for grafana_templating_variable_resolve_time_seconds. The histogram bucket with the highest count tells you which variable is slow.
3. Run the variable query directly. Take the query field, paste it into Prometheus’s /graph, and time the result. A variable query that takes five seconds is the slow dashboard’s cause.
4. Disable the chain. Set each variable to hide: variable and type: custom with a single value. The dashboard renders without the chain. If it renders correctly, the chain is the bug.
5. Inspect the cookie. Application > Cookies > grafana_session. A cookie with stale variable state is the silent failure shape; clear the cookie and reload.

Security implications

• Cookie state is per-browser, not per-session. A viewer who logs out and logs back in keeps the cookie state. A viewer who clears the cookie loses the state. Treat the cookie as a UX cache, not as an authentication boundary.
• skipUrlSync: true is a privacy knob. A variable marked skipUrlSync does not appear in the URL. A viewer who shares the URL does not pin the variable. Use this for per-tab variables that should not leak into shared links.
• The variable query runs against the data source with the data source’s permissions. The cookie does not carry credentials; the data source plugin re-authenticates on every request.

Performance implications

• The chain is serial. Five variables at 200 ms each is one second before any panel runs.
• The panel queries are parallel within a dashboard, but every panel is blocked until the chain has resolved.
• A wide-label variable is the dominant cost. A 50,000-value variable is a 50,000-value response from the data source.
• A refresh: 2 variable re-runs the chain on every time-range change. Long-range dashboards re-resolve the chain on every interaction.

Production guidance

• Keep variable chains short. Three variables is a reasonable upper bound; five is a smell.
• Use recording rules to precompute the value list for expensive variables. A label_values(rule_metric, label) query runs in milliseconds; the same query against up{job="..."} may run in seconds.
• Set refresh: 1 unless refresh: 2 is operationally required. Document the reason in the dashboard description.
• Use skipUrlSync: true on per-tab variables. The URL stays clean; the cookie holds the state.
• Measure variable-query latency with grafana_templating_variable_resolve_time_seconds. Treat a per-variable resolve time over 500 ms as a bug.

Verification

You should now be able to answer:

• What is the N-by-M query blast, and how does it scale with the value count of each variable?
• Why is a wide-label label_values query expensive even with a regex filter on the variable?
• What is the chained-variable storm, and how does it manifest when the parent variable is All?
• Where does Grafana persist variable state across reloads, and what does the cookie contain?
• How do you measure variable-query cost on the Grafana server?

Quiz