Common Variable Patterns

The on-call engineer opens a Kubernetes service dashboard and chooses a namespace. The dropdown lists 137 of them. 89 are kube-system, kube-public, istio-system, linkerd, monitoring, and the rest are control-plane namespaces that should not be in the service filter at all. The dashboard takes eleven seconds to load. The first panel says “No data” because the regex matched the literal kube-system|kube-public|istio-system|... against the service label and produced a 1,400-character regex.

This is what the canonical variable layout exists to prevent. A well-shaped layout does three things: it filters the value list at the source, it orders the chain from broadest to narrowest, and it treats the All-value as a first-class citizen that the regex must include.

What it is

The canonical variable layout for a Kubernetes-flavoured production dashboard is a chain of three query variables, each rooted in the same Prometheus metric and ordered from broadest to narrowest:

  +-------------------+    +--------------------+    +-------------------+
  |  $cluster         |--> |  $namespace        |-->|  $service         |
  |  label_values     |    |  label_values      |    |  label_values     |
  |  (up, cluster)    |    |  (up, namespace)   |    |  (up, service)    |
  +-------------------+    +--------------------+    +-------------------+

Each variable is filtered on the previous variable’s current selection through =~. Each has multi: true and includeAll: true so the viewer can fan out or collapse the filter. Each has refresh: 1 so the chain resolves on dashboard load.

The canonical extensions for log and trace signals are the same chain against a different data source:

  +-------------------+    +--------------------+    +-------------------+
  |  $cluster         |--> |  $namespace        |-->|  $service         |
  |  Loki:            |    |  Loki:             |    |  Loki:            |
  |  label_values     |    |  label_values      |    |  label_values     |
  |  (... | cluster)  |    |  (... | namespace) |    |  (... | service)  |
  +-------------------+    +--------------------+    +-------------------+

The shape is the same. The label extraction differs: Loki variables use label_values against a Loki query; Tempo variables use label_values against a Tempo query.

Why a sysadmin cares

Three operational pains map directly to the canonical layout:

1. Predictable navigation. A viewer who knows the layout can predict the chain: cluster narrows to namespace narrows to service. The dropdowns behave the same way on every dashboard that follows the pattern.
2. Bounded value lists. A query variable that does not filter the value set at the source produces a dropdown that scrolls forever. The canonical layout uses a regex field on the variable to exclude control namespaces at the source.
3. Survivable All-value. The All-value is the most-used selection in production. A regex exclusion that accidentally excludes the All-value’s expansion turns “no filter” into “no data”. The canonical layout pairs includeAll: true with allValue: ".*" and writes the regex so it matches what All expands to.

The wrong shape shows up as a dropdown that takes minutes to load and a panel that returns nothing when the viewer expects “everything”.

The All-value and the regex exclusion

The All-value expands to allValue. When allValue: ".*", the data source receives cluster=~".*". The exclusion regex on a downstream variable must permit .* to pass through, or the chain breaks at the All selection.

The failure shape is subtle: a regex like cluster=~"(prod|staging).*" excludes every label value that does not start with prod or staging. When the viewer selects All on the parent variable, cluster expands to .*, which does not start with prod or staging, and the downstream variable’s value list is empty.

The fix is either to use =~".*" (no exclusion) for the chain or to expand the exclusion regex to match .* as well: cluster=~"(prod|staging|.*)". The latter is redundant but documents the intent.

How it works

The canonical layout has three moving parts:

  +-------------------+
  |  $cluster         |
  |                   |
  |  query:  label_values(up{job="kube-state"}, cluster)
  |  regex:  /^(prod|staging)$/
  +-------------------+
            |
            v
  +-------------------+
  |  $namespace       |
  |                   |
  |  query:  label_values(up{job="kube-state", cluster=~"$cluster"},
  |                         namespace)
  |  regex:  /^(kube-.*|istio-.*|.*)$/
  +-------------------+
            |
            v
  +-------------------+
  |  $service         |
  |                   |
  |  query:  label_values(up{job="kube-state",
  |                            cluster=~"$cluster",
  |                            namespace=~"$namespace"},
  |                           service)
  |  regex:  /^(kube-state|node-exporter)$/
  +-------------------+

The regex field on each variable is a client-side filter applied after the data source returns the value list. The regex is not PromQL; it is Grafana’s own value-list filter, written as a JavaScript regex with optional slashes.

Three details to notice:

• The chain expands the value set, then narrows. The $cluster query returns every cluster Prometheus has; the regex narrows to prod and staging. The $namespace query returns every namespace Prometheus has for the selected cluster(s); the regex permits the All-value (.*) and matches common control namespaces.
• The regex must include the All expansion. When allValue: ".*", the regex on the downstream variable must match .* or the chain breaks at All. The canonical pattern is (my-pattern|.*).
• The regex is not a security boundary. The regex filters the value list; it does not filter the data the panels render. A viewer who picks a cluster outside the regex still sees panels that match that cluster; the regex only affects what is selectable.

How to configure it

The canonical templating.list for a Kubernetes dashboard:

{
  "templating": {
    "list": [
      {
        "name":        "cluster",
        "label":       "Cluster",
        "type":        "query",
        "datasource":  { "type": "prometheus", "uid": "prom-prod" },
        "query":       "label_values(up{job=\"kube-state\"}, cluster)",
        "regex":       "/^(prod|staging)$/",
        "refresh":     1,
        "includeAll":  true,
        "allValue":    ".*",
        "multi":       true,
        "sort":        1,
        "current":     { "selected": true, "text": "All", "value": "$__all" }
      },
      {
        "name":        "namespace",
        "label":       "Namespace",
        "type":        "query",
        "datasource":  { "type": "prometheus", "uid": "prom-prod" },
        "query":       "label_values(up{job=\"kube-state\", cluster=~\"$cluster\"}, namespace)",
        "regex":       "/^(kube-system|kube-public|istio-system|.*)$/",
        "refresh":     1,
        "includeAll":  true,
        "allValue":    ".*",
        "multi":       true,
        "sort":        1,
        "current":     { "selected": true, "text": "All", "value": "$__all" }
      },
      {
        "name":        "service",
        "label":       "Service",
        "type":        "query",
        "datasource":  { "type": "prometheus", "uid": "prom-prod" },
        "query":       "label_values(up{job=\"kube-state\", cluster=~\"$cluster\", namespace=~\"$namespace\"}, service)",
        "regex":       "/^(kube-state|node-exporter|.*)$/",
        "refresh":     1,
        "includeAll":  true,
        "allValue":    ".*",
        "multi":       true,
        "sort":        1,
        "current":     { "selected": true, "text": "All", "value": "$__all" }
      }
    ]
  }
}

Three patterns to notice in this layout:

• The regex field has slashes. Grafana parses the regex as a JavaScript RegExp; the slashes are conventional, not required. The regex matches the entire value (^...$ anchors) and is anchored to the start and end.
• The regex permits .*. The namespace regex lists control-plane namespaces explicitly and adds .* to cover All. Without .*, All breaks the chain.
• multi: true plus includeAll: true. The viewer can pick a single cluster, several clusters, or All. The data source receives a regex in either case; the panel query uses =~ to match multi-value selections.

For Loki and Tempo, the chain is the same shape with a different datasource field and a label-extraction query:

{
  "name":       "cluster",
  "datasource": { "type": "loki", "uid": "loki-prod" },
  "query":      "label_values({job=\"kube-state\"}, cluster)"
}

{
  "name":       "service",
  "datasource": { "type": "tempo", "uid": "tempo-prod" },
  "query":      "label_values({}, service)"
}

The regex, refresh, includeAll, and allValue fields are identical. The chain order is identical.

How to validate it

Three checks confirm the chain is correctly wired.

Severity: READ-ONLY.

# 1. The JSON declares the variables in chain order with
#    the expected regex and allValue.
curl -s -u admin:$ADMIN \
  https://grafana.example.com/api/dashboards/uid/k8s-svc \
  | jq '.dashboard.templating.list[]
        | {name, regex, allValue, includeAll, multi}'

{
  "name":        "cluster",
  "regex":       "/^(prod|staging)$/",
  "allValue":    ".*",
  "includeAll":  true,
  "multi":       true
}
{
  "name":        "namespace",
  "regex":       "/^(kube-system|kube-public|istio-system|.*)$/",
  "allValue":    ".*",
  "includeAll":  true,
  "multi":       true
}
{
  "name":        "service",
  "regex":       "/^(kube-state|node-exporter|.*)$/",
  "allValue":    ".*",
  "includeAll":  true,
  "multi":       true
}

# 2. Each variable's query returns a value set. The
#    regex is not applied yet; the data source returns
#    the full label list.
curl -G -s http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=label_values(up{job="kube-state"}, cluster)' \
  --data-urlencode 'time='$(date +%s) \
  | jq '.data.result | map(.metric.cluster)'

[
  "prod",
  "staging",
  "dev"
]

# 3. The chain narrows at each step. With $cluster=prod,
#    the namespace list is the namespaces in prod only.
curl -G -s http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=label_values(up{job="kube-state",cluster="prod"}, namespace)' \
  --data-urlencode 'time='$(date +%s) \
  | jq '.data.result | map(.metric.namespace) | length'
# 41

The final check confirms the chain narrows correctly. A chain that returns the same number of namespaces regardless of cluster has a broken query.

How it can fail

Five failure shapes appear repeatedly with the canonical layout:

1. Regex excludes the All-value. The namespace regex is /^(kube-system|kube-public)$/ with no .* cover. When the viewer selects All on $cluster, the namespace list is empty. Symptom: namespace dropdown is empty when cluster is All.
2. Chain order reversed. The $service variable’s query references $namespace before $namespace has resolved. The service query runs against all namespaces, every cluster. Symptom: dashboard load takes ten seconds; service list is unbounded.
3. Anchor regex too tight. A regex of ^prod$ does not match prod-1 or prod-2. Symptom: cluster dropdown excludes multi-region variants.
4. Anchor regex too loose. A regex of prod (no anchors) matches prod, production, preprod. Symptom: cluster dropdown includes names that do not exist in the data source; the panel shows empty series for them.
5. refresh: 2 on a chain variable. A refresh: 2 on $cluster re-runs the variable query every time-range change. Symptom: dashboard load is slow on long ranges because the chain re-resolves.

How to troubleshoot it

The diagnostic order:

1. Inspect the chain order. Topological-sort by $ references. The chain must run from broadest to narrowest.
2. Inspect the regex on each variable. The regex must include .* (or whatever allValue is) to survive the All selection.
3. Run the variable query directly. Take the query field from each variable, paste it into Prometheus’s /graph, and confirm the value set.
4. Switch to a single cluster in the URL. Pin ?var-cluster=prod and confirm the namespace and service lists narrow.
5. Disable the regex temporarily. Set regex: "" on the variable. The full label list from the data source appears. If the chain still works, the regex is the bug.

Security implications

• The regex is a value-list filter, not a security boundary. The regex narrows what the viewer can pick; it does not narrow what the panel can render. A viewer who manually edits the URL to var-cluster=anything sees the panels filtered on anything regardless of the regex.
• The query field runs against the data source with the data source’s permissions. A viewer’s selection propagates to the panel query; the data source receives a regex matcher, not a value list.
• Multi-value selection can be unbounded. A multi: true variable with allValue: ".*" lets the viewer match every value the data source has. Against Prometheus, this is bounded by what the data source exposes. Against a non-paged query, this can be expensive.

Performance implications

• The chain is three round trips to the data source per dashboard load. The cost is amortised across every panel that uses the chain.
• The regex filter is client-side. A regex of 1,000 alternatives runs in the browser in microseconds.
• A refresh: 1 chain re-runs on every dashboard load. A refresh: 2 chain re-runs on every time-range change. Either way, the cost is paid by the data source, not by Grafana.

Production guidance

• Default new dashboards to the canonical layout. $cluster, $namespace, $service in that order, with multi: true, includeAll: true, allValue: ".*", and regex that permits .*.
• Document the regex in the dashboard description. The regex is the contract that controls what the dropdown shows; the viewer needs to know what is excluded.
• Use refresh: 1 unless refresh: 2 is operationally required. The cost of refresh: 2 is hidden in dashboard load latency.
• Verify the chain order by inspecting the JSON. The chain must run from broadest to narrowest; a reversed chain is a slow dashboard.

Verification

You should now be able to answer:

• What is the canonical chain order for a Kubernetes dashboard, and why does order matter?
• Why does a regex exclusion that omits .* break the All selection?
• What does multi: true combined with includeAll: true plus allValue: ".*" give the viewer?
• How do you test the chain in isolation?
• What does refresh: 1 versus refresh: 2 cost in terms of dashboard load latency?

Quiz