Break/Fix: Dashboard Variable Causes Huge Query

Reported symptoms

The workloads dashboard has been getting slower for three weeks. This morning it stopped finishing at all: the header renders, the variable dropdowns say “loading”, and after seventy seconds the browser tab is still empty.

The ticket has been open for eleven days and has collected four observations that do not obviously belong to the same problem.

• Every panel is fast. Once the dashboard finally paints, each panel redraws in under 300 ms. Opening any of the same queries in Explore returns immediately. Whatever is slow is not the panels.
• Other teams are affected. The payments dashboard and the ingress dashboard, which share nothing with this one except the Prometheus behind them, go slow in the same minutes and recover in the same minutes. Nobody has connected the two reports because they were filed by different teams against different services.
• Alerts are late. Rule groups that nobody has edited have started missing evaluation iterations during business hours. The alerting team opened their own ticket.
• The platform looks healthy. Prometheus CPU is around 45% and resident memory is flat across the whole period. The platform team looked at the capacity graphs, correctly concluded that Prometheus is not undersized, and handed the ticket back.

And one panel, intermittently and only sometimes, shows an error its neighbours never show:

query processing would load too many samples into memory in query execution

The dashboard’s variable dropdown, meanwhile, contains twelve entries. Three people have looked at it and independently concluded that the variables on this dashboard are small.

Evidence provided

Start where the time is going. The browser network panel during a cold load, sorted by duration, puts the slowest request at the top - and it is not a panel query.

$ RANGE_START=$(date -d '7 days ago' +%s); RANGE_END=$(date +%s)
time curl -s -G http://prometheus:9090/api/v1/label/pod/values \
--data-urlencode 'match[]=container_cpu_usage_seconds_total' \
--data-urlencode "start=$RANGE_START" --data-urlencode "end=$RANGE_END" \
| jq '.data | length'

178412

real    0m21.284s

The same label, as it stands in the head block right now:

$ curl -s http://prometheus:9090/api/v1/status/tsdb \
| jq '.data.labelValueCountByLabelName[] | select(.name == "pod")'

{
"name": "pod",
"value": 2914
}

The variable definition, from the dashboard JSON:

{
  "name":       "pod",
  "type":       "query",
  "datasource": { "type": "prometheus", "uid": "prom-prod" },
  "query":      "label_values(container_cpu_usage_seconds_total, pod)",
  "regex":      "/^checkout-.*/",
  "refresh":    2,
  "multi":      true,
  "includeAll": true,
  "allValue":   ".*"
}

The commit that introduced it, three weeks ago:

$ git show --stat HEAD~14 -- dashboards/prod-sre/workloads.json
    dash: filter the pod dropdown to checkout workloads

    The recording rule doesn't carry the pod label, so point the
    variable at the raw metric and trim the list with a regex.
    Dropdown looks the same as before.

The engine’s admission gate during a slow window:

$ curl -s -G http://prometheus:9090/api/v1/query \
--data-urlencode 'query=prometheus_engine_queries or prometheus_engine_queries_concurrent_max' \
| jq -r '.data.result[] | "\(.metric.__name__) \(.value[1])"'

prometheus_engine_queries 20
prometheus_engine_queries_concurrent_max 20

And the rule groups that nobody touched:

increase(prometheus_rule_group_iterations_missed_total[1h]) > 0

Finally, the selector the failing panel actually sends, read from panel Inspect with All selected in the dropdown:

sum by (pod) (rate(container_cpu_usage_seconds_total{pod=~".*"}[5m]))

Work the evidence before reading on

Six questions, in the order that costs least to answer.

1. The dropdown shows twelve entries and the API call returns 178,412 values. Where does the narrowing happen, and does it happen before or after the data source has done its work?
2. The head block holds 2,914 distinct pod names and a seven-day query returns 178,412. What is the difference between those two numbers a statement about?
3. Panels are fast and the dashboard is slow. What in Grafana’s load sequence makes those two facts compatible rather than contradictory?
4. refresh: 2 re-runs the variable on every time-range change. The NOC wall display refreshes every ten seconds - does that change the time range? What does it change, and is that better or worse than you first assumed?
5. Read pod=~".*" as the PromQL engine reads it. Which series does that selector exclude?
6. Prometheus CPU is 45% and everything queues anyway. What kind of limit produces queueing without saturation, and where would you look for it?

Before continuing: three other teams are slow at the same moments as this dashboard. What do they share, given that they share no metrics, no dashboards and no services?

Root cause

The regex narrows the dropdown and nothing else

This is the fact the whole incident hangs on, and it is genuinely counter-intuitive.

A Grafana query variable resolves in two stages. The data source is asked for a value list; the browser then applies the variable’s regex to the list it received. The regex is a display filter. It runs last, in JavaScript, on values the data source has already found, assembled and transmitted.

So the dashboard that shows twelve entries and the dashboard that shows 178,412 entries issue exactly the same query and cost exactly the same. The only difference is what the human sees. Three engineers looked at a twelve-item dropdown and concluded the variable was small, and every one of them was reasoning about the wrong end of the pipeline.

Repointing the variable changed its cost by two orders of magnitude

The variable used to read a recording-rule metric: a small, deliberately maintained series set whose entire purpose was to be cheap to enumerate. The call returned in tens of milliseconds.

The new filter needed the pod label, which the recording rule did not carry. The obvious move - point the variable at the raw metric instead - is what the commit did, and it is what turned a bounded index lookup into an unbounded one.

The unboundedness has a second dimension that is easy to miss: label_values is evaluated over the dashboard’s time range. The number of pods alive right now is 2,914. The number of distinct pod names that existed at any point in the last seven days is 178,412, because pods are replaced constantly and every replacement is a new label value in the index for as long as its samples are retained. The same variable is a cheap query at one hour and an expensive one at seven days, and nothing in the dashboard says so.

Panels are fast because they never get to start

Grafana resolves variables before it runs panel queries, because a panel query cannot be built until the variables it interpolates have values. The variable chain is therefore a barrier: every panel waits behind it.

That is the whole explanation for “each panel renders in 300 ms and the dashboard takes seventy seconds”. Measuring panels was measuring the part that was never slow. refresh: 2 widens the barrier from once per load to once per time-range change as well.

All is not a filter

pod=~".*" looks like a filter and is not one. The regex matches every value, including the empty string, so it also matches series that carry no pod label at all. The selector narrows nothing; it is container_cpu_usage_seconds_total with extra characters.

The panel then aggregates roughly 2,900 live series across about 1,600 steps of a seven-day range, reading a five-minute window of samples at each step. That is on the order of 90 million samples against a default limit of 50 million, which is why that one panel intermittently reports that the query would load too many samples - and why its neighbours, which are scoped to a handful of series, never do.

That error is the platform working correctly. It is a guard, and it is currently the only component in the system telling the truth about this panel.

Why other teams went slow

The query engine admits a fixed number of concurrent queries - twenty by default - and holds the rest in a queue. Rule evaluation runs its expressions through the same engine as dashboard queries do.

A wall display refreshing this dashboard every ten seconds is a permanent tenant of that gate. Add a few humans opening the same dashboard during business hours and the gate stays full, so every other query on the instance waits: the payments dashboard, the ingress dashboard, and the rule groups that then start missing iterations.

This also explains the healthy capacity graphs. The constraint being hit is an admission limit, not a resource limit. CPU at 45% with everything queueing is exactly the signature of a gate rather than a bottleneck, and the platform team was right about the hardware and wrong about the conclusion.

Resolution

1. Take the relief first and label it as relief. Move the NOC wall display to a fixed short time range and a slower refresh, or pause it. That removes the largest block of unattended load within a minute, changes nothing about the defect, and needs an owner and an end date or it silently becomes the fix.
2. Bound the variable at the data source. Give label_values a selector that constrains the index lookup before the answer is assembled, so the endpoint is asked about one namespace or one workload rather than every pod the retention window has ever seen.
3. If the filter genuinely needs a label the cheap metric does not carry, restore the cheap metric rather than abandoning it: a recording rule that carries pod for the workloads in scope makes the dropdown cheap again. Budget for it honestly - it adds series and evaluation cost of its own, and that cost is now yours to watch.
4. Keep the regex if it helps the reader, but stop treating it as the bound. It was never a bound and the next person will read it as one unless the variable description says otherwise.
5. Set refresh back to on-load unless the value list must genuinely track the time range. On-time-range-change turns every zoom into a full re-resolve that blocks every panel on the dashboard.
6. Stop All meaning everything. Drop includeAll, or give allValue a bounded selector, or rewrite the panel around topk so an unbounded selection still produces a bounded result.
7. Leave the sample limit alone. Raising it removes the only component currently reporting that the panel is unreasonable, and converts a fast, local, loud refusal into a slow query that consumes the engine and then fails anyway.
8. Leave the concurrency limit alone too. Raising it does not create capacity; it lets more expensive queries run at once and moves the failure from queueing into memory, where it is far harder to reason about.
9. Turn on the Prometheus query log if it is not already on, and keep it on. It is the only artefact that names the query rather than describing the symptom, and this investigation took eleven days without it.
10. Close the loop with the two teams who filed the other tickets. They were reporting a real fault in a shared resource and were told their dashboards looked fine.

Verification

1. Time the variable, not the panels. Run the label-values call by hand over the widest time range the dashboard supports and require an answer in tens of milliseconds. The panels were never slow and measuring them again will produce the same reassuring numbers that stalled the ticket for eleven days.
2. Load the dashboard cold. A private window with no cached variable state, on the seven-day range, measured to first paint - that is what the wall display and the 03:00 on-call engineer both experience, and a warm session does not.
3. Watch the admission gate through a load. prometheus_engine_queries must stay clearly below prometheus_engine_queries_concurrent_max. A gate still pinned at its ceiling means the constraint has not moved and the estate-wide symptom will come back.
4. Confirm the collateral damage stopped: increase(prometheus_rule_group_iterations_missed_total[1h]) must be zero across the rule groups that were affected, and group evaluation duration back to its pre-incident shape. This is the measurement that proves the fix reached beyond one dashboard.
5. Exercise All on purpose. Someone will pick it. Confirm the panel now returns a bounded result or fails fast, rather than holding an engine slot for thirty seconds and then erroring.
6. Re-read the query log after the change. The dashboard's queries should no longer appear among the slowest entries; if they still do, the bound you added is not where the cost is and the next step is to find out where it went.
7. Check what you broke. Confirm with the people who used that dropdown that the values they need are still in it, since bounding the variable removed the ability to select any pod in the estate and that was a capability somebody was using.
8. If a recording rule was added, verify its own cost: its evaluation duration, and the series it adds. A fix that quietly relocates the load has not been verified until the new location has been measured.

Prevention

• A short dropdown is a fact about the browser. Every client-side narrowing in Grafana runs after the data source has already done and paid for the work, so an interface that looks cheap says nothing about what the platform was asked to do.
• Bound value lists with a selector, not with a regex. Where a dropdown is read often, give it a recording rule whose entire purpose is to be the cheap thing that dropdown reads.
• Remember that label_values is evaluated over the dashboard time range. On a label with churn, the same variable is two orders of magnitude more expensive at seven days than at one hour, and nothing on the dashboard warns you which one you are running.
• Never let All expand to a match-everything regex on a high-cardinality label. A selector that also matches series lacking the label is not a filter and should not be spelled like one.
• Review dashboard cost, not dashboard appearance. Open the change in a private window at the widest realistic time range and read the slowest request in the network panel. Thirty seconds, and it catches this entire class.
• Keep the query log enabled in production. Symptoms describe; only the log names.
• Treat a shared Prometheus as a shared resource with an admission budget, and watch missed rule iterations as a first-class signal. That metric is how one team’s dashboard becomes another team’s late page, and it is the only place the connection is visible.
• Give wall displays the scrutiny of production jobs. They are unattended load, they run whatever time range someone set months ago, and unlike a human they never get bored and close the tab.