Break/Fix: Broken Dashboard Query · Break/Fix · Observability

Reported symptoms

At 14:34 CheckoutErrorRatioHigh fires. The on-call engineer follows the runbook link in the alert annotation, which opens the checkout service dashboard with the affected namespaces already pinned.

The panel named “Checkout 5xx ratio” says No data.

At 14:41 the incident commander writes into the timeline: error ratio flat, errors are zero, looking elsewhere. The team spends the next eleven minutes on the load balancer. The error ratio was 4.8% the whole time.

By the time anyone questions the panel, the shape of the problem looks like four unrelated things:

The panel directly to its right - request rate, same data source, same time range, same dashboard - renders perfectly.
Copying the panel’s expression into Explore returns data immediately. Two engineers take this as proof that the query is correct, which it is, and therefore that the panel must be a rendering bug, which it is not.
The engineer who built the dashboard cannot reproduce it. The panel renders for them, in their browser, while they are being asked to look at it.
The same three panels - two on this dashboard, one on the payments dashboard - have been reported before. Each was fixed in the Grafana UI by a different person, each fix was confirmed working, and each time the fault returned within the hour. The ticket has been reopened twice and is currently labelled “intermittent Grafana bug”.

Three of those four observations are true and none of them is about a bug in Grafana.

Evidence provided

The first useful move is to stop looking at the query in the panel editor and look at the query the data source received. Panel menu, Inspect, Query tab:

Expr:  sum(rate(http_requests_total{job="checkout",namespace="checkout-eu|checkout-us",code=~"5.."}[5m]))
       /
       sum(rate(http_requests_total{job="checkout",namespace="checkout-eu|checkout-us"}[5m]))
Step:  15s

The same view on the panel next to it, which works:

Expr:  sum by (namespace) (rate(http_requests_total{job="checkout",namespace=~"checkout-eu|checkout-us"}[5m]))
Step:  15s

Run the failing expression against Prometheus by hand. Note what comes back.

Read-only / SafeHTTP 200. success. empty.

$ curl -s -G http://prometheus:9090/api/v1/query \
--data-urlencode 'query=http_requests_total{job="checkout",namespace="checkout-eu|checkout-us"}'

{"status":"success","data":{"resultType":"vector","result":[]}}

Illustrative output

Read-only / Safethe label values that do exist

$ curl -s -G http://prometheus:9090/api/v1/series \
--data-urlencode 'match[]=http_requests_total{job="checkout"}' \
| jq -r '[.data[].namespace] | unique[]'

checkout-eu
checkout-us

Illustrative output

The variable definition, from the dashboard JSON:

{
  "name":       "namespace",
  "type":       "query",
  "datasource": { "type": "prometheus", "uid": "prom-prod" },
  "query":      "label_values(http_requests_total{job=\"checkout\"}, namespace)",
  "refresh":    1,
  "multi":      true,
  "includeAll": true,
  "allValue":   ".*"
}

The URL the alert annotation opens, and the URL the author’s bookmark opens:

alert link:  /d/checkout-svc/checkout?var-namespace=checkout-eu&var-namespace=checkout-us&from=now-3h&to=now
bookmark:    /d/checkout-svc/checkout?var-namespace=checkout-eu

And the provider file that owns the dashboard:

# /etc/grafana/provisioning/dashboards/prod.yaml
apiVersion: 1
providers:
  - name:                  prod-sre
    orgId:                 1
    folderUid:             sre
    type:                  file
    disableDeletion:       false
    updateIntervalSeconds: 30
    allowUiUpdates:        false
    options:
      path: /etc/grafana/provisioning/dashboards/prod-sre

Work the evidence before reading on

Everything above is a fact about something. The work is deciding what each one is a fact about.

Put the two Inspect expressions side by side. They differ by one character. Which character, and what does that character mean to the PromQL parser?
The substituted value is checkout-eu|checkout-us. Prometheus was asked whether a series has a namespace label equal to that. What is the honest answer, and is the answer an error?
The author’s URL pins one namespace and the alert’s URL pins two. What is different about the string Grafana builds in each case?
The variable also offers All, expanding to .*. Trace what that produces in the failing panel. Does it help?
Two people fixed this in the UI and watched it work. Read the provider file and say how long their fix was alive.
The panel was correct for eighteen months and nobody edited it. What changed in the dashboard that could break a panel without touching the panel?

Before continuing: the query returned HTTP 200 with an empty result array. Which of the two words “No data” is the misleading one?

Root cause

The matcher could not match, and nothing was wrong

= in PromQL is an exact label matcher. It compares the label value byte for byte against the string you gave it. =~ is a regex matcher, and | is alternation inside a regex.

When a Grafana variable is single-valued, both forms behave the same way for a plain value, which is why namespace="$namespace" looked correct for eighteen months and passed every review. When the variable is multi-valued, Grafana has to render a set of values into one string. The Prometheus data source joins them with |, escaping regex metacharacters as it goes, precisely so that the result drops into a =~ matcher and works. A data source with no interpolation rule of its own joins them with a comma instead. Either way the panel is now holding a string that only means anything to a regex matcher.

Substituted into =, the string is not alternation. It is a namespace name. Prometheus is being asked for series whose namespace label is literally checkout-eu|checkout-us, and no such series exists. The answer is an empty vector: HTTP 200, "status":"success", "result":[]. There is no error to find because no error occurred, and every layer downstream is behaving correctly - the data source plugin returns an empty frame, and Grafana draws an empty frame as “No data”.

All fails the same way and for the same reason. allValue: ".*" is written for a regex matcher; through = it asks for a namespace literally named .*.

This also explains the two observations that looked like contradictions. Explore returned data because the engineer pasted the pre-substitution expression and Explore resolved the variable against their own single-value selection. The author could not reproduce it because their browser had one namespace persisted from last week, and one selected value produces a string that an exact matcher can still match.

The change that broke it did not touch the panel

Nobody edited a query. Someone made namespace multi-value so that a second region could be viewed alongside the first - a small, obviously useful change that touched one field in the variable editor.

That edit changes the type of the string every panel interpolates, from “a value” to “a set rendered as a regex”. Every panel using =~ absorbed it without comment. Every panel using = became silently wrong. Grafana does not warn, no panel is modified, and the JSON diff is one boolean.

The fix kept being reverted

The dashboard is file-provisioned with allowUiUpdates: false. Grafana reconciles the provider every thirty seconds and rewrites the database row from the file on disk whenever they differ.

Both UI fixes were therefore genuine and correct and lived for under a minute. Both engineers watched the panel come back to life, believed the ticket, and closed it. The file in the repository never changed, so the reconcile put = back, and the fault reappeared as an “intermittent Grafana bug” - which is the only description that fits if you do not know the reconcile is happening.

Under the hood

Two mechanisms are worth knowing precisely, because both are invisible in the place an operator naturally looks.

Substitution happens before the data source sees anything. The panel editor shows the authored text with $namespace in it. The data source never sees $namespace; it receives a finished string. The only place in Grafana that shows the finished string is Inspect, Query. Any diagnosis conducted in the panel editor is reasoning about text that was never sent, which is why three people looked straight at the defect without seeing it - the character that matters is =, and it looks identical whether the variable holds one value or five.

The empty vector is a success. Prometheus distinguishes between a query it cannot evaluate and a query that selects nothing. A parse error is HTTP 400 with a message naming the position. A selector that matches no series is HTTP 200 with an empty result array, because “no series match this selector” is a correct and complete answer to the question asked. There is no severity attached to it and no log line generated, and there should not be: absent() exists precisely because an empty result is a legitimate value that operators sometimes need to alert on.

The consequence is that a mistyped label value, a mistyped metric name, a matcher against a variable that resolved to an empty string and this multi-value defect all produce byte-identical responses. Distinguishing them requires asking a second question - /api/v1/series for the metric, and the distinct values of the label - which takes seconds and localises the fault to the metric name, the label name, or the label value.

A panel that says nothing is not a panel that says zero

The eleven minutes were not lost to the matcher. They were lost at 14:41, when an experienced engineer looked at an empty panel and wrote “errors are zero” into the timeline.

That reading is the natural one. A ratio panel that has drawn a line every day for eighteen months draws no line today, and the mind supplies the missing line at the bottom of the axis. The panel offered no help in resisting it: an empty frame and a genuinely flat zero look almost identical at a glance, and the small grey “No data” is easy to read as a rendering quirk during an incident.

The defence is structural rather than a matter of care. A ratio should not be displayed without its denominator somewhere in view, because a ratio with no denominator cannot distinguish “no errors” from “no traffic” from “no answer” - three states with completely different responses. Put the raw request rate beside the ratio, or stack them, and the empty panel becomes obviously an unanswered question the moment traffic is visibly flowing.

The same failure shape appears wherever monitoring is read under pressure: an empty log panel read as “no errors logged”, a trace search with no results read as “no slow requests”, a gap in a graph read as “the value was zero”. In every case absence of evidence was recorded as evidence of absence, in writing, in an incident timeline other people then trusted.

Resolution

Change the matcher in the repository, not in the UI. The panel expression becomes namespace=~"$namespace". A UI edit on this dashboard has a lifetime of one reconcile interval, which is how this ticket was closed twice without being fixed.
Apply the same change to all three affected panels, then open the source dashboard they were copied from and fix it there too. A copied defect propagates on the next copy, and this one has already propagated twice.
Audit the rest of the dashboard repository for an exact matcher fed by a variable. Every one of them is either correct because the variable can never hold a set, or a latent instance of this fault waiting for someone to tick multi.
Decide about All rather than inheriting it. With =~ and allValue: ".*" the panel now aggregates every namespace the metric has, including ones this dashboard was never meant to cover. That is a different wrong answer, not a fix.
Where a variable value can contain regex metacharacters, use the explicit ${namespace:regex} form so Grafana emits a properly escaped alternation instead of a string whose safety depends on the values happening to be simple.
Give the ratio panel its denominator. Put the raw request rate beside it or under it, so an empty ratio is visibly an unanswered question while traffic is flowing.
Put the repository path in the dashboard description, where an operator will read it at 03:00. The provisioning model is correct; it being invisible is what cost two engineers their afternoon.
If the change cannot ship today, hold deliberately: pin the alert annotation link to a single namespace so the panel resolves, and record an owner, an expiry date and a note in the incident channel that the panel is filtered. An undated workaround outlives the memory of why it exists.
Correct the incident timeline. The line that says "errors are zero" is now known to be false, and a postmortem that leaves it standing teaches the wrong lesson to everyone who reads it later.

Verification

Reproduce through the path that failed. Open the alert annotation link, with both namespaces pinned, in a private window carrying no persisted variable state. The author browser is the least representative client on the estate and is the reason this went three rounds.
Read Inspect, Query and confirm the substituted expression contains namespace=~"checkout-eu|checkout-us". The panel editor text is not evidence; it is the string before the substitution that matters.
Run that substituted expression against /api/v1/query and require a non-empty result array. A rendered panel can be a cached frame; the API answer cannot.
Select All and confirm the panel still resolves, then confirm the number it shows is the number you intended it to show now that it spans every namespace.
Wait out one full provisioning interval and reload. A fix that has not survived a reconcile has not been shown to be a fix - this is the exact check that was skipped twice.
Prove the empty case now reads as empty. Select a namespace with no traffic and confirm the panel shows something a reader under pressure cannot mistake for zero.
Grep the dashboard repository for an exact matcher against an interpolated variable and require no matches outside the panels you deliberately reviewed. Wire the same grep into CI so the next copy cannot reintroduce it.
Confirm the alert and the panel now agree. Fire the alert in a test rule against the same window and check that the panel is non-empty for the period the alert covers; a page and a panel that disagree is the condition this incident was made of.

Prevention

Treat making a variable multi-value as a breaking change to every panel that interpolates it. It is one checkbox, it modifies no panel, and it changes the type of the string every panel receives. Nothing warns, so the discipline has to come from the reviewer.
Default to =~ for any matcher fed by a variable. Reserve = for values that are structurally incapable of being a list, and say so in a comment when you use it.
Grep for the pattern in CI. An exact matcher against $var is cheap to detect in dashboard JSON and impossible to spot by eye in a panel editor.
Review the substituted query, not the authored one. Inspect, Query is the only view that shows what the data source received, and it is the first click in any “the panel is empty” investigation.
Test dashboards from a clean session. Persisted variable state means the author is the one person who cannot see the fault, and the author is usually the person asked to confirm it.
Make the source of truth visible. A file-provisioned dashboard with allowUiUpdates: false is a good default and a trap for anyone who does not know it applies; the dashboard description is where they will find out.
Never let a ratio panel stand alone. Without its denominator in view it cannot distinguish no errors from no traffic from no answer.
Write “No data is not zero” into the incident checklist, and mean it: the expensive part of this incident was a true observation recorded as a false conclusion.

What if the variable had been left single-value and the panel copied to a second region instead?

That is the path the team nearly took, and it is worth knowing why it is worse rather than better.

Duplicating the dashboard per region keeps every = matcher working, because each copy pins one namespace. It also doubles the number of dashboards that must be edited in step, and the panels drift within a quarter - one copy gains a fix the other does not, and the two regions are then being judged by different queries. The failure mode is quieter than this one and considerably harder to find, because both dashboards render.

The multi-value variable is the right design. What was missing was the recognition that changing it is a schema change for every consumer. The same recognition applies to renaming a variable, changing allValue, or switching a variable from query to textbox - each is a one-field edit that changes what every panel receives, and none of them shows up as a panel change in review.

If you want the guard rail rather than the discipline, it is the CI grep plus a rendering test: load the dashboard headlessly with the multi-value selection that the alert links use, and assert that no panel returns an empty frame while the corresponding alert is firing. That test would have failed on the commit that ticked the box.

Why this matters

Nothing in this incident was broken. Prometheus answered correctly, the data source plugin behaved correctly, Grafana rendered the empty frame it was given, and the provisioning loop did exactly what it was configured to do. The alert fired on time and was right.

What failed was the joint between two correct systems and the human reading of the result. A variable changed type and no consumer was told. A UI affordance offered a fix that the platform was configured to discard. And an empty panel, which is a statement about the question, was recorded as a statement about the world.

That last one is the transferable lesson. Monitoring earns its keep by being read under pressure by someone who has to decide quickly, and the cost of an ambiguous display is paid entirely in those minutes. A panel that cannot distinguish “I have no answer” from “the answer is zero” is not a small usability complaint - it is a mechanism for writing false statements into an incident timeline, and this one did.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

The matcher could not match, and nothing was wrong

The change that broke it did not touch the panel

The fix kept being reverted

Resolution

Verification

Prevention