Query Wrong

A Grafana panel that has worked for nine months goes empty. The on-call engineer opens the panel, clicks “Edit”, sees the query, and assumes the metric was renamed. They rewrite the query to a new metric name; the panel is still empty. They add a label filter that does not exist; the panel is still empty. They curl /api/v1/series?match[]=node_cpu and find the metric, with the labels they expect, in the index. The metric is there; the query is the bug. The engineer then compares the query character by character against the /api/v1/metadata output and finds a typo: the panel used mode="idle" but the metric name is node_cpu_seconds_total and the label is mode. The fix is one character; the diagnosis took forty minutes because the engineer did not run the comparison first.

The query bug is the sixth and last link in the chain. It is the link that the operator owns entirely, and the link where the diagnostic is fastest. This lesson is the discipline of running the diagnostic.

What it is

A query bug is any condition where the PromQL expression in a dashboard panel or alert rule returns no data even though the underlying metric is present in the TSDB. The metric is in /api/v1/series; the expression does not match it; the panel is empty.

Five distinct classes appear in production:

1. Metric name typo. The panel uses node_cpu_seconds_total but the actual metric is node_cpu_seconds (a counter without the _total suffix in an older exporter) or node_cpu_usage_seconds_total (a renamed metric).
2. Label mismatch. The panel filters on mode="idle" but the actual label is cpu="0" and mode="user", or the case is different, or the value is plural.
3. Wrong aggregation. The panel uses sum by (instance) (rate(metric[5m])) but the labels are not on every series in the range vector, so sum returns NaN.
4. Type error. The panel uses delta(metric[5m]) against a gauge, or rate(metric[5m]) against a counter that has only just been created and has fewer than two samples in the window.
5. Range or instant vector confusion. The panel uses metric[5m] (a range vector) where an instant vector is required, or rate(metric) where rate(metric[5m]) is required.

The first class is the most common; the others appear when a panel is updated or copied. The diagnostic that catches all five is to run the query against /api/v1/query and read the response.

Why a sysadmin cares

A query bug is the cheapest missing-metric failure to fix once diagnosed, and the most expensive to diagnose without the right command. An operator who rewrites the query without comparing it to the metric metadata spends the incident changing the wrong thing. An operator who runs the comparison first finds the bug in seconds.

Three production pains follow:

1. Panel that lies. A panel with a wrong query shows “no data” for a metric that is actually fine. The on-call engineer assumes the metric is missing and starts debugging the wrong layer.
2. Alert that never fires. An alert rule with a wrong query never enters the firing state. The metric is in the index; the threshold is correct; the query is the bug. The alert is silent.
3. Dashboard rebuild. A panel that has been wrong for months is rebuilt from a copy that has the same bug. The “fix” preserves the failure shape. The discipline is to test the panel query before rebuilding it.

How it works

A PromQL expression is evaluated in two stages: parse, then evaluate. The parse stage reads the expression; the evaluate stage walks the time-series database and returns matching series.

  query string
        |
        v
  +-----------+
  | parse     |  PromQL grammar; syntax errors fail here
  +-----------+
        |
        v
  +-----------+
  | evaluate  |  AST -> series selection -> result vector
  +-----------+  range and instant vectors are returned
        |         with the requested timestamp
        v
  Grafana panel or alert evaluation

A failure in the parse stage is loud: Prometheus returns an HTTP 400 with the error message. A failure in the evaluate stage is quiet: the expression parses, the database is consulted, and the result vector is empty. The operator sees an empty panel and assumes the metric is missing.

The expression is matched against the index in two passes. The metric name is matched first (a lookup against /api/v1/series?match[]=name). The label matchers are matched next (a per-series filter). An empty result can mean the metric name is wrong, the label filter is wrong, or both.

Under the hood

How to configure it

The PromQL expression that surfaces every class of bug and is easy to test:

# A correct expression for CPU idle ratio per instance
# - rate() requires a counter; _total is stripped automatically
# - the range window is 5 minutes
# - the aggregation is sum by (instance)
# - the result is the ratio of idle CPU to total CPU
- expr: |
    sum by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
    /
    sum by (instance) (rate(node_cpu_seconds_total[5m]))
  record: node:cpu:idle_ratio

The five parts that change when a query is wrong:

• Metric name: node_cpu_seconds_total is the actual exporter metric; the _total suffix matches Prometheus’s counter convention but is part of the name when used in a dashboard query.
• Label matcher: mode="idle" matches the actual label value; case and spelling matter.
• Range window: [5m] is a range vector; the function rate() requires a range.
• Function: rate() is for counters; delta() is for gauges.
• Aggregation: sum by (instance) collapses the result to one series per instance; without it, the panel shows one series per CPU.

A typo in any of the five parts produces an empty result without an error.

How to validate it

The validation ladder for a query bug.

# Step 1: confirm the metric is in the index
curl -s -G 'http://prom:9090/api/v1/series' \
  --data-urlencode 'match[]=node_cpu_seconds_total' \
  | jq '.data | length'
# expected (healthy): a non-zero number
# expected (bug):     zero (metric does not exist; not a query bug)

# Step 2: list the actual labels for the metric
curl -s -G 'http://prom:9090/api/v1/series' \
  --data-urlencode 'match[]=node_cpu_seconds_total' \
  | jq '.data[0] | keys'
# expected: every label name the metric has

# Step 3: list the actual values for the suspect label
curl -s -G 'http://prom:9090/api/v1/series' \
  --data-urlencode 'match[]=node_cpu_seconds_total' \
  | jq -r '[.data[] | .mode] | unique[]'
# expected: the actual mode values; compare against the
# matcher's value

# Step 4: run the query as the panel would
curl -s -G 'http://prom:9090/api/v1/query' \
  --data-urlencode 'query=node_cpu_seconds_total{mode="idle"}' \
  | jq '.data.result | length'
# expected: non-zero if the query is correct

# Step 5: explain the query
curl -s -G 'http://prom:9090/api/v1/query' \
  --data-urlencode 'query=node_cpu_seconds_total{mode="idle"}' \
  --data-urlencode 'explain=true' \
  | jq '.data'
# expected: a query plan that walks the index

# Step 6: validate the query offline with promtool
promtool query --no-headers \
  'http://prom:9090/api/v1/query' \
  'node_cpu_seconds_total{mode="idle"}'
# expected: a non-empty result

# Step 7: validate the unit test with promtool test rules
# (covered in the alerting lessons)

The first three steps localise the bug to the metric, the label, or both. Steps 4 and 5 confirm the query. Step 6 is the offline equivalent. Step 7 is the unit test that catches the bug before the alert is deployed.

How it can fail

Six failure shapes appear repeatedly. The first three are typos; the second three are semantic.

1. Metric name typo. The panel uses node_cpu_seconds but the actual metric is node_cpu_seconds_total (or vice versa). Symptom: empty result; the metric is in /api/v1/series under a different name. The fix is to match the actual name from /api/v1/metadata.
2. Label value typo. The panel uses mode="IDLE" (upper case) or mode="idle " (trailing space). Symptom: empty result; the value is not in the index. The fix is to match the case and spelling in /api/v1/series.
3. Label name typo. The panel uses cpu="0" but the actual label is core or cpu_id. Symptom: empty result; the label is not in the index. The fix is to match the actual label name from /api/v1/series.
4. Counter rate on a gauge. The panel uses rate(metric[5m]) but the metric is a gauge. Symptom: empty result; rate() on a gauge with monotonic behaviour returns no useful values. The fix is to use delta() or to switch to the appropriate function.
5. Aggregation over no labels. The panel uses sum by (instance) (metric) but the metric has no instance label (it was dropped by metric_relabel_configs, or it never had one). Symptom: empty result; the aggregation drops every series. The fix is to remove the by (instance) clause or to use a label the metric has.
6. Range vector where instant is required. The panel uses metric[5m] directly in an alert expression. Symptom: parse error or empty result depending on the engine version. The fix is to wrap the range vector in rate() or increase().

How to troubleshoot it

Security implications

A query can leak data across tenants or expose high-cardinality labels that the operator did not intend. The disciplines:

• label_values() and query are available to anyone with read access to the Prometheus API. A leaked token can enumerate label values for sensitive labels (user IDs, request UUIDs) at whatever cardinality the TSDB holds. Restrict API access to the on-call jump box.
• A query that returns a label value verbatim can be the leak path for secrets in metric labels. The earlier lesson on metric_relabel_configs covers the defence.
• An alert expression with a wrong matcher can fire on the wrong series. The defence is promtool test rules; the alerting lessons cover the pattern.

Performance implications

A wrong query is usually cheap; a correct but expensive query is the real performance trap.

• count by (__name__) ({__name__=~".+"}) enumerates every metric in the TSDB. Useful for inventory; expensive on large deployments. Cache the result; do not run on every panel refresh.
• histogram_quantile(0.95, sum by (le, instance) (rate( histogram_bucket[5m]))) is correct but expensive on high-cardinality histograms. Cap the cardinality upstream with metric_relabel_configs.
• Long range windows. A query with [1h] on a high-cardinality metric scans an hour of data per refresh. Prefer [5m] for panel queries; use recording rules for longer windows.

Production guidance

• Test every alert query with promtool test rules before deployment. The alerting lessons cover the syntax.
• Run /api/v1/query?explain=true on every dashboard panel during a quarterly review. The output names the index lookup the engine performs.
• Cap label cardinality with metric_relabel_configs so dashboard queries cannot enumerate a million label values.
• Use recording rules for aggregations that are reused across panels. The rule runs once per evaluation interval; the panels read the rule’s output.
• Version the dashboard JSON. A panel that has been wrong for nine months is a panel that was never reviewed; the review is the discipline.
• Add a comment to every non-trivial query explaining the metric name, the label matchers, and the aggregation. The comment is for the engineer who inherits the panel.

Verification

You should now be able to answer:

• What is the difference between a missing metric and a wrong query, and how do you tell them apart?
• Which three read-only API calls localise a query bug before editing the panel?
• What does /api/v1/query?explain=true return, and how do you read it?
• Why is rate(metric[5m]) not the same as rate(metric), and what is the failure shape when the difference is ignored?
• How do you write a unit test for an alert query that catches the bug before deployment?

Quiz