Query Correctness

The on-call engineer opened the request-rate panel during a scaling incident. The panel showed “No data” for the previous twenty minutes. They checked the alert that paged them, “request rate above 1000 rps”, and saw that the alert had fired. So requests were happening. The panel was wrong. The dashboard was wrong. The on-call engineer reverted to a notebook and a sum(rate(http_requests_total[5m])) typed into the Prometheus UI, and shipped the fix without ever trusting the dashboard again.

This is what panel-query correctness is for. Every panel in a Grafana 11 dashboard declares one or more queries in panels[].targets[]. Each query is an expression in the data source’s query language (PromQL, LogQL, TraceQL). The panel renders whatever the data source returns. If the query is wrong, the panel renders wrong. If the panel renders wrong silently — empty series, no error message — the dashboard is a lie.

Query correctness is the per-panel contract that the expression returns a non-empty, in-schema result against the live data source for the panel’s intended time range.

What it is

A Grafana 11 panel declares its queries in panels[].targets[]. Each entry has:

  +------------+----------------------------------------------+
  | Field      | Purpose                                      |
  +------------+----------------------------------------------+
  |  refId     | Local identifier; "A" by convention; "B"    |
  |            | for a second query that the panel transforms |
  +------------+----------------------------------------------+
  |  datasource| The data source UID or variable to query     |
  +------------+----------------------------------------------+
  |  expr      | The expression (PromQL / LogQL / TraceQL)   |
  +------------+----------------------------------------------+
  |  instant   | true for an instant query (stat panel);     |
  |            | false for a range query (graph panel)        |
  +------------+----------------------------------------------+
  |  range     | true for a range query (graph panel)        |
  +------------+----------------------------------------------+
  |  legendFormat | Template for the legend label             |
  +------------+----------------------------------------------+
  |  exemplar  | true to attach trace exemplars (Prometheus) |
  +------------+----------------------------------------------+

The expression is a string of query text. Grafana interpolates the dashboard’s variables before sending the query to the data source. The data source returns a response; the panel renders the response.

Query correctness has three checks:

1. Resolves. The expression parses; the data source does not return a syntax error.
2. Returns data. The expression returns at least one series for the panel’s intended time range.
3. Returns the expected data. The series labels match the panel’s intent (the panel expects service, the series has service; the panel expects a counter, the series is a counter).

A panel that fails check 1 is broken visibly. A panel that fails check 2 is broken silently. A panel that fails check 3 is broken dangerously — it renders, but with the wrong answer.

Why a sysadmin cares

Three operational pains map directly to panel-query correctness:

1. The silent empty panel. A panel whose metric was renamed looks identical to a panel whose service is down. Both render “No data”; the operator cannot tell the difference without opening the data source.
2. The wrong-rate window. A rate(http_requests_total[1m]) on a counter scraped every 30 s returns a flatter, noisier rate than the operator expects. The panel is not wrong; the rate window is wrong; the operator’s intuition is wrong.
3. The wrong label schema. A panel filters on job="checkout" but the service now exposes service="checkout". The panel renders empty; the alert (which uses the same label) does not fire; the service is invisible to the dashboard.

The wrong shape shows up as a wall of green panels that does not reflect the system, or as a single panel that disagrees with three other panels on the same metric.

Where queries come from

Three sources:

• Author-written. A panel author wrote the expr themselves. The expr is in the dashboard JSON.
• Generated from a recording rule. A recording rule pre-computes a metric; the panel queries the recording rule. The recording rule is the contract; the panel expr is just a reference to it.
• Generated from a variable. A variable expansion produces a query fragment that the panel expr combines. The expr is correct only if the variable is correct.

Each source has a different failure mode. An author-written expr drifts when the metric is renamed. A recording-rule reference drifts when the recording rule is renamed. A variable-generated expr drifts when the variable’s value list or refresh policy changes.

How it works

The query-correctness pipeline:

  dashboard JSON
        |
        v
  +---------------------+
  |  parse panels[]     |   jq '.dashboard.panels[]'
  +---------------------+
        |
        v
  +---------------------+
  |  for each panel:    |
  |  walk targets[]     |   jq '.targets[]'
  +---------------------+
        |
        v
  +---------------------+
  |  interpolate vars   |   for each variable value, substitute $var
  +---------------------+
        |
        v
  +---------------------+
  |  send to data       |   POST /api/ds/query to the panel's data source
  |  source             |
  +---------------------+
        |
        v
  +---------------------+
  |  inspect response   |   empty? label-schema mismatch? rate window?
  +---------------------+
        |
        v
  +---------------------+
  |  report             |   list of broken queries per panel
  +---------------------+

The variable interpolation is the under-appreciated step. A panel expr rate(http_requests_total{service="$svc"}[5m]) with $svc = "checkout" produces rate(http_requests_total{service="checkout"}[5m]). The expr looks fine in the JSON; it is wrong only after interpolation. A correctness check that runs the expr as literally written will miss variable-driven bugs.

How to configure it

The canonical pattern: a CI script that walks every panel, sends every expr to a staging data source, and fails the merge if any expr returns an empty data array.

#!/usr/bin/env bash
# scripts/check-panel-queries.sh
# Severity: READ-ONLY against staging, CONFIGURATION against CI.
set -euo pipefail

GRAFANA_URL=${GRAFANA_URL:-https://grafana-staging.example.com}
ADMIN_USER=${ADMIN_USER:-admin}
ADMIN_PASS=${ADMIN_PASS:?admin password required}
PROM_URL=${PROM_URL:-http://prometheus-staging:9090}

check_panel() {
  local json_file=$1
  local panel_idx=$2
  local expr=$3
  local result
  result=$(curl -G -s "$PROM_URL/api/v1/query" \
    --data-urlencode "query=$expr" \
    --data-urlencode "time=$(date +%s)" \
    | jq '.data.result | length')
  if [[ "$result" == "0" ]]; then
    echo "FAIL: $json_file panel $panel_idx expr returned empty:"
    echo "      $expr"
    return 1
  fi
}

for json_file in dashboards/*.json; do
  panel_count=$(jq '.panels | length' "$json_file")
  for i in $(seq 0 $((panel_count - 1))); do
    expr=$(jq -r ".panels[$i].targets[0].expr // empty" "$json_file")
    [[ -z "$expr" ]] && continue
    check_panel "$json_file" "$i" "$expr"
  done
done

A minimal dashboard excerpt that the script validates:

{
  "panels": [
    {
      "type": "timeseries",
      "title": "Request rate by service",
      "datasource": { "type": "prometheus", "uid": "prom-prod" },
      "targets": [
        {
          "refId":  "A",
          "datasource": { "type": "prometheus", "uid": "prom-prod" },
          "expr":    "sum by (service) (rate(http_requests_total[5m]))",
          "legendFormat": "{{service}}"
        }
      ]
    },
    {
      "type": "stat",
      "title": "p99 latency",
      "targets": [
        {
          "refId":  "A",
          "datasource": { "type": "prometheus", "uid": "prom-prod" },
          "expr":    "histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))"
        }
      ]
    }
  ]
}

For LogQL panels:

{
  "type": "logs",
  "title": "Error logs",
  "datasource": { "type": "loki", "uid": "loki-prod" },
  "targets": [
    {
      "refId":  "A",
      "datasource": { "type": "loki", "uid": "loki-prod" },
      "expr":    "{service=\"checkout\"} |= \"error\" | json"
    }
  ]
}

The CI step validates the LogQL expr against the staging Loki with query=<expr>&limit=1 and confirms a non-empty response.

How to validate it

Four checks confirm the per-panel contract is live.

Severity: READ-ONLY.

# 1. Every panel in the dashboard declares at least one
#    target with a non-empty expr.
curl -s -u admin:$ADMIN \
  https://grafana.example.com/api/dashboards/uid/svc-overview \
  | jq '[.dashboard.panels[] | .targets[]?.expr]
        | map(select(. == "" or . == null)) | length'
# 0

# 2. Every panel expr parses against the data source.
#    promtool check rules is a static check; the curl
#    below is a runtime check against a fresh endpoint.
for expr in $(curl -s -u admin:$ADMIN \
  https://grafana.example.com/api/dashboards/uid/svc-overview \
  | jq -r '.dashboard.panels[].targets[].expr'); do
  curl -G -s http://prometheus:9090/api/v1/query \
    --data-urlencode "query=$expr" \
    --data-urlencode "time=$(date +%s)" \
    | jq -e '.status == "success"' > /dev/null || \
      echo "FAIL parse: $expr"
done
# (empty)

# 3. Every panel expr returns at least one series for
#    the current time range. An expr that parses but
#    returns empty is the silent-empty failure shape.
for expr in $(curl -s -u admin:$ADMIN \
  https://grafana.example.com/api/dashboards/uid/svc-overview \
  | jq -r '.dashboard.panels[].targets[].expr'); do
  result=$(curl -G -s http://prometheus:9090/api/v1/query \
    --data-urlencode "query=$expr" \
    --data-urlencode "time=$(date +%s)" \
    | jq '.data.result | length')
  [[ "$result" == "0" ]] && echo "EMPTY: $expr"
done
# (empty)

# 4. Every variable combination renders. The CI step
#    above does this implicitly when it interpolates
#    every variable value into every expr.
for val in prod staging dev; do
  expr="sum by (service) (rate(http_requests_total{env='$val'}[5m]))"
  curl -G -s http://prometheus:9090/api/v1/query \
    --data-urlencode "query=$expr" \
    --data-urlencode "time=$(date +%s)" \
    | jq -e '.data.result | length > 0' > /dev/null || \
      echo "EMPTY for env=$val"
done
# (empty)

How it can fail

Six failure shapes appear repeatedly with panel-query correctness:

1. Metric renamed upstream. The instrumentation team renames http_requests_total to http_server_requests_seconds_count (the OpenTelemetry default) and ships the change. The panel expr returns empty. Symptom: every panel with the old metric name shows “No data” silently; alerts on the same metric stop firing.
2. Label schema changed. The service starts exposing service="checkout" instead of job="checkout". The panel’s label matcher job="checkout" matches nothing. Symptom: the panel is empty while the metric itself is present; alerts on the same label also miss.
3. rate() on a gauge. The panel expr uses rate(memory_usage_bytes[5m]) on a gauge that does not monotonically increase. The rate returns negative values the panel filters out. Symptom: the panel renders a flat line at zero; the underlying gauge is healthy.
4. Subquery misuse. A panel expr uses rate(http_requests_total[5m]) inside a subquery (...)[1h:] and the inner range is shorter than the outer step. Prometheus returns an empty series. Symptom: the panel renders “No data” only when the time range is wide enough for the subquery to engage.
5. Time-range unit confusion. A panel expr uses [5m] but the dashboard’s time-range picker is set to “Last 5 minutes”, making the rate window equal to the visible range. The rate returns a flat line because there is only one sample. Symptom: the panel renders a single point stretched across the range.
6. Data source UID changed. The Prometheus data source UID was renamed during a provisioning change. The panel still references the old UID; Grafana falls back to the default data source, which is a different Prometheus. Symptom: the panel renders data from a different environment than the dashboard title says.

How to troubleshoot it

The diagnostic order:

1. Open the panel. Click Edit. Inspect the expr field. Copy it.
2. Open the data source directly. Prometheus’s /graph, Loki’s Explore, Tempo’s Search. Paste the expr. Confirm whether the data source itself returns data.
3. If the data source returns empty: the expr is wrong against the current schema. Diff the expr’s metric and label names against what the data source actually exposes. label_values(up, __name__) lists every metric name; cross-check the dashboard’s metrics against the list.
4. If the data source returns data: the panel rendering is the issue. Check the panel’s data source UID against the dashboard’s intended data source. Check the panel’s interval step against the dashboard’s __interval and the data source’s scrape interval.
5. If the panel renders but the values look wrong: the rate window or aggregation is the issue. Add a sum by (le) for a histogram, or extend the rate window to four times the scrape interval.
6. If the failure is intermittent: the variable expansion is the issue. A variable with a multi-value selection produces N queries; one of them may be empty while the others are not.

Security implications

• The dashboard expr runs with the data source’s permissions. The expr does not carry the viewer’s identity. The data source enforces RBAC independently of the dashboard.
• A CI step that sends every expr to a data source must use a service account. The credentials live in the CI secret store; the staging data source is permissioned to allow the CI account.
• Expr inspection is a leak surface. The expr field reveals what the dashboard is looking at. Treat the dashboard JSON as an artefact whose disclosure is bounded by the dashboard’s own permissions.

Performance implications

• Each panel expr is a data source call. A dashboard with 20 panels is 20 calls per refresh. A dashboard that refreshes every 5 s is 240 calls per minute.
• A rate() over a wide range is expensive. A rate(http_requests_total[1h]) on a counter with 100,000 active series scans a large slice of the TSDB head. The cost is per call.
• A histogram_quantile() over a high-cardinality histogram is the most expensive panel query. A dashboard with ten p99 panels on different services can dominate a Prometheus load profile.
• Variable-driven queries multiply the cost. A panel expr with three variable values is three queries per refresh. A repeat-by-variable panel is N queries.

Production guidance

• Run a CI step on every dashboard pull request that sends every expr to a staging data source and fails the merge on empty results. This is the single highest-leverage correctness check.
• Use recording rules for any expr that is reused across panels or dashboards. The recording rule is the contract; the panel expr is a reference.
• Document the rate window. A [5m] window assumes a scrape interval below 30 s. A [1m] window on a counter scraped every 30 s returns a flat line.
• Avoid rate() on gauges. Use delta() only on counters; use the gauge value directly for everything else.
• Diff the dashboard JSON against the data source’s metric catalog in CI. up{__name__=~".+"} | label_values(__name__) returns every metric Prometheus has; cross-check the dashboard’s metrics against the list.

Verification

You should now be able to answer:

• What are the three checks that define panel-query correctness?
• Where do panel queries live in the dashboard JSON?
• How does variable interpolation affect query correctness?
• What is the silent-empty-panel failure shape, and how does a CI step catch it?
• Why is rate() on a gauge the wrong shape?

Quiz