Break/Fix: PromQL Returns No Data

Reported symptoms

The edge-api dashboard has ten panels. Four of them have read “No data” without interruption since 03 August. The other six are fine, and have been fine the whole time, on the same dashboard, against the same data source, for the same service.

That split is the reason the ticket sat for three weeks. A broken data source breaks ten panels. A dead exporter breaks ten panels. Four out of ten looks like four broken panels.

Three other observations are on the ticket, filed separately and never joined up.

An alert has been silent through two incidents. EdgeApiErrorRateHigh last fired on 02 August. Since then the service has had two incidents in which the error rate was elevated for over twenty minutes and visible, in real time, on a panel two rows down. Nobody paged. The rule was reviewed twice; the threshold is right, the routing is right, the expression parses.

Two panels flicker. They draw normally for a few minutes, then gap, then come back. Three people have looked for a pattern - deploys, traffic peaks, the hour of the day - and none of them found one, because there isn’t one to find.

Two attempts to fix a blank panel made it worse. The first engineer found a genuine typo: the panel filtered method="get" against an exporter that emits GET. That is a real bug, it was corrected, and the panel stayed blank. The second engineer replaced rate() with irate(), got nothing, tried increase(), got nothing, removed the function altogether, and left the panel plotting the raw counter. It draws a line that climbs forever and never falls. It is on the dashboard today, and someone has been reading request volume off it.

Meanwhile the platform team’s own health view is green. The SLO burn-rate panels use 5m, 1h and 6h windows and have been correct throughout. up{job="edge-api"} is 1 on all 24 instances, scrape duration is normal, and no target has a lastError.

Evidence provided

The metric is not missing. That is the first thing to establish, and it takes one query.

$ curl -sG http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=count(http_requests_total{job="edge-api"})' \
| jq -r '.data.result[0].value[1]'

312

So the question is not whether the samples exist. It is how many of them fall inside the window the panel asked for. That is also one query, and it is the one nobody ran.

$ curl -sG http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=count_over_time(http_requests_total{job="edge-api"}[1m])' \
| jq -r '[.data.result[].value[1]] | unique | join(", ")'

1

$ curl -sG http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=count_over_time(http_requests_total{job="edge-api"}[5m])' \
| jq -r '[.data.result[].value[1]] | unique | join(", ")'

5

One sample per minute, on a job whose panels were written when the interval was fifteen seconds. The configuration confirms it, and shows why the blast radius stopped at three jobs.

$ curl -s http://prometheus.internal.example.com:9090/api/v1/status/config | jq -r '.data.yaml' | grep -E 'scrape_interval|job_name'

  scrape_interval: 15s
- job_name: edge-api
  scrape_interval: 60s
- job_name: payments-api
  scrape_interval: 60s
- job_name: node
  scrape_interval: 60s
- job_name: blackbox
- job_name: prometheus

And the two expressions, side by side, at the same instant.

$ for W in 1m 5m; do
printf '%s -> ' "$W"
curl -sG http://prometheus.internal.example.com:9090/api/v1/query \
  --data-urlencode "query=count(rate(http_requests_total{job='edge-api'}[$W]))" \
  | jq -r '.data.result[0].value[1] // "empty result"'
done

1m -> empty result
5m -> 312

Work the evidence before reading on

A bare selector returns 312 series. rate() over the same selector returns nothing. Both statements are true at the same instant, and neither is an error.

1. count_over_time over a one-minute window returns 1. What is the minimum number of samples rate() needs inside its range, and what does the engine do with a series that has fewer - return zero, return an error, or something else?
2. Six panels on the dashboard are fine. What do you predict is true of their range windows, before you go and look?
3. Two panels flicker with no pattern. Given a 60-second interval, what window would work most of the time and fail occasionally, and what is the occasional event?
4. The alert has been silent through two incidents. Was its expression evaluating to false, or evaluating to nothing? What does each of those produce in ALERTS, and which one leaves no trace at all?
5. Why did the bare selector keep returning data, hours after anyone would have expected the panel to be the more forgiving of the two?

Before continuing: name the change that caused this, and explain why it broke four panels and not ten.

Root cause

1. A rate is a difference, and a difference needs two points

rate(), irate(), increase() and delta() are all computed from the samples inside a range vector. With one sample there is nothing to subtract from anything. The engine does not raise an error and does not substitute zero: it drops that series from the output vector entirely.

That is the whole failure. An expression that selects 312 series produces an output of length zero, the panel renders “No data”, and nothing anywhere reports a problem, because nothing has gone wrong in the sense that any component recognises.

2. The window did not change; the spacing under it did

On 03 August a cost-reduction change raised scrape_interval from 15s to 60s on the three highest-volume jobs. The global default stayed at 15s. That is why the estate looks healthy and one dashboard does not: every job except three is unaffected, and on the affected dashboard only the panels whose windows were narrow enough to matter went dark.

At 15 seconds, a [1m] window held four samples. At 60 seconds it holds one. The expression was never edited. Nobody who reviewed the change was thinking about panels, because the change was filed as a storage change.

3. The gradation is the tell, and it looked like three bugs

The middle row is the flicker. A [2m] window at a 60-second interval sits exactly on the minimum, so any single missed scrape - a timeout, a rolling restart, a long garbage-collection pause, a target briefly refusing a connection - takes it to one sample and the series vanishes for that evaluation. Missed scrapes are not rare and they are not periodic, which is why three people looked for a pattern and found none.

The bottom row is why the platform’s own health view stayed green and hardened the belief that the pipeline was fine.

4. The bare selector was the most misleading evidence available

Two engineers concluded the metric was healthy because typing the selector into the expression browser returned data immediately. It did, and it would have gone on doing so for five minutes after the last sample ever written, because an instant selector returns the most recent sample within the query lookback delta rather than a sample at the evaluation timestamp.

So the cheapest-looking check in the toolbox is the one that cannot distinguish “healthy” from “one sample per minute” from “stopped four minutes ago”. count_over_time over the same window the panel uses can, and it is the same number of keystrokes.

Resolution

1. Decide which lever you are pulling and write it down: resolution or window. They cost different things - one gives back the ingestion saving, the other spends detection latency - and pulling both by reflex means paying twice for a single problem.
2. Inventory before editing. List every rule expression and every dashboard range whose window is under four times the scrape interval of the job it selects. The four blank panels are the visible subset; the flickering ones are the same defect one row further along, and there will be others on dashboards nobody opened this month.
3. Fix the alert rules before the panels. A blank panel is a failure someone can see and has already reported. A silent alert is one nobody can see, and this one stayed silent through two real incidents.
4. Set every affected window to at least four times the job interval - 240 seconds at a 60-second scrape - and re-derive the for clause deliberately rather than leaving the one that was chosen for a faster job.
5. Write the resulting detection latency into the rule annotation. On-call is working from the old number, and the difference between "we page within ninety seconds" and "we page within nine minutes" changes what they do while waiting.
6. Replay both incidents through the corrected expression over their historical range. The raw samples are still in the TSDB at 60-second spacing, so this is a query, not an argument - either the rebuilt rule would have crossed the threshold or it would not.
7. Repair the panel that was left plotting the raw counter, and say in the channel what it has been showing. Someone has been reading request volume off a line that only ever rises.
8. Only now decide whether 60 seconds is tolerable for this job. Any burst shorter than four minutes is below the resolution of everything you can now build on it. If that is unacceptable, revert this job to 15s and record the ingestion cost as a choice that was made rather than a regression that crept back.
9. Add the window-versus-interval check to CI in the same change, so the next interval adjustment is caught by the pipeline instead of by an on-call engineer three weeks later.

The corrected rule, with the two numbers that used to be implicit made explicit:

# /etc/prometheus/rules/edge-api.yaml
groups:
  - name: edge-api
    interval: 60s
    rules:
      # edge-api is scraped every 60s. The window must be at
      # least 4x that so one missed scrape does not empty it.
      # Detection latency: up to 4m of window plus the 2m for
      # clause, so roughly 6m worst case. Stated here because
      # the runbook quotes it.
      - alert: EdgeApiErrorRateHigh
        expr: |
          sum by (job) (rate(http_requests_total{job="edge-api",code=~"5.."}[4m]))
            /
          sum by (job) (rate(http_requests_total{job="edge-api"}[4m]))
            > 0.05
        for: 2m
        labels:
          severity: page
        annotations:
          summary: 'edge-api 5xx ratio above 5 percent'
          description: |
            Window 4m at a 60s scrape interval; worst-case detection
            approximately 6m. A burst shorter than the window is averaged
            below the threshold and will not page.

Verification

1. The window can be filled. count_over_time(expr[4m]) returns at least four for every series the expression selects. This is the check, and it is the one that would have caught the original change on the day it landed.
2. The expression produces output. rate() over the new window returns all 312 series at an instant where the old window returned none.
3. The rule fires on production-shaped data. Run promtool test rules with input samples spaced 60 seconds apart. A fixture written at the old 15-second spacing will pass against a window production cannot fill, so the fixture has to be corrected too.
4. The incidents would have paged. Evaluate the corrected expression over each incident window and confirm the threshold is crossed. If it is not, the window is too wide for the failure shape and the trade-off needs revisiting before this is called done.
5. The flicker is gone under a real missed scrape. In staging, stop one exporter for a single interval and confirm the panel keeps its line across the gap instead of breaking.
6. The rule can still be quiet. Evaluate it over a normal window and confirm no alert. A rule that fires on everything has not been fixed either, and widening a window is exactly the kind of edit that can do it.
7. The guard can fail. Reintroduce a 1m window in a branch and confirm CI rejects it. A check that has only ever passed has not been shown to work.

Prevention

• Lint the ratio, not the window. Read every rate, irate, increase and delta range out of the rule files and dashboard JSON, resolve the scrape interval of the job the expression selects, and fail when the window is under four times it. That is arithmetic on two numbers you already have, and it closes the class rather than the instance.
• Treat scrape_interval as a query-affecting change. It is filed as a storage setting and reviewed as one. The review that matters is the list of windows the change invalidates, not the diff of the YAML.
• Alert on rules that have gone quiet. An expression that has selected no series for days is far more likely to be broken than to be describing a service that is perfectly healthy. Silence is a state, and it is worth measuring.
• Make no-data a visible state in Grafana. A blank panel reads as “quiet service” to everyone who did not build it. Panels that drive decisions should say they have no data rather than showing an empty rectangle.
• Publish the detection latency per alert. Writing it into the annotation makes a window change visibly a change to a number someone else depends on, instead of an internal detail of an expression.
• Never repair a panel by deleting the function. If rate() returns nothing, the answer is upstream of the panel. A raw counter plotted as a rate is the worst available outcome, because it is the only one that stops people asking.