absent() and Missing Data

A dashboard panel shows a flat green line at zero. The label says up == 0. The operator concludes the service is down. The alert fires. The on-call engineer opens an SSH session and finds the service answering requests happily. The problem is that the panel was not zero — it was empty. The expression was

up == 0

and there was no up series at all because the scrape pool had not yet added the target. An empty vector compared to a scalar returns an empty vector, and a Grafana panel that has no data fills with a default zero on the threshold alert side. Or, in this case, the alert and the panel disagreed.

This lesson is the boundary between “no sample” and “value 0.” Both look the same in many Grafana visualisations. They are distinct in PromQL and the distinction matters. The wrong expression either pages the operator when nothing is wrong, or silently stays green when something is.

What it is

PromQL has two notions of missing data and they behave differently in every operator:

• A no-data state is an empty vector. The selector http_requests_total{service="orders"} returns an empty vector when no series matches the labels. The expression has no entries; the engine has nothing to evaluate.
• A zero value is a sample with the value 0.0. The selector matches a series; the latest sample is zero.

The two are visually identical in a Grafana line panel. They are functionally different in every operator that consumes the vector:

expression              empty vector          vector with value 0
------------------------+---------------------+--------------------
up == 0                 empty vector          single sample 0 == 1
up > 0                empty vector          empty vector (0 > 0 fails)
absent(up{job="x"})     single sample 1       empty vector
absent_over_time(...).  count of absent        count of zero
                        samples in window      samples in window

Three rules summarise the boundary:

• A scalar operator applied to an empty vector returns an empty vector. The empty result propagates through the pipeline.
• absent(metric) returns a single-sample vector when no series matches metric. The synthetic label metric carries the metric name as a label value. An alert on absent(...) > -1 is conventionally used to page on missing data.
• Grafana panel-level no-data settings convert empty vectors to either a panel with no rendering or a “no data” annotation. They do not convert empty vectors to the value 0.

The lesson returns to each shape in production depth.

Why a sysadmin cares

Most production incidents start with something the platform did not alert on. The most expensive of these is the one where the platform said everything was fine because the metric was missing, not because it was zero.

Two shapes are common:

• Silent removal. A service is scaled to zero. The exporter stops emitting. The up metric disappears. The alert up == 0 was on the “any series reports zero” interpretation and stays silent because there is no series to report zero on. The operator looking at the dashboard sees “no data” and assumes the panel is broken.
• Silent loss. An exporter crashes for one reason or another. The platform sees a single up == 1 sample, then the series becomes stale (lesson 02). The alert up == 0 does not fire because the last sample is 1; the series is still in the head as stale. The operator looking at the dashboard sees no line at all (because Grafana hides stale series) and the platform does not page.

Both shapes call for an absent-family alert that survives the “no sample” boundary correctly.

How it works

The evaluation order of an alert:

rules:
  - alert: MetricMissing
    expr: absent(http_requests_total{job="checkout"})
    for: 5m

evaluation
   |
   v
run absent(http_requests_total{job="checkout"}) over [5m]
   |
   v
result is empty vector  ---> absent() returns a synthetic
                              sample labelled with the metric
                              name; absent() over a non-empty
                              range is a vector-of-vectors
   |
   v
for: 5m duration  ---> alertmanager-style hysteresis;
                       page only when the condition has held
   |
   v
fire when threshold (> 0) and the for-window has expired

The shape of the alert evaluation:

• absent(metric) is an instant-vector function. At evaluation time it returns either an empty vector or a single sample.
• absent_over_time(metric[5m]) returns a vector with one entry per series that was never observed in the window. It is the range-vector version of absent().
• count_over_time(...) returns a count of samples that were observed. Comparing the two gives a “have we seen the series at all in the last window” check.

The interaction with Grafana:

• Grafana 11.x distinguishes empty results from zero values in panel settings. The “No value” option maps to: “reduce empty result to the value 0,” “show ‘no data’”, “show the last value,” or “display the cell as blank.” Each is correct in some panel types and wrong in others.
• The default for time-series panels is “show no data.” For stat panels it is “show 0.” For table panels it is “show blank row.” The mapping is documented per panel type.
• Alert evaluation in Grafana reads the same state model: an empty query is “no_data” and triggers the “no_data” alert rule when configured.

How to configure it

The configuration surface has two halves: the rules file and the Grafana panel.

Rules file

# /etc/prometheus/rules/availability.yml
groups:
  - name: scrape.health
    interval: 30s
    rules:

      # A: fires when the named metric has no series at all at
      # the evaluation instant. Cheap but does not survive the
      # stale-series boundary (lesson 02).
      - alert: JobMissing
        expr: absent(up{job="checkout"})
        for: 5m
        labels:
          severity: page
          category: scrape
        annotations:
          summary: 'No up{job="checkout"} series for 5m'
          runbook_url: 'https://runbooks.example/job-missing'

      # B: fires when the metric has had no sample in the last
      # 10m. Survives the stale boundary because stale samples
      # still count towards the count. This is the production
      # default for "the metric is missing" alerts.
      - alert: JobMissingSustained
        expr: |
          count_over_time(up{job="checkout"}[10m]) == 0
        for: 5m
        labels:
          severity: page
          category: scrape
        annotations:
          summary: 'No up{job="checkout"} samples in last 10m'

      # C: fleet-completeness. Survives both stale and
      # absent boundaries by counting against an inventory.
      # The expected count is a constant per job; the alert
      # fires when the platform sees fewer.
      - alert: JobBelowExpectedCount
        expr: count(up{job="checkout"}) < 6
        for: 5m
        labels:
          severity: page
          category: scrape
        annotations:
          summary: 'Only {{ $value }} instances of checkout are reporting'

Three patterns cover the boundary. A combined alert typically includes all three for the same job, each with a different label so the operator knows which shape triggered.

Grafana panel

# Grafana dashboard provisioning for the matching stat panel
# /etc/grafana/provisioning/dashboards/checkout.yml
panels:
  - type: stat
    title: 'Checkout instances reporting'
    datasource: prometheus
    targets:
      - expr: count(up{job="checkout"})
        # The "No value" setting determines what an empty
        # query shows. "Show 'No data'" surfaces the absence
        # to the operator.
        noValueMessage: 'No data — instances are missing'
    options:
      reduceOptions:
        calcs: ['lastNotNull']
        # Show 'No data' on empty result, not 0. The default
        # for stat panels in 11.x is to show 0; explicit is
        # safer.
        fields: ''
      colorMode: 'background'
      graphMode: 'none'
    fieldConfig:
      defaults:
        thresholds:
          # 0 is no instances at all; 1 is degraded; the
          # expected 6 is healthy. The 0 threshold carries
          # the danger colour.
          mode: 'absolute'
          steps:
            - value: null
              color: 'red'
            - value: 1
              color: 'yellow'
            - value: 6
              color: 'green'

The combination is the value here. The alert pages when the service is missing; the panel turns red (because it has data to evaluate); the alert’s for window prevents transient absence from paging.

How to validate it

Three commands confirm the alert pipeline handles the boundary correctly.

# 1. Check the alert expression produces the expected empty
#    result on a non-existent metric.
curl -sf http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=absent(up{job="this_job_does_not_exist"})'
# {"status":"success","data":{"resultType":"vector","result":[
#   {"metric":{"__name__":"up","job":"this_job_does_not_exist"},
#    "value":[1724000000,"1"]}
# ]}}

# 2. Confirm the count_over_time variant returns zero when
#    the series has not emitted in the window.
curl -sf http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=count_over_time(up{job="this_job_does_not_exist"}[10m])'
# {"status":"success","data":{"resultType":"vector","result":[]}}
# Empty vector == "the series did not appear in the last 10m."

# 3. Run the fixture test against an empty-vector case.
promtool test rules /etc/prometheus/tests/availability_test.yml
# The fixture asserts:
#   - on a series with a single sample 15m ago, the alert
#     does not fire for the count_over_time variant
#   - on no series at all, the alert fires

A fixture for the empty-vector case:

# /etc/prometheus/tests/availability_test.yml
rule_files:
  - /etc/prometheus/rules/availability.yml
evaluation_interval: 1m
tests:
  # Empty head: no series at all, alert fires after for-window
  - interval: 1m
    input: []
    alert_rule_test:
      - eval_time: 6m
        alertname: JobMissingSustained
        exp_alerts:
          - exp_labels:
              severity: page
              category: scrape
            exp_annotations:
              summary: 'No up samples in last 10m'

The fixture exercises the no-data case without depending on a live exporter. It catches the most common drift: someone “simplifies” the rule to up == 0 and the unit test fails.

How it can fail

1. absent(metric) against a stale series stays silent. The series exists in the head; the stale marker makes it invisible to queries but still present for absent(). The alert that should fire does not. Detected by inspecting scrape_stale_marker_present along with the alert state.
2. The boolean trap on up == 0. Against a no-data head up is missing; up == 0 returns the empty vector. Alertmanager evaluates the expression each interval and sees an empty result; the alert state is no_data, not firing. The operator is told neither that the service is up nor down. The fix is a sibling alert of the shape count_over_time(up[5m]) > 0 on the same for window, not a comparison that depends on a series that may not exist.
3. Grafana panel converts empty to zero. A stat panel with the default “show 0 on empty” turns the no-data state into a green “0 instances reporting” line. The operator thinks the deployment has been removed. The fix is to set the panel’s noValue setting to “show ‘no data’” and to colour the panel against count(up).
4. absent_over_time with the wrong range. A 30-day window is too long to surface a recent absence; the count over the window stays above zero for days. The alert is silent. The fix is a 10-minute window for the immediate absence alert and a longer window as a separate “data integration lag” alert.
5. A recording rule consumes no data and produces a missing rule result. The alert depends on the recording rule and the recording rule depends on a metric that is absent. The dependent alert never fires. Detected only by inspecting the alert evaluation chain.
6. The expected-count alert overcounts. The constant in count(up) < 6 is not updated when the fleet shrinks on purpose. The alert fires correctly but for the wrong reason. The fix is to compute the expected count from a service-inventory source, not a hard-coded constant.

How to troubleshoot it

Ordered diagnostics:

1. Confirm the state. curl /api/v1/query?query=up{job="X"} returns the vector state. An empty result is “missing”; a sample with value 0 is “down”; a sample with value 1 is “up.” These are three different states.
2. Confirm the alert’s evaluation state. ALERTS\{alertname= "JobMissing"\} returns the alert’s state vector. The value field is the alert state as a string: inactive | pending | firing | no_data | error. The state no_data means the alert’s expression evaluated to empty; the alert is not in firing and not in resolved.
3. Confirm Grafana’s panel state. Open the panel’s settings and inspect the “No value” configuration. Setting it to “show ‘No data’” surfaces absence rather than hiding it.
4. Confirm the rule file matches the team intent. Run promtool test rules against the fixture. Inspect the fixture for the empty-vector case.
5. Check fleet inventory versus the count. The “JobBelowExpectedCount” alert depends on a constant. The constant should be derived from a service inventory, not a hard-coded number.
6. Replay against a fixture. promtool test rules will consume stale samples the same way the live engine does.

Security implications

Missing data is not a security boundary on its own. The interaction with absent() and count_over_time is that an attacker who can stop the exporter from emitting a specific metric can hide their activity behind an alert that is in no_data state. For high-integrity metrics (authentication events, audit metrics) the alert should not rely on absent() alone. Pair the missing-data alert with an authentication-event alert on a separate metric that cannot be silenced by stopping the exporter.

Performance implications

The performance cost of absent() and count_over_time() is similar to that of any other query: it touches the head block for every series in the selector plus the time range of the window. A 10-minute window at a 15-second scrape interval is 40 samples per series; per-series the cost is negligible. Across thousands of series the cost is meaningful. The production guidance is to keep the absent-family alerts on a small, fixed set of selectors and not on a query that touches the entire head.

Production guidance

• Use count_over_time(metric[10m]) == 0 for the primary “missing” alert. absent() alone is insufficient because the stale boundary hides the absence.
• Use count(metric) < expected_count for fleet- completeness. The constant should be a label or a derived value from a service inventory, not a hardcoded number.
• Set Grafana panel noValue to “show ‘No data’” rather than “show 0” for availability panels. Zero is a value; no data is a different state.
• Unit-test every alert against a fixture that exercises the no-data case. The fixture catches the “I refactored the rule to == instead of count_over_time” failure shape.
• Pair every missing-data alert with an authentication-event alert for high-integrity metrics.

Verification

You should now be able to answer:

• What is the difference between an empty vector and a vector with the value 0 in PromQL?
• Why does absent() alone not detect a target that has been stale for an hour?
• When should the alert shape be absent(metric), when count_over_time(metric[5m]) == 0, and when count(metric) < expected_count?
• What is the production setting for a Grafana stat panel’s “No value” option for an availability panel?
• Name one alert shape that catches the “missing” case for a high-integrity metric.

Quiz