Prometheus Rule Compatibility

A platform team runs a 2.51 Prometheus. A recording rule computes p99 request latency with histogram_quantile(0.99, sum by (le, job) (rate(http_request_duration_seconds_bucket[5m]))). The team upgrades to 2.55. The rule parses cleanly. Evaluation proceeds. The first dashboard refresh shows p99 latency at roughly half the previous value. The team has spent ninety minutes chasing a phantom regression before they realise that the histogram_quantile interpolation behaviour was tightened in a 2.5x release to handle empty buckets more strictly.

This is the rule-compatibility failure shape. The rule file parsed cleanly on both versions. The PromQL expression parsed cleanly on both versions. What changed was the semantics of a single function for a single edge case. The team had no rule test suite; they discovered the regression on a dashboard.

What it is

Rule compatibility is the contract between the operator’s PromQL expressions and the binary that evaluates them. The contract has three layers:

  +--------------------+------------------------------------------+
  | layer              | what lives there                         |
  +--------------------+------------------------------------------+
  | YAML syntax        | groups, rules, labels, annotations      |
  | PromQL syntax      | expressions and aggregators              |
  | PromQL semantics   | edge-case behaviour of operators and    |
  |                    | functions; flag changes; default shifts |
  +--------------------+------------------------------------------+

Within a major release, the YAML and PromQL syntaxes are stable. The semantic layer is the one that surprises operators across minor bumps: changes to histogram_quantile for empty buckets, topk ordering, deriv against counter resets, absent() over a vector with stale markers, the new treatment of 0 vs NaN in aggregations.

Each upstream release note has a [CHANGE] line that lists behavioural PromQL changes. The operator’s job is to read each release note for the version range they span and grep their rule files for any PromQL construct listed there.

Why a sysadmin cares

A rule compatibility break is the third-most expensive Prometheus incident. Three shapes recur:

1. The semantic edge case. A rule expression hits an edge case that the new binary resolves differently. The rule continues to evaluate, but the output is wrong. Detection surface is the downstream dashboard.
2. The deprecation warning that becomes an error. A function or aggregator is deprecated in 2.x and removed in 2.y. The rule file fails to load on the new binary. The fix is to grep the rule files for the deprecated symbol and rewrite ahead of the upgrade.
3. The new function that wants a different signature. A new clamp_min/clamp_max synonym appears; an aggregator accepts a new option; the rule file passes the option through, and the rule’s behaviour shifts. Detection is the rule test suite or the dashboard.

How it works

The validation shape has three ordered steps. Each has a mechanical pass / fail.

  step 1: promtool check rules     PASS = exit 0 for every rule file
       |
       v
  step 2: promtool test rules      PASS = every test case evaluates
       |                              to its expected series
       v
  step 3: sample evaluation on
          the upgrade binary       PASS = output matches a synthetic
                                          input within tolerance

promtool check rules is a structural validator. promtool test rules is a behavioural one: it parses test/ blocks embedded in the rule file (or alongside it), replays a sequence of synthetic samples through the rule, and compares the output to expected series. The test suite is the operator’s contract for rule behaviour; it must be expanded every time a new rule is added.

How to configure it

The discipline lives in the rule file’s test/ block and in the CI pipeline that runs it.

Rule file with embedded test suite

# /etc/prometheus/rules/sla.yml
groups:
  - name: sla
    interval: 30s
    rules:
      - record: api:request_latency:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (le, job) (rate(http_request_duration_seconds_bucket[5m]))
          )

      - alert: ApiHighErrorBudgetBurn
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
          / sum by (job) (rate(http_requests_total[5m]))
          > 0.02
        for: 10m
        labels:
          severity: page
        annotations:
          summary: 'job {{ $labels.job }} exceeds 2% error budget burn'
          runbook: 'https://runbooks.internal/api/error-budget'

# test/ block is parsed by promtool test rules, not by the live
# rule manager.
rule_files:
  - sla.yml
evaluation_interval: 30s
tests:
  - interval: 1m
    name: p99 stable
    input_series:
      - series: 'http_request_duration_seconds_bucket{job="api",le="0.1"}'
        values: '0+1x60'
      - series: 'http_request_duration_seconds_bucket{job="api",le="0.5"}'
        values: '0+5x60'
      - series: 'http_request_duration_seconds_bucket{job="api",le="+Inf"}'
        values: '100+0x60'
    expected_output_series:
      - 'api:request_latency:p99{job="api"} 0.5'

The expected_output_series is the contract. The promtool test rules invocation runs the synthetic series through the rule and asserts the output equals the expected value. A behavioural change between binary versions shows up here.

CI pipeline for rule tests

# .github/workflows/prom-rules-test.yml
name: prom-rules-test
on:
  pull_request:
    paths:
      - 'monitoring/prometheus/rules/**'
jobs:
  test:
    runs-on: ubuntu-latest
    container:
      image: prom/prometheus:v2.55.1
    steps:
      - name: Check out
        uses: actions/checkout@v4
      - name: Static check
        run: |
          for f in monitoring/prometheus/rules/*.yml; do
            promtool check rules "$f"
          done
      - name: Behavioural test
        run: |
          for f in monitoring/prometheus/rules/*.yml; do
            promtool test rules "$f"
          done

The CI image is the upgrade image. The promtool is the upgrade binary’s tooling. A behavioural regression in the upgrade binary is caught at PR time, not at incident time.

Documented semantic-change matrix

# monitoring/prometheus/SEMANTIC_CHANGES.md (free-form, version-controlled)
# Tracking which upstream releases changed which functions.
#
# 2.49: histogram_quantile returns NaN (not 0) for empty buckets
# 2.51: topk returns deterministic order on ties
# 2.55: rate() handles counter resets by ignoring the reset sample
#
# Each line above corresponds to a documented change in the
# upstream release notes. The rule files affected by each
# change are listed in PR descriptions.

The semantic-change matrix is the operator’s bridge between “the release notes mentioned this” and “we have a test for this.” A change noted in the matrix has a corresponding test case in a rule file’s expected_output_series. A change without a test case is a gap.

How to validate it

Five mechanical checks confirm rule compatibility.

# READ-ONLY: structural validator.
for f in /etc/prometheus/rules/*.yml; do
  promtool check rules "$f"
done
# (no output, exit 0)

# READ-ONLY: behavioural validator.
for f in /etc/prometheus/rules/*.yml; do
  promtool test rules "$f"
done
# (every test case passes)

# READ-ONLY: rule manager health.
curl -fsS http://prometheus.internal:9090/api/v1/rules \
  | jq '.data.groups[].rules[] | select(.health != "ok")'
# (empty array means every rule is healthy)

# READ-ONLY: last evaluation timestamp.
curl -fsS http://prometheus.internal:9090/api/v1/rules \
  | jq '.data.groups[].rules[] | {name: .name, lastEval: .lastEval}'
# (every lastEval is within the last evaluation_interval)

# READ-ONLY: alert evaluation status.
curl -fsS http://prometheus.internal:9090/api/v1/alerts \
  | jq '.data.alerts | length'
# (matches the expected count of firing and pending alerts)

A clean validation: promtool check rules exits 0 for every file; promtool test rules passes every test case; the rule manager reports health: ok for every rule; the last-evaluation timestamps are recent; the alert counts match expectation.

How it can fail

Six shapes recur.

1. Deprecated aggregator removed. The rule uses topk_old or quantile_old (historical names from very early releases). The new binary fails to parse the rule file. The fix is to rename to topk / quantile ahead of the upgrade.
2. histogram_quantile NaN propagation. The rule computes p99 from a sparse histogram. The previous binary emitted 0 for empty buckets; the new binary emits NaN. Downstream alerts that compared against > X see no matches (NaN compares as false). The fix is to wrap with or vector(0) or to reauthor the rule for NaN-safe semantics.
3. rate() reset handling. A counter resets mid-window and the previous binary counted the reset sample; the new binary ignores it. The rate shifts. The fix is to add a test case that includes a reset.
4. absent() behaviour change. A new release treats the absent() input vector differently (e.g. respects staleness markers). An alert that depended on the older behaviour stops firing.
5. Test suite silently failing. A CI job ran promtool test rules but did not parse the output. A behavioural regression slipped past because the exit code was masked by a subsequent step. The fix is to set set -euo pipefail at the top of the CI script.
6. Rule group interval mismatch. A rule’s interval (declared in the YAML) does not match the global evaluation_interval. The new binary may honour the group-level interval strictly where the old binary rounded. The fix is to align them.

How to troubleshoot it

The diagnostic order when the upgrade has affected rule behaviour:

1. What does promtool test rules say against the new binary? The first sign of a behavioural problem is a failed test case.
2. What does /api/v1/rules say? The endpoint reports the rule’s health, last evaluation timestamp, and last error. A health: err line identifies the rule and the error string.
3. What does the rule manager log say? kubectl logs ... | grep -E 'rule|evaluat' against the time of the upgrade. The first rule-level error after the swap is the closest indicator.
4. Did a dashboard consume the output? A regression may surface first on a dashboard reading a recording-rule series. Walk back from the panel to the rule.
5. What does the upstream release note say? The [CHANGE] section of each release note lists semantic changes. grep -f the rule files against the listed functions.

Security implications

Rule files have one direct security touchpoint and one indirect one:

• The for: clause and alertmanager routing. A rule that fires more often than the alertmanager can deduplicate creates notification pressure. An upgrade on top of an untested rule file may surface a notification-storm shape that the alertmanager cannot absorb.
• Secrets in annotations. A rule annotation may interpolate a label that contains a credential. The secret-shipping discipline (separate lesson) covers this in detail; the rule-compatibility lesson inherits the same posture (no secrets in labels, no secrets in annotations).

Performance implications

The performance cost of a rule change is on the order of one evaluation interval. Most teams do not see a spike; teams with very large rule groups (hundreds of files, tens of thousands of series) see a longer evaluation wall-clock time on the new binary, visible as evaluation_duration near the evaluation_interval.

Production guidance

• Run promtool check rules against the upgrade image in CI for every rule file.
• Run promtool test rules against the upgrade image in CI for every rule file with a test/ block.
• Track semantic changes in a version-controlled file. The file is the bridge between release notes and the test suite.
• Pin evaluation_interval and per-group interval explicitly. Make them visible in the diff.
• Test rules that depend on external data (recording rules consuming remote-write inputs) need a synthetic sample set as well as a real-data smoke test.
• Document every alert’s for: clause, route, and runbook URL in the rule file. The annotations are the audit record.

Verification

You should now be able to answer:

• What is the difference between a rule syntax error and a rule semantic regression?
• How does promtool test rules validate a rule that promtool check rules cannot?
• Why should the CI test suite run against the upgrade image, not the running one?
• How does the operator identify which PromQL functions changed between 2.51 and 2.55?

Quiz