promtool test rules

A team inherits 240 alerting rules from a contractor who left two years ago. Half have no runbook URL. A third reference metrics that no longer exist. None have unit tests. The team spent six weeks auditing the rule set by hand, and burned three quarters of that auditing rules that turned out to be wrong in obvious ways a unit test would have caught — a rule with a misplaced > sign, a rule whose for: clause was written in seconds instead of minutes, a rule whose expression referenced a metric that was renamed in the exporter upgrade eighteen months ago. The lesson is that rule testing is not optional discipline. It is the same discipline as unit testing application code: cheap to automate, expensive to skip.

What it is

promtool test rules <fixture.yml> evaluates the rules referenced in the fixture against synthetic time series and asserts the expected alert states at specified evaluation timestamps. The fixture is YAML, the synthetic series are defined inline, and the assertions cover both the alert’s firing state and its labels and annotations.

The fixture’s anatomy:

rule_files:
  - ../app-checkout.yml         # relative to the fixture file

evaluation_interval: 30s        # the interval used to advance
                                # the clock between eval_time steps

tests:
  - interval: 30s               # the interval for this test's
                                # input series
    input_series:               # synthetic time series
      - series: 'http_requests_total{...}'
        values: '0 100 100 100 100 ...'
    alert_rule_test:            # assertions for alerting rules
      - eval_time: 4m
        alertname: OrdersApiHighErrorRate
        exp_alerts: []          # no alert expected at this time
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        exp_alerts:
          - exp_labels:
              severity: critical
              team: checkout
            exp_annotations:
              summary: 'orders-api 5xx ratio above 5% in eu-west-1'
              runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'
    promql_expr_test:          # assertions for recording rules
      - expr: job:up:avg5m
        eval_time: 1m
        exp_samples:
          - labels: 'job=node'
            value: 1

Three properties:

1. It runs offline. The test loads the synthetic series into an in-memory storage.Storage; no network, no live Prometheus. The fixture runs in milliseconds on commodity hardware.
2. It uses the same evaluation engine as the running daemon. A rule that passes the unit test will produce the expected output under the simulated conditions in production.
3. It is read-only. The test does not touch the running Prometheus, does not need credentials, and does not require any external state.

Why a sysadmin cares

The unit test is the only gate that catches a rule that parses correctly but does the wrong thing. Five shapes that the static checks miss but the unit test catches:

1. > instead of >=. A rule with errors > 0 fires on the first error and stops; a rule with errors >= 0 fires on every scrape. Both parse; only one is right. The unit test that asserts the alert state at a known input catches the off-by-one.
2. for: 5s instead of for: 5m. A rule with a for: value in seconds flaps on every scrape interval. The static check passes; the unit test that spans the for: window catches the flap.
3. Expression references a missing metric. The static check passes because PromQL syntax is valid; the unit test fails because no series exist for the expression. The test failure names the missing metric.
4. Labels and annotations are wrong. A rule whose summary: contains a typo, or whose labels: are missing severity or team, parses fine. The unit test’s exp_labels: and exp_annotations: blocks catch the mistake before production.
5. Recording rule produces the wrong value. A recording rule whose expr is logically wrong (missing by clause, inverted ratio) parses fine and produces a series, but the series is not what the downstream alerts and dashboards expect. The promql_expr_test block catches the wrong value.

The unit test is the only gate that catches each of these five shapes. The static checks are necessary but not sufficient.

How it works

  promtool test rules test/app-checkout_test.yml
        |
        v
  Load rule_files: [../app-checkout.yml]
        |
        v
  Build an in-memory storage.Storage
        |
        v
  For each test:
    For each input_series:
      generate samples at the named interval, using the
      named values (a whitespace-separated list, with
      special tokens like _ for the previous value)
        |
        v
    Advance the clock to each eval_time
        |
        v
    For each alert_rule_test:
      Run the named rule's expr against the synthetic series
      at eval_time
      Compare the resulting alert states (firing, pending,
      inactive) against exp_alerts
        |
        v
    For each promql_expr_test:
      Run the named expr against the synthetic series
      at eval_time
      Compare the result samples against exp_samples
        |
        v
  On any mismatch: print file:line and the diff, exit 1
  On all tests pass: print SUCCESS, exit 0

The test uses the same PromQL evaluator and the same template engine as the running daemon. A test that asserts the alert’s summary: matches the expected string catches both the typo and the wrong-label mistake.

How to configure it

The fixture file is the configuration. The CI step that runs the test is a shell loop.

The rule file, abridged:

# observability/prometheus/rules/app-checkout.yml
groups:
  - name: checkout-slo
    interval: 30s
    rules:
      - alert: OrdersApiHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api",
                                     status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: checkout
        annotations:
          summary: 'orders-api 5xx ratio above 5% in {{ $labels.region }}'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

The fixture, in observability/prometheus/rules/test/checkout_test.yml:

rule_files:
  - ../app-checkout.yml

evaluation_interval: 30s

tests:
  # Test 1: a 4-minute breach should NOT fire (within for: dwell).
  - interval: 30s
    input_series:
      - series: 'http_requests_total{service="orders-api",region="eu-west-1",status="200"}'
        values: '0 100 100 100 100 100 100 100 100'
      - series: 'http_requests_total{service="orders-api",region="eu-west-1",status="500"}'
        values: '0 1 1 1 1 1 1 1 1'
    alert_rule_test:
      - eval_time: 4m
        alertname: OrdersApiHighErrorRate
        exp_alerts: []

  # Test 2: a 6-minute breach SHOULD fire (past for: dwell).
  - interval: 30s
    input_series:
      - series: 'http_requests_total{service="orders-api",region="eu-west-1",status="200"}'
        values: '0 100 100 100 100 100 100 100 100'
      - series: 'http_requests_total{service="orders-api",region="eu-west-1",status="500"}'
        values: '0 10 10 10 10 10 10 10 10'
    alert_rule_test:
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        exp_alerts:
          - exp_labels:
              severity: critical
              team: checkout
              service: orders-api
              region: eu-west-1
            exp_annotations:
              summary: 'orders-api 5xx ratio above 5% in eu-west-1'
              runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

  # Test 3: a recording rule that should produce a known value.
  - interval: 30s
    input_series:
      - series: 'up{job="node",instance="host-a"}'
        values: '0+1x20'
      - series: 'up{job="node",instance="host-b"}'
        values: '0+1x20'
    promql_expr_test:
      - expr: avg by (job) (up)
        eval_time: 5m
        exp_samples:
          - labels: 'job=node'
            value: 1

Three things to notice:

• Test 1 has a 1% error rate (1 in 100), which is below the 5% threshold. The alert should not fire at 4m (within the for: 5m dwell). exp_alerts: [] asserts no alert exists.
• Test 2 has a 10% error rate (10 in 100), which is above the threshold. The alert should fire at 6m (past the dwell). exp_alerts: lists the expected alert with the expected labels and annotations.
• Test 3 is a promql_expr_test for a recording rule. It asserts the recording rule’s output at eval_time: 5m matches the expected samples.

The eval_time values must be valid multiples of the evaluation_interval. With evaluation_interval: 30s and a test at eval_time: 6m, the test runner advances the clock through the same evaluations a real Prometheus would, with the synthetic series supplying the samples.

The GitHub Actions job that runs the test:

# .github/workflows/promtool-test.yml
name: promtool-test-rules

on:
  pull_request:
    paths:
      - 'observability/prometheus/rules/**'
      - '.github/workflows/promtool-test.yml'

permissions:
  contents: read

jobs:
  test:
    name: promtool test rules
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install promtool
        run: |
          PROMTOOL_VERSION=2.55.1
          curl -sSL \
            "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/prometheus-${PROMTOOL_VERSION}.linux-amd64.tar.gz" \
            | tar xz -C /tmp
          sudo mv \
            /tmp/prometheus-${PROMTOOL_VERSION}.linux-amd64/promtool \
            /usr/local/bin/
      - name: Run rule tests
        run: |
          set -e
          for f in observability/prometheus/rules/test/*.yml; do
            echo "Testing $f"
            promtool test rules "$f"
          done

A failed test exits non-zero; the CI gate blocks the merge.

How to validate it

Three checks confirm the gate is wired correctly.

1. The test passes against a known-good fixture.

promtool test rules observability/prometheus/rules/test/checkout_test.yml

Expected output, exit 0:

SUCCESS

A SUCCESS with no further output means every test in the fixture passed.

2. The test fails against a deliberately broken rule.

Edit the rule’s expr to swap > with >=:

expr: ... > 0.05

becomes

expr: ... >= 0.05

Rerun the test. Expected output, exit 1:

FAILED:
  unit test "Test 2: 6-minute breach SHOULD fire" failed:
    expected 1 alerts, got 0

The error names the test, the expected alert count, and the actual count. The fix is to revert the rule and update the test if the new behaviour is correct.

3. The lint pass catches a missing exp_labels: field.

Edit a fixture to omit exp_labels.team:

exp_alerts:
  - exp_labels:
      severity: critical
      # team: checkout   (removed for the test)

Rerun the test. The actual alert has the team: checkout label (because the rule sets it), but the expected list does not. The mismatch fails the test.

How it can fail

Six failure modes specific to unit testing rules:

1. The fixture’s eval_time does not span the for: window. Symptom: the test asserts the alert should be firing at eval_time: 2m, but the rule’s for: 5m means the alert is still pending at 2m. The test fails. Cause: the fixture’s time window is shorter than the rule’s for: value. Either lengthen the test window (extend input_series values) or shorten the rule’s for: value.

2. Two tests share state through the in-memory storage. Symptom: a test passes when run alone but fails when run after another test. Cause: the promtool test rules runner creates a new in-memory storage per fixture file, not per test. Tests inside the same fixture file share the storage if they reuse label sets. The simplest fix is to give each test a unique label value (for example, region=eu-west-1 versus region=us-east-1).

3. The fixture references the wrong rule file path. Symptom: the test reports “rule not found” for every alertname in the fixture. Cause: the path in rule_files: is wrong relative to the fixture file’s directory. Paths are relative to the fixture file, not to the current working directory.

4. The rule’s annotation template references a label that does not exist. Symptom: the test fails on exp_annotations: because the expanded summary does not match. Cause: the rule’s summary: template uses {{ $labels.foo }} but foo is not in the rule’s output labels. Either add foo to the rule’s labels: block or remove it from the template.

5. The fixture’s evaluation_interval does not match the rule group’s interval:. Symptom: the test reports the rule is pending when the fixture asserts firing. Cause: the for: dwell counts in interval: ticks, not in evaluation_interval ticks. If the rule’s group interval is 1m and the fixture’s evaluation_interval is 30s, the dwell is half what the fixture expects. Set the fixture’s evaluation_interval to match the group’s interval:.

6. The synthetic series contains NaN or +Inf. Symptom: the test fails with “division by zero” or “comparison with NaN is always false”. Cause: the values: list contains a special-value token that the PromQL evaluator rejects. Replace NaN with a numeric value (or omit the sample with an empty token).

How to troubleshoot it

In order:

1. Read the test failure output. promtool test rules prints the test name, the expected value, and the actual value. The fix is in the diff.
2. Run one test at a time. Comment out all but one test in the fixture and rerun. Isolating the failing test confirms whether the failure is order-dependent.
3. Check the eval_time progression. Each eval_time must be later than the previous one and a multiple of the evaluation_interval.
4. Check the rule’s interval: and for: against the fixture’s evaluation_interval. A mismatch in either changes the dwell and the firing timestamp.
5. Print the expanded annotation. Add a temporary promtool query instant against a debug fixture to see what the template engine actually produces.

Security implications

• The fixture file may contain realistic-looking credentials. A fixture that simulates a real runbook_url with an embedded token commits that token to the repo. Use placeholder values; verify with a secret scanner.
• The test runner does not connect to the live Prometheus or Alertmanager. It is read-only and does not expose any new endpoint. Safe to run from any workstation.
• The fixture’s input_series should not contain real user data. Use synthetic labels and values; confirm with a privacy review.

Performance implications

The test is fast. A fixture with ten tests and twenty input series runs in well under a second on commodity hardware. The cost is dominated by the in-memory evaluation of each expression at each eval_time. CI budgets the test at 1–5 s per fixture. There is no production cost from running the test; it is purely a pre-merge gate.

Production guidance

• Write a unit test for every alerting rule that pages humans. The cost is small; the benefit is catching the semantic mistake that no static check sees.
• Write a promql_expr_test for every recording rule whose output is consumed by alerts or dashboards. The test asserts the value, not just the existence.
• Keep test fixtures small. A test with ten samples is enough; a test with a thousand samples is slow and unreadable.
• Pin the promtool version in CI to the same version as production. The unit test uses the same evaluator as the daemon; a version drift changes the behaviour.
• Schedule a recurring review of the test fixtures. As the rules change, the fixtures must change with them.

Verification

You should now be able to answer:

• What four blocks does a promtool test rules fixture contain?
• What is the difference between alert_rule_test and promql_expr_test?
• Why does a test that fails in isolation but passes when run after another test indicate shared state?
• Why must the fixture’s evaluation_interval match the rule group’s interval:?
• What is the right cadence for unit-testing alerting rules?

Quiz