Rule Test Anatomy

An engineer refactors a recording rule to add a region label to the aggregation level. The local promtool test rules run passes. The change merges. Twenty minutes later, the latency dashboard flashes empty across every panel that consumed the recorded metric. The CI was green. The merge was clean. The production break was loud.

The fixture passed because the test only asserted that the recorded metric existed, not that its label set matched what the dashboards expected. The unit-test file format was correct; the test it described was useless. This is the anatomy lesson: the format is small, the format is strict, and a fixture that uses the format without thinking about what it asserts is worse than no fixture at all.

What it is

A rule unit test is a YAML file that promtool test rules consumes to evaluate one or more rule files against synthetic time series and assert on the produced alerts or metric values. The test is offline. No Prometheus needs to run. No scrape needs to happen. The fixture is the production environment, miniaturised and frozen.

The conventional layout:

rules/
  orders-api.yml         <-- the rule file under test
  test/
    orders-api_test.yml  <-- the unit-test fixture

The convention is not enforced by promtool. It is enforced by the team. A consistent layout makes CI globs trivial and makes the relationship between rule and test obvious from a directory listing.

The fixture is one YAML document. Four top-level keys appear in every well-formed fixture:

• rule_files: — list of paths to the rule files under test.
• evaluation_interval: — the cadence at which the in-memory rule manager advances the clock.
• tests: — list of named scenarios. Each scenario declares the input series and the assertion.
• Per-test interval: — overrides the global cadence for the scenario. Useful when one scenario wants a faster clock than another.

The assertion block inside each test depends on the rule type:

• Recording rules assert on the produced time series. The block is unit_test: and contains exp_series: with a list of metrics: (name, labels, values).
• Alerting rules assert on the produced alert state. The block is alert_rule_test: and contains exp_alerts: with a list of label / annotation pairs.

A single fixture may mix recording and alerting rule files and use both assertion blocks across different tests. The block choice is per-test, not per-fixture.

Why a sysadmin cares

Every alerting rule that reaches production is code that wakes a human at some point in the future. The cost of the rule being wrong is paid in minutes of sleep, in minutes of incident response chasing a false alarm, and in credibility when the rota learns to ignore a page because it has been wrong before. The unit test is the only layer that catches a rule that parses, evaluates, and ships, but fires under the wrong conditions.

Recording rules are caches. A wrong value in the cache poisons every consumer downstream: dashboards, alerts, SLO reports, executive scorecards. The unit test is the only layer that catches a recording rule that parses, evaluates, and ships, but produces a number that disagrees with what its expr: would return against the real TSDB.

A team that adopts promtool test rules discipline catches both classes of mistake before they reach production. A team that does not adopts the consequences.

How it works

The flow is the same for recording and alerting rules; only the assertion shape changes.

  rule_files: [orders-api.yml]
        |
        v
  in-memory storage seeded with input_series
        |
        v
  rule manager starts with evaluation_interval
        |
        v
  for each test:
        |   tick the clock to eval_time
        |   evaluate the rule against the seeded storage
        |   assert on the produced alert or series
        v
  report PASS or FAIL with diff

The clock advances deterministically. The first eval_time in a test is the first tick. The rule manager evaluates the rule at that timestamp, then at every subsequent eval_time, and compares the produced state against the exp_alerts: or exp_series: block.

For alerting rules, promtool evaluates the for: dwell for every evaluation: a pending alert only fires after it has been pending for the configured duration. A test that asserts firing at eval_time: 4m against a rule with for: 5m will report the alert as pending, not firing. The test must wait.

For recording rules, there is no for: to wait through. The metric value at eval_time is what the rule produced for that tick. The test asserts the value directly.

How to configure it

A fixture file lives next to the rule file it tests. The convention is test/<rule_file>_test.yml. The CI step walks the directory and runs promtool test rules against each.

A worked fixture that covers both recording and alerting rules for a hypothetical orders-api service:

# rules/test/orders-api_test.yml
rule_files:
  - ../orders-api.yml

evaluation_interval: 30s

tests:
  # ---- Recording rule test ----
  # The rule under test:
  #   record: job:http_requests:rate5m
  #   expr:   sum by (job) (rate(http_requests_total[5m]))
  - interval: 30s
    name: recording rule produces per-job rate
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        # 10 increments of 1 every 30s for 5m
        values: '0+1x10'
    unit_test:
      - eval_time: 5m
        exp_series:
          - metrics:
              - name: job:http_requests:rate5m
                labels: 'job="orders-api"'
                # rate over [5m] is the count of increments / 300s
                # 10 samples / 300s = 0.0333 per second
                value: 0.03333333333333333

  # ---- Alerting rule test: under threshold ----
  # The rule under test:
  #   alert:  OrdersApiHighErrorRate
  #   expr:   job:http_requests_error:ratio5m > 0.05
  #   for:    5m
  - interval: 30s
    name: error ratio at 1% does not fire
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        values: '0+100x10'
      - series: 'http_requests_total{job="orders-api",status="500"}'
        values: '0+1x10'
    alert_rule_test:
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        exp_alerts: []

  # ---- Alerting rule test: above threshold ----
  - interval: 30s
    name: error ratio at 10% fires after 5m dwell
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        values: '0+100x10'
      - series: 'http_requests_total{job="orders-api",status="500"}'
        values: '0+10x10'
    alert_rule_test:
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        exp_alerts:
          - exp_labels:
              severity: critical
              team: checkout
              job: orders-api
            exp_annotations:
              summary: 'orders-api error ratio above 5% (job=orders-api)'
              runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

Six things to notice:

• rule_files: points at ../orders-api.yml. The path is relative to the fixture file. CI is run from the repo root, so the relative path works for both the developer laptop and the CI runner.
• input_series: values use the shorthand syntax 0+1x10 for “start at 0, then increment by 1 ten times, one sample per interval”. The full form is 0 1 2 3 4 5 6 7 8 9 10. The shorthand is easier to read for monotonic counters.
• The recording-rule test asserts a specific numeric value (0.03333333333333333). promtool does not approximate; the value must match. Use the exact float the rule produces for the synthetic input.
• The under-threshold alerting test asserts exp_alerts: [] (empty list). No alert should exist at eval_time: 6m.
• The above-threshold alerting test asserts the full label set and annotation set. A reviewer who changes severity or team without updating the fixture will see the CI go red.
• The eval_time: 6m aligns with the for: 5m dwell. promtool starts the dwell at the first evaluation tick where the condition is true; firing happens after for: has elapsed.

How to validate it

Three checks. The first two are the static and dynamic checks on the fixture itself; the third is the live confirmation on the running Prometheus.

# 1. The rule files under test still parse.
promtool check rules rules/orders-api.yml

Expected output:

Checking rules/orders-api.yml
  SUCCESS: found 1 rules, 1 alerts

# 2. The fixture passes against the rule files.
promtool test rules rules/test/orders-api_test.yml

Expected output:

SUCCESS

A failure produces a diff:

FAILED
  test: error ratio at 10% fires after 5m dwell
  eval_time: 6m
    expected: OrdersApiHighErrorRate firing
    actual:   no alerts

The diff names the test, the eval_time of the failure, the expected alert state, and the actual alert state. Adjust the rule or the fixture until they agree.

# 3. The rule is loaded in the running Prometheus.
curl -s http://localhost:9090/api/v1/rules \
  | jq '.data.groups[].rules[]
        | select(.name | test("^job:http_requests:rate5m|OrdersApiHighErrorRate"))
        | {name, health, lastError}'

Expected output:

{
  "name": "job:http_requests:rate5m",
  "health": "ok",
  "lastError": ""
}

health: ok and empty lastError are the green light. A populated lastError means the rule loaded but failed to evaluate on the most recent tick; investigate the live TSDB state rather than the fixture.

How it can fail

Six fixture-format mistakes appear repeatedly in real CI logs.

1. rule_files: path does not resolve. Symptom: promtool test rules reports no rule files and exits non-zero. Cause: the relative path is wrong (trailing slash, missing ..), or the CI runner runs from a different directory. Fix: make rule_files: absolute (/etc/prometheus/rules/orders-api.yml) or confirm the CI step cds into the rule directory.
2. input_series: series name does not match the rule’s expr: selector. Symptom: the rule produces no output, the test reports no alerts (or empty exp_series). Cause: typo in the series label set, or a label that the selector filters out. Fix: render the series string from the actual expr: and copy it verbatim.
3. eval_time falls before the first sample. Symptom: promtool reports error: no samples for series at eval_time. Cause: the values: string ends before eval_time. Fix: extend the synthetic series or move eval_time earlier.
4. exp_alerts: lists a label that the rule never sets. Symptom: the test fails with expected label not found. Cause: the rule’s expr: does not include the label (for example, region was removed in a refactor). Fix: update exp_alerts: to match the rule’s actual output.
5. unit_test: block asserts presence, not value. Symptom: the test passes even when the rule produces a wildly wrong number. Cause: exp_series: lists the metric but omits value:, or value: is 0 as a placeholder. Fix: assert the actual numeric value the rule produces for the input.
6. Wrong for: dwell calculation. Symptom: the rule fires in production but the fixture reports exp_alerts: [], or vice versa. Cause: the test author miscounted the eval_time ticks against the dwell. Fix: walk through the rule manager’s first evaluation where the condition is true, add for:, and assert firing only after that tick.

How to troubleshoot it

In order:

1. Does the rule file parse? promtool check rules rules/orders-api.yml. A parse error here means the fixture will also fail; fix the rule first.
2. Does the fixture resolve? promtool test rules rules/test/orders-api_test.yml with --debug if your build supports it. The output names the offending file and the failing assertion.
3. Do the synthetic series resolve against the selector? Walk the rule’s expr: and confirm each label set in input_series: matches. A label the selector filters out is invisible to the rule.
4. Does the timing add up? Confirm that eval_time is after the synthetic series has data and after for: dwell has elapsed. The arithmetic is the most common fixture mistake.
5. Does the assertion match the rule’s output? Compute the rule’s expr: by hand against the synthetic series for the test’s eval_time. The number you compute is what exp_series: should contain.
6. Does the rule still load in production? After the fixture passes, confirm /api/v1/rules still lists the rule with health: ok.

Security implications

Rule fixtures are committed to the repository alongside the rules they test. Three exposures matter.

1. Fixtures may reference real-looking secrets. An exp_annotations: block that hardcodes a Slack webhook URL or an API key commits that secret to git history. Use placeholder values (https://runbooks.example.com/...) and confirm with a secret scanner in CI.
2. Fixtures expose internal topology. A series string like http_requests_total{job="payments-prod-eu-west-1",status="500"} reveals service names, regions, and label conventions that an attacker would otherwise have to discover. Treat fixture files with the same disclosure posture as dashboards.
3. promtool test rules does not authenticate. The tool is local. There is no remote endpoint. The risk is in the fixture content, not in the runner.

Performance implications

The unit-test runner is fast. A fixture with three scenarios and ten samples each runs in milliseconds on a developer laptop. The cost is in CI minutes, not in production CPU.

Two costs to watch:

• Long input_series strings. A test that uses hundreds of samples slows the runner. Keep fixtures small: enough samples to exercise the rule, no more.
• Large rule_files: lists. A fixture that imports every rule file in the repository re-evaluates every rule for every test. Split fixtures by domain (one per service) so a single CI run only evaluates the rules under test.

A typical CI step that walks every fixture under rules/test/ finishes in seconds.

Production guidance

• One fixture per rule file. Name it test/<rule_file>_test.yml. CI globs rules/test/*_test.yml and runs promtool test rules against each.
• Pin the promtool version in CI to match production. A two-version drift (CI on 2.54, production on 2.55) catches you out. The action prometheus/promtool-github-action accepts a version argument; pin it.
• Use relative paths in rule_files:. A fixture committed to the repository should run on any machine without edits. Relative paths from the fixture file achieve that.
• Cover both recording and alerting tests in the same fixture when a rule file contains both. A promtool test rules run executes every test in the fixture; mixing the block types is the natural way to assert the rule group as a whole.
• Keep eval_time ticks small in number but sufficient in duration. Ten ticks at 30s cover five minutes, which is enough for most for: dwells. A hundred-tick fixture does not exercise the rule better; it just slows the runner.

Verification

You should now be able to answer:

• What are the four top-level keys of a promtool test rules fixture, and what does each do?
• What is the difference between the unit_test: block (used for recording rules) and the alert_rule_test: block (used for alerting rules)?
• Why does an alerting-rule test need to wait through for: dwell, and what is the timing arithmetic?
• What is the conventional directory layout for a rule file and its fixture, and how does the CI step find the fixture?
• What does a passing fixture prove, and what does a fixture that asserts only “presence” prove instead?

Quiz