Test Cases Per Rule

A team inherits a rule set with one test per rule. The test file shows three scenarios per rule: an alert firing at the threshold, an alert not firing below the threshold, and a recording rule producing a value. The CI is green. Six months later, an alerting rule pages the rota at 03:00 for a problem the alert was designed to ignore. The investigation finds that the rule was refactored four months ago; the threshold became a strict greater-than; the original test asserted only the firing case, never the firing-only-above-threshold case. The test passed because the happy path still fired. The failure shape lived in the path no test covered.

This is what the test-cases lesson exists to prevent. A single test per rule is the false sense of security. The unit test is useful only when it covers the canonical scenarios and the edge cases that the rule’s shape implies.

What it is

A rule test case is a named scenario in the fixture’s tests: list that drives a specific input shape through the rule and asserts on the output. The canonical taxonomy has four cases that map directly to the rule’s intended behaviour:

• True positive. Input above the threshold; alert fires (recording rule: output value matches). This is the happy path the rule was written for.
• True negative. Input below the threshold; alert does not fire (recording rule: output value matches the expected sub-threshold figure).
• False positive guard. Input that looks like it should fire but should not: counter resets, missing data, label matches outside the rule’s scope. The guard asserts the rule stays quiet.
• False negative guard. Input that looks like it should stay quiet but should not: a sustained breach across the for: dwell, a second region crossing the threshold. The guard asserts the rule fires for the right reason, not just any reason.

Beyond the canonical four, a rule’s shape implies edge cases worth covering:

• Boundary. Input at the threshold itself (> 0.05 versus >= 0.05). One extra test catches the off-by-one mistake.
• Counter reset. A counter that drops to zero mid-window because the exporter restarted. rate() should detect the reset; a poorly-written rule will undercount.
• Absent data. The source metric disappears entirely. A rule built with absent() overlay produces a synthetic zero; a rule built without it goes silent.
• Empty aggregation. sum by (region) (...) against a region with zero matching series. The aggregation should emit no series; a poorly-written rule can emit a stray zero.

A rule with one test covers the happy path. A rule with the canonical four covers the contract. A rule with the canonical four plus the shape-appropriate edge cases covers the failure modes the rule is most likely to encounter in production.

Why a sysadmin cares

A test case is not a chore; it is a specification. The canonical four cases together say: “this rule fires only above the threshold, only after the dwell, only for the label sets the rule is scoped to, and only when the data is actually present.” A team that adopts the canonical four as the default turns the test suite into the rule’s contract with the on-call rota.

The trade-off is real. A fixture with five scenarios per rule is five times the work of one scenario. The discipline is to weigh that cost against the cost of a misfiring rule. For a rule that pages a human, the cost of misfiring is minutes of incident response, possibly hours. For a rule that powers a dashboard panel, the cost is a misread graph. The number of scenarios should track the consequence of the rule being wrong, not the rule’s line count.

How it works

The fixture’s tests: list is a sequence of independent scenarios. Each scenario has its own interval:, its own input_series:, and its own assertion block. Failure of one scenario does not affect the others; promtool reports each scenario by name and stops at the first divergence within it.

The shape of a complete fixture for one alerting rule:

tests:
  - true positive      <-- threshold crossed, fires after dwell
  - true negative      <-- threshold not crossed, never fires
  - false pos guard    <-- counter reset, dwell-violating input
  - false neg guard    <-- sustained breach for label X, not Y
  - boundary           <-- input exactly at threshold
  - absent data        <-- source metric disappears

For a recording rule, the assertions move from alert states to numeric values, but the same six scenarios apply: above threshold, below threshold, counter reset (does rate() see it), boundary value, sustained production (does the recorded metric stay stable), absent data (does the recorded metric disappear or stay stale).

How to configure it

A worked fixture for a single alerting rule, with the four canonical scenarios and three edge cases:

# rules/test/orders-api_test.yml
rule_files:
  - ../orders-api.yml

evaluation_interval: 30s

tests:
  # ---- 1. True positive ----
  # 10% error rate sustained for 6m; the rule should fire at 6m
  # (for: 5m starts at the first tick where condition is true).
  - interval: 30s
    name: true positive - sustained breach fires after dwell
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        values: '0+100x12'
      - series: 'http_requests_total{job="orders-api",status="500"}'
        values: '0+10x12'
    alert_rule_test:
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        exp_alerts:
          - exp_labels:
              severity: critical
              team: checkout
              job: orders-api
            exp_annotations:
              summary: 'orders-api error ratio above 5% (job=orders-api)'
              runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

  # ---- 2. True negative ----
  # 1% error rate; ratio stays below 5%; the rule must not fire.
  - interval: 30s
    name: true negative - sub-threshold is silent
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        values: '0+100x12'
      - series: 'http_requests_total{job="orders-api",status="500"}'
        values: '0+1x12'
    alert_rule_test:
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        exp_alerts: []

  # ---- 3. False positive guard ----
  # Brief spike to 20% then back to 1%; never sustained long
  # enough to satisfy for: 5m; the rule must not fire.
  - interval: 30s
    name: false positive guard - brief spike does not fire
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        values: '0+100x12'
      - series: 'http_requests_total{job="orders-api",status="500"}'
        # spike for 2m then back to 1
        values: '0+10x4 0+1x8'
    alert_rule_test:
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        exp_alerts: []

  # ---- 4. False negative guard ----
  # Another job crosses the threshold; this rule must not fire
  # for that job because the rule's selector excludes it.
  - interval: 30s
    name: false negative guard - other jobs do not trigger this alert
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        values: '0+100x12'
      - series: 'http_requests_total{job="orders-api",status="500"}'
        values: '0+1x12'
      - series: 'http_requests_total{job="checkout-api",status="500"}'
        values: '0+100x12'
      - series: 'http_requests_total{job="checkout-api",status="200"}'
        values: '0+100x12'
    alert_rule_test:
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        exp_alerts: []

  # ---- 5. Boundary ----
  # Error ratio exactly at 5%. Strict greater-than must not fire;
  # greater-than-or-equal would fire. Pick the operator that
  # matches production and assert accordingly.
  - interval: 30s
    name: boundary - ratio at 5% depends on operator
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        values: '0+95x12'
      - series: 'http_requests_total{job="orders-api",status="500"}'
        values: '0+5x12'
    alert_rule_test:
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        # Adjust exp_alerts based on whether the rule uses > or >=.
        exp_alerts: []

  # ---- 6. Counter reset ----
  # Counter drops to 0 mid-window; rate() should detect the reset
  # and produce a high value, then the rule should fire.
  - interval: 30s
    name: counter reset - rate() detects and rule fires
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        values: '0+100x6 0+100x6'
      - series: 'http_requests_total{job="orders-api",status="500"}'
        values: '0+10x6 0+10x6'
    alert_rule_test:
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        exp_alerts:
          - exp_labels:
              severity: critical
              team: checkout
              job: orders-api

  # ---- 7. Absent data ----
  # Source metric disappears; the rule produces no result; the
  # alert must NOT fire because absent data is not a breach.
  - interval: 30s
    name: absent data - rule is silent when source is missing
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        values: '0+100x6'
      - series: 'http_requests_total{job="orders-api",status="500"}'
        values: '0+10x6'
      # After 3m, the 500 series stops reporting.
    alert_rule_test:
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        exp_alerts: []

Seven scenarios. The fixture runs in milliseconds. The CI run that includes this fixture blocks a rule change that would break any of the seven cases.

How to validate it

Two checks. The first confirms the fixture passes against the current rule; the second confirms each scenario asserts on a distinct shape.

# 1. All scenarios pass.
promtool test rules rules/test/orders-api_test.yml

Expected output:

SUCCESS

A failure names the failing scenario and the assertion that diverged:

FAILED
  test: true positive - sustained breach fires after dwell
  eval_time: 6m
    expected: OrdersApiHighErrorRate firing
    actual:   no alerts

# 2. Walk each scenario name and confirm each covers a
#    distinct shape. A review step, not an automated check.
grep -E 'name:' rules/test/orders-api_test.yml

Expected output: a list of named scenarios, each covering a distinct shape. A fixture whose scenarios all look the same is the failure shape in disguise.

How it can fail

Six common mistakes:

1. Only the happy path is tested. Symptom: the rule passes CI, fires in production for the wrong reason. Cause: the rule author assumed the firing case proves the rule. Fix: add the canonical four.
2. No counter-reset test. Symptom: a rule that depends on rate() produces wrong values after an exporter restart; the test never noticed because the fixture never reset a counter. Fix: add a scenario where one series drops to zero mid-window.
3. No absent-data test. Symptom: the source metric disappears; the alert goes silent; no one notices because the rule never asserted behaviour on absence. Fix: add a scenario where the source stops reporting after a few minutes; assert the alert does not fire.
4. Boundary test asserts the wrong operator. Symptom: threshold-related bugs surface only at the boundary. Cause: the rule uses > and the boundary test asserts firing at exactly the threshold. Fix: align the test to the rule’s operator and document the choice.
5. Synthetic series do not match production cardinality. Symptom: the rule passes CI but production evaluation is slow or fails. Cause: the fixture uses one series per label; production has thousands. Fix: add scenarios that exercise larger fan-outs.
6. Scenarios share state by accident. Symptom: a test fails after an unrelated scenario is added; investigation shows the new test depends on input series declared under a previous test. Cause: the runner does isolate scenarios but the fixture’s input_series: is being misread. Fix: re-read the runner’s per-scenario isolation contract and declare every series the scenario needs.

How to troubleshoot it

In order:

1. Run the fixture with --debug if available. The runner names the failing scenario and the assertion that diverged. Start from there.
2. Walk the failing scenario’s input_series: and confirm each series is declared. A missing series turns the rule’s selector into an empty result.
3. Walk the timing. Confirm eval_time is after the last sample in values: and after for: has elapsed for firing assertions.
4. Compute the rule’s expr: by hand against the synthetic series for the failing eval_time. The number you compute is what exp_alerts: or exp_series: should contain. A mismatch means either the rule or the fixture is wrong; determine which by reading the rule carefully.
5. Run only the failing scenario. Move it to its own fixture file temporarily, run promtool test rules against that file, and iterate. Restore the file once the scenario passes.
6. Confirm the scenario is testing what its name: says it is testing. A scenario named “false positive guard” that asserts firing is mislabeled. Rename it or rewrite it.

Security implications

Test cases multiply the number of fixture files and the number of strings inside them. Three exposures matter.

1. Fixture content reveals internal topology. A scenario for http_requests_total{job="payments-prod-eu-west-1"} commits service names, regions, and label conventions to the repository. Treat fixture content with the same disclosure posture as dashboards.
2. Edge-case scenarios can leak boundary behaviour. A scenario that asserts a threshold of exactly 5% reveals the rule’s threshold to anyone with repo access. That is acceptable for an internal team; it is not acceptable if the repository is public.
3. The fixture does not introduce a new attack surface. The runner is local; there is no remote endpoint; there is no authentication. The risk is in the fixture content.

Performance implications

A fixture with seven scenarios and twelve samples per series runs in milliseconds. The cost is in CI minutes. A CI step that walks every fixture under rules/test/ adds seconds to the pipeline run, not minutes.

The cost grows linearly with scenario count and series count. A fixture with thirty scenarios and hundred-sample series runs in hundreds of milliseconds; still fast. A fixture that loads every rule file in the repository and asserts on every scenario is the failure shape; split the fixtures by domain.

Production guidance

• Adopt the canonical four as the default. Every alerting rule gets at least true positive, true negative, false positive guard, and false negative guard. Recording rules get the value-equivalent.
• Add the edge cases the rule’s shape implies. A counter-based rule gets a counter-reset scenario. A rule built with absent() overlay gets an absent-data scenario. A rule with a strict threshold gets a boundary scenario.
• One scenario, one name:. The name says what the scenario asserts; the assertion matches the name. A reader who scans the name: list should understand the fixture’s coverage at a glance.
• Reject PRs that shrink coverage. A code-review check that fails when a scenario is removed without an accompanying rule change keeps the discipline over time.
• Re-author fixtures after every production incident. A rule that pages for the wrong reason is a rule whose fixture is missing a scenario. The post-incident review adds the scenario before the fix merges.

Verification

You should now be able to answer:

• What are the four canonical test scenarios for an alerting rule, and what does each prove?
• Why is the false positive guard the scenario that catches the most expensive production failure?
• What edge cases apply to a counter-based rule, and how does a counter-reset scenario expose a rate() bug?
• Why is per-scenario isolation in the runner important for authoring canonical fixtures?
• What is the trade-off between scenario count and CI run time, and where does the line fall for a rule that pages humans?

Quiz