Regression Tests

An on-call engineer pages at 03:00 for an alert that fired for the wrong reason. The investigation finds that the rule’s threshold comparison used > when the intent was >=. The edge case at the threshold itself was missed. The engineer fixes the operator, runs the rule through a quick local check, and merges. Six months later, a refactor of the rule replaces the comparison with a different expression. The threshold edge case returns. The rule fires for the wrong reason again. The on-call rota is paged at 04:00.

The fix was correct. The discipline was incomplete. The fix landed without a regression test, so when the refactor came through, the bug had no fixture to fail. The lesson is that every production incident produces a test. The discipline is the workflow that ensures it.

What it is

A regression test is a fixture scenario written after a production incident or bug, with input_series: that reproduces the input shape that exposed the bug and exp_alerts: or exp_series: that locks in the fixed behaviour. The defining property of a regression test is its provenance: it was written in response to a real failure, not in anticipation of a possible failure.

The discipline that produces regression tests is the post-incident workflow. Every production bug, every wrong page, every missed alert is a rule whose fixture is missing or wrong. The fix is two parts: the code change that fixes the bug, and the fixture that proves the fix is correct and that catches the same bug if it returns.

The regression test is not a happy-path test. The happy path is “the rule fires when it should”; the regression test is “the rule does not behave the way it behaved at 03:00 when the input looks like this”. The regression test is also not an edge-case test in the abstract. The edge- case test is “the rule handles boundary, counter reset, and absent data”; the regression test is “the rule does not regress to the specific shape that caused this incident”.

A team that adopts regression tests converts every incident into a permanent test that catches the same shape. Over years, the fixture suite accumulates the institutional memory of every failure the team has had. A new engineer who refactors a rule against the suite is implicitly checking their change against every historical failure shape.

Why a sysadmin cares

A bug that fixed itself is a rare event. Most bugs recur. The same code path that failed once will fail again under similar conditions: a refactor that touches the path, a configuration change that exposes a different input shape, a scale event that pushes the rule against a corner case the original author never considered. The cost of the recurrence is paid in minutes of incident response, often by someone who has never seen the original failure.

A regression test is the cheapest insurance against that cost. The authoring time is the time to reproduce the input shape in a fixture, assert the fixed behaviour, and confirm the fixture would have failed against the pre-fix rule. The benefit is that the next refactor that touches the same code path runs the fixture and either passes (the fix is intact) or fails (the bug is back).

The trade-off is real. A team that adds regression tests to every incident grows the fixture suite. The suite gets larger; the CI gets slower; the per-PR test run gets longer. The discipline is to keep the regression scenarios focused on the specific failure shape, not to expand them into coverage fillers. A regression test for an incident should test exactly the shape that caused the incident, no more.

How it works

The workflow has five steps. The post-incident review drives the first four; the fifth is the verification that the fixture does what it claims.

  incident at 03:00
        |
        v
  1. reproduce: capture the input shape that caused the
     incident (alert that fired, value that was wrong)
        |
        v
  2. fix: change the rule so the failure shape produces
     the correct output
        |
        v
  3. fixture: write a regression scenario with the input
     shape and the fixed assertion
        |
        v
  4. verify the fixture would have caught the bug:
     run it against the pre-fix rule; the fixture must fail
        |
        v
  5. ship the rule change, the fixture, and a PR description
     that links the fixture to the incident

The fourth step is the discipline that distinguishes a regression test from a happy-path test. The fixture must fail against the pre-fix rule. If it passes against the pre-fix rule, the fixture does not actually catch the bug it claims to catch; it is a happy-path test in disguise.

How to configure it

A worked regression test for a real incident: the OrdersApiHighErrorRate alert fired at 03:00 because the threshold comparison used a strict > instead of >=. The input was a service whose error rate sat at exactly 5.0 percent for the entire dwell window. The rule should have fired at exactly 5.0 percent; it did not.

The fixture for the regression:

# rules/test/orders-api_test.yml
rule_files:
  - ../orders-api.yml

evaluation_interval: 30s

tests:
  # ---- Regression test: incident 2026-04-17 ----
  # The rule's threshold comparison was strict >.
  # At error ratio = 5.0 percent (the threshold), the rule
  # did not fire. Service was paging humans for an outage
  # condition the alert was supposed to catch.
  #
  # The fix changed > to >=. This fixture locks the
  # behaviour at the threshold itself.
  - interval: 30s
    name: 'regression 2026-04-17: error ratio at exactly 5% must fire'
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        # 95 increments of 1 every 30s for 5 minutes
        values: '0+95x10'
      - series: 'http_requests_total{job="orders-api",status="500"}'
        # 5 increments of 1 every 30s for 5 minutes
        # Total: 100 requests, 5 errors = exactly 5%
        values: '0+5x10'
    alert_rule_test:
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        exp_alerts:
          - exp_labels:
              severity: critical
              team: checkout
              job: orders-api
            exp_annotations:
              summary: 'orders-api error ratio above 5% (job=orders-api)'
              runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

Five things to notice:

• The scenario name references the incident. A reader who scans the fixture’s name: list sees the date and the shape. The link to the incident doc lives in the PR description.
• The input reproduces the failure shape exactly. The error ratio sits at 5.0 percent for the entire window. This is the precise input that caused the original incident.
• The assertion matches the fixed behaviour. The rule with >= fires at 5.0 percent. The fixture asserts firing.
• The pre-fix verification. A developer who writes this fixture runs it against the pre-fix rule (the one with >). The fixture fails with exp_alerts not empty and actual empty. That confirms the fixture catches the bug.
• The fixture lives next to the canonical scenarios. The regression test is one entry in the fixture’s tests: list, alongside the canonical four. It is not a separate file; it is part of the rule’s contract.

The PR description that ships the fix:

fix: change > to >= in OrdersApiHighErrorRate threshold

Incident: 2026-04-17 03:00 UTC. Service orders-api sat at
exactly 5.0 percent error rate for the dwell window. The
strict > comparison meant the rule did not fire. The page
came from the runbook owner manually, not from the alert.

Fix: change > to >= in the threshold comparison.

Regression test: rules/test/orders-api_test.yml, scenario
'regression 2026-04-17: error ratio at exactly 5% must fire'.
Verified to fail against the pre-fix rule (the rule with
>); passes against the fixed rule.

Refs: incident doc INC-2026-04-17-001.

How to validate it

Three checks.

# 1. The new fixture passes against the fixed rule.
promtool test rules rules/test/orders-api_test.yml

Expected output:

SUCCESS

# 2. The fixture would have caught the bug.
# Save the pre-fix rule, swap it in, run, restore.
git show HEAD~1:rules/orders-api.yml > /tmp/pre-fix.yml
sed -i 's|rule_files:|rule_files:\n  - /tmp/pre-fix.yml|' \
  rules/test/orders-api_test.yml
promtool test rules rules/test/orders-api_test.yml
# Expected: FAILED on the regression scenario with
#   expected: OrdersApiHighErrorRate firing
#   actual:   no alerts
git checkout rules/test/orders-api_test.yml

The check confirms the fixture fails against the pre-fix rule. If it passes, the fixture does not actually catch the bug; rewrite it until it does.

# 3. The fixture is referenced from the PR description
#    and from the incident doc.
grep -r 'regression 2026-04-17' rules/test/ docs/incidents/

Expected output: the fixture name appears in the fixture file and in the incident doc. A regression test that is not discoverable from the incident doc is a regression test that cannot be traced back to the failure it catches.

How it can fail

Six specific failure modes.

1. Fix without a regression test. Symptom: a bug ships a fix but no fixture. The next refactor reintroduces the bug because no fixture catches it. Cause: the post- incident workflow did not require a fixture as part of the fix. Fix: make the fixture a blocking part of the PR; reject the PR until the fixture lands.
2. Fixture passes against the pre-fix rule. Symptom: the regression test is shipped; six months later, the same bug returns; the fixture does not catch it. Cause: the pre-fix verification was skipped. Fix: enforce the pre-fix verification as part of the PR workflow.
3. Fixture reproduces the symptom, not the cause. Symptom: the fixture locks in a behaviour that hides the original cause. The next refactor that fixes the symptom but not the cause still passes the fixture. Cause: the author asserted on the output, not on the shape that exposed the bug. Fix: assert on the input shape that exposed the bug, not on the output the team saw.
4. Fixture uses the wrong labels or the wrong eval_time. Symptom: the fixture passes against the pre-fix rule because the wrong labels or eval_time prevent the assertion from triggering. Cause: the author did not run the pre-fix verification. Fix: run it; the verification catches the mismatch.
5. PR description does not link the fixture to the incident. Symptom: a future engineer cannot trace the fixture back to the failure it catches; the institutional memory is lost. Cause: the PR template does not require the link. Fix: add the link as a blocking field in the PR template.
6. Fixture bloats into a coverage filler. Symptom: the fixture grows scenarios until it duplicates the canonical four. Cause: the author conflated regression test with coverage. Fix: keep the regression scenario focused on the specific failure shape; do not add coverage scenarios to the regression fixture.

How to troubleshoot it

In order:

1. Run the fixture against the current rule. Does it pass? A pass against the current rule is necessary but not sufficient. The next check is whether it would have caught the bug.
2. Run the fixture against the pre-fix rule. Does it fail with the right assertion? A wrong assertion means the fixture is asserting on a different shape than the one that caused the incident.
3. Walk the input shape. Does the input_series: reproduce the failure shape? The error ratio at 5.0 percent for the canonical threshold incident is a specific input. A fixture that uses 10 percent does not reproduce the boundary case.
4. Walk the timing. Does the eval_time align with the for: dwell? A fixture that asserts firing before the dwell has elapsed is asserting pending, not firing.
5. Confirm the PR description links the fixture to the incident. A regression test without a link is a fixture whose provenance is lost.
6. Confirm the incident doc references the fixture. The reverse direction matters: a reader of the incident doc should be able to find the regression test.

Security implications

Regression fixtures are committed to the repository alongside the rules they test. The risks are the same as for any rule fixture: committed secrets, internal topology in series names, and rule expressions that reveal internal naming conventions.

Two additional risks specific to regression tests:

1. Regression fixtures reveal incident details. A scenario named regression 2026-04-17: error ratio at exactly 5% must fire reveals the date and nature of the incident. If the repository is public, that information is disclosed. Treat regression fixture names with the same disclosure posture as incident docs.
2. Regression fixtures can leak the rule’s intended threshold. A fixture that asserts firing at exactly 5 percent reveals the rule’s threshold. A reader with repo access can infer the threshold without reading the rule itself.

Performance implications

A regression fixture adds one scenario to an existing fixture file. The CI cost is the cost of running that scenario, which is milliseconds. The accumulated regression suite over years may add hundreds of scenarios; the CI cost is still seconds.

The hidden cost is in the post-incident workflow. Every incident produces authoring time for the fixture, time to run the pre-fix verification, and time to update the PR template. Plan for that time when budgeting incident response.

Production guidance

• Every production bug produces a regression test. The post-incident review lists the fixture as a blocking artefact of the fix.
• Verify the fixture catches the original bug. Run the pre-fix verification as part of the PR workflow. A regression test that does not fail against the pre-fix rule is not a regression test.
• Link the fixture to the incident. The PR description names the incident doc; the incident doc names the fixture. The link is bidirectional.
• Keep regression scenarios focused. One scenario per incident; one shape per scenario; one assertion per scenario. A regression test that covers three shapes is three regression tests disguised as one.
• Re-author regression fixtures after every refactor of the rule. If the refactor invalidates the input shape, the fixture must change with the rule.
• Reject PRs that ship a bug fix without a fixture. The PR template makes the fixture a blocking field.

Verification

You should now be able to answer:

• What is the difference between a regression test and a happy-path test, and why does the distinction matter?
• What are the five steps of the post-incident workflow, and why is the pre-fix verification the step that makes a regression test a regression test?
• What does a regression fixture look like for an incident that fired at the wrong threshold, and what input reproduces the boundary?
• Why must the PR description link the fixture to the incident doc, and why must the link be bidirectional?
• What is the discipline when a regression fixture bloats into a coverage filler, and how do you keep the scenario focused on the specific failure shape?

Quiz