Rule Test Coverage

A team inherits a 240-rule set from a contractor who left two years ago. The CI reports 95 percent coverage: 228 of 240 rules have at least one unit test. The team trusts the number and moves on. Six months later, an outage affects a tier-one service. The alerting rule that should have paged was missing from the test fixtures entirely. It was one of the twelve uncovered rules. The CI was green. The rota paged was for a different service.

Coverage that tracks “rules with at least one test” hides the gap that matters. The discipline is not a percentage. The discipline is a per-tier expectation that maps the rule’s consequence to the scenarios the fixture must cover.

What it is

Rule test coverage is the discipline of ensuring that every production rule has a fixture that proves it behaves correctly under the scenarios the rule’s shape implies. The discipline has three components:

• Rule-with-test ratio. Does every rule file in the repository have a matching fixture under test/?
• Scenarios-per-rule. How many scenarios does the fixture cover? One scenario is a happy path; the canonical four are a contract; the edge cases are the failure modes.
• Tier-appropriate expectations. Are the rules that page humans covered with the canonical four plus the shape-appropriate edge cases? Are the rules that power dashboards covered with the value-equivalent?

Coverage is not a single number. A team that tracks “95 percent of rules have at least one test” hides the gap that matters: the missing five percent are often the highest- severity rules, the ones nobody has touched in months, the ones whose absence is invisible until production breaks.

A useful coverage model assigns each rule a tier based on its consequence and a minimum scenario count per tier:

A team that adopts the model stops tracking “95 percent have a test” and starts tracking “are tier-1 rules covered with at least five scenarios?” That question exposes the gap the percentage hides.

Why a sysadmin cares

A rule with no test is a rule whose correctness is unverified. The rule may be correct; the rule may be wrong; the team does not know because no fixture ever exercised it. The risk grows with the rule’s consequence. A tier-1 rule that pages humans at 03:00 for the wrong reason is an operational incident. A tier-3 rule that produces a dashboard panel reading “no data” is a missed visualisation.

Coverage is the discipline that converts “I think this rule is correct” into “the fixture proves this rule is correct.” A team that adopts the per-tier model catches the gap in review, before the rule ships, before the wrong page lands in the on-call’s pocket.

The trade-off is real. Coverage is work. A team that adopts the model commits to authoring fixtures for every rule above tier 4 and to re-authoring fixtures when rules change. The benefit is the gap between “the rule exists” and “the rule is correct” closes mechanically rather than by hope.

How it works

Coverage is enforced by three layers:

  per-PR CI step
        |
        |   a) promtool check rules on every rule file
        |   b) promtool test rules on every fixture
        |   c) fixture presence check
        |      (rule file => fixture required)
        v
  merge gate
        |
        v
  coverage report (weekly)
        |
        |   rules with fixtures
        |   scenarios per fixture
        |   tier-appropriate scenario count
        v
  coverage review (quarterly)
        |
        |   tier-1 rules: any with < 5 scenarios?
        |   tier-2 rules: any with < 4 scenarios?
        |   tier-3 rules: any with < 3 scenarios?
        v
  remediation list

The PR-time check is the cheap, fast layer that catches the missing-fixture case as it lands. The weekly report is the layer that catches the rule added before the CI was set up. The quarterly review is the layer that catches the gap the percentages hide: the tier-1 rule that has one scenario and should have five.

How to configure it

A worked coverage check, with a per-tier expectation file and a CI step that enforces it.

The expectation file, in coverage/expectations.yml:

# Per-tier minimum scenario counts. CI fails when a fixture
# does not meet its tier's expectation.
tiers:
  - tier: 1
    description: 'alerts that page humans with severity critical'
    label_selector: 'severity="critical"'
    min_scenarios: 5
  - tier: 2
    description: 'alerts that page humans with severity warning'
    label_selector: 'severity="warning"'
    min_scenarios: 4
  - tier: 3
    description: 'recording rules consumed by alerts or dashboards'
    label_selector: 'kind="recording"'
    min_scenarios: 3
  - tier: 4
    description: 'recording rules consumed by ad-hoc queries'
    label_selector: 'kind="recording-orphan"'
    min_scenarios: 1

The CI step, in .github/workflows/coverage.yml:

name: rule-coverage
on:
  pull_request:
    paths:
      - 'rules/**'
      - 'coverage/**'
      - '.github/workflows/coverage.yml'

jobs:
  fixture-presence:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: every rule file has a fixture
        run: |
          set -euo pipefail
          for rule_file in rules/*.yml; do
            base=$(basename "$rule_file" .yml)
            fixture="rules/test/${base}_test.yml"
            if [ ! -f "$fixture" ]; then
              echo "::error file=$rule_file::missing fixture: $fixture"
              exit 1
            fi
          done
      - name: every tier-1 fixture has at least 5 scenarios
        run: |
          python3 tools/check_scenario_counts.py \
            --expectations coverage/expectations.yml \
            --fixtures-dir rules/test/

The Python tool walks the fixtures, parses each YAML, and confirms the tests: list meets the per-tier expectation. A fixture with three scenarios for a tier-1 rule fails the build with a clear message.

For the weekly coverage report, the same tool produces a markdown table:

| Rule file           | Tier | Scenarios | Status  |
|---------------------|------|-----------|---------|
| orders-api.yml      | 1    | 6         | OK      |
| checkout-api.yml    | 1    | 2         | LOW     |
| payments-api.yml    | 2    | 5         | OK      |
| aggregations.yml    | 3    | 4         | OK      |
| orphan-rule.yml     | 4    | 1         | OK      |

The report names the rule file, the tier, the scenario count, and the status (OK / LOW / MISSING). The weekly review acts on the LOW and MISSING entries.

How to validate it

Three checks.

# 1. Every rule file has a matching fixture.
for rule_file in rules/*.yml; do
  base=$(basename "$rule_file" .yml)
  fixture="rules/test/${base}_test.yml"
  [ -f "$fixture" ] || echo "missing: $fixture"
done

Expected output: empty (no missing fixtures). Any output names a rule file without a fixture; CI must fail.

# 2. Every fixture passes against its rule file.
for f in rules/test/*_test.yml; do
  promtool test rules "$f" >/dev/null \
    || echo "test failure: $f"
done

Expected output: empty (no test failures).

# 3. The per-tier scenario count meets the expectation.
python3 tools/check_scenario_counts.py \
  --expectations coverage/expectations.yml \
  --fixtures-dir rules/test/

Expected output:

OK: 240 rules, 240 fixtures, all tiers meet expectations

A non-zero exit lists the rules whose scenarios are below the per-tier floor.

How it can fail

Six specific failure modes.

1. Coverage tracked as a percentage only. Symptom: the team reports “95 percent of rules have a test” and the missing 5 percent are the highest-severity rules. Cause: the coverage check counts rule-with-test ratio without per-tier weighting. Fix: split the check by tier and enforce the per-tier scenario floor.
2. Fixture presence check not wired into CI. Symptom: a rule is added without a fixture and the merge goes through. Cause: the CI step that checks fixture presence was never added, or was added but not on the path that includes new rule files. Fix: confirm the CI step matches on rules/** and runs on every PR.
3. Per-tier expectation file drifts. Symptom: the expectations say tier-1 needs 5 scenarios, but the team informally agrees tier-1 is fine with 3. Cause: the expectations file is a wiki page rather than a CI input. Fix: store the expectations in the repository as a YAML file that CI reads.
4. Coverage review is annual, not quarterly. Symptom: the gap grows between reviews; the annual review finds dozens of low-coverage tier-1 rules. Cause: the review is too infrequent. Fix: schedule the review quarterly and assign owners per tier.
5. Coverage review tracks count, not consequence. Symptom: the review says “all rules have at least 1 scenario”; the team is satisfied; the gap is invisible. Cause: the review ignores per-tier weighting. Fix: weight by tier in the review output and act on the weighted low-coverage entries first.
6. New labels added but tier expectations not updated. Symptom: a label is added that puts a rule into tier-1, but the fixture still has 2 scenarios. Cause: the tier classification is by hand. Fix: automate the tier classification from the rule’s labels and re-run the coverage check after any label change.

How to troubleshoot it

In order:

1. Run the fixture-presence check. Names the rules without fixtures.
2. Run promtool test rules against every fixture. Names the fixtures that fail against the current rule files.
3. Run the per-tier scenario count tool. Names the fixtures whose scenario count is below the tier floor.
4. Walk the per-tier gap list. For each under-covered rule, classify the gap: is the fixture too thin, or is the tier expectation wrong? A fixture that covers the canonical four for a tier-2 rule is at the floor; a tier-1 rule with the same four is below the floor and needs an edge-case scenario.
5. For rules without fixtures, classify the rule. A rule with no fixture is a rule whose correctness is unverified; either delete the rule or add the fixture. The middle ground (rule ships without fixture, “we will add it later”) is the failure shape.
6. Confirm the tier classification is current. A label change can move a rule between tiers; the coverage tool re-classifies from the rule’s current labels.

Security implications

The coverage check reads the rule directory and the fixture directory. There is no network exposure; the tool runs in CI. Three risks matter.

1. The expectation file reveals internal labels. A label_selector: severity=critical in coverage/expectations.yml reveals the severity values the team uses. Treat the expectation file with the same disclosure posture as rule files.
2. The coverage report reveals which rules are under-covered. A weekly report that lists tier-1 rules below the scenario floor reveals the rules most likely to misfire. The report is internal; the same disclosure posture applies.
3. The coverage check does not introduce a new attack surface. It reads the filesystem; it does not connect to a remote endpoint.

Performance implications

The fixture-presence check walks the rule directory. For a repository with 240 rules, the walk is sub-second. The per-tier scenario count tool parses every fixture YAML; for 240 fixtures, the parse is also sub-second. The cost is in CI seconds, not in production CPU.

A coverage tool that loads every fixture into memory and parses it twice (once for the count, once for the test) is fine for hundreds of rules. For thousands of rules, stream the parsing or split the workload across CI jobs.

Production guidance

• Adopt the per-tier model. Tier 1 (page-critical) gets five scenarios minimum; tier 4 (ad-hoc) gets one. The floor matches the consequence.
• Enforce fixture presence at PR time. A rule file without a matching fixture is a blocking CI failure. The discipline is to wire the check into every PR.
• Track per-tier scenario counts in a weekly report. A percentage hides the gap; a per-tier table exposes it. The report is one column per tier, one row per rule file.
• Schedule a quarterly coverage review. The review acts on the weighted gap: tier-1 rules below the floor are remediated first; tier-4 rules below the floor are deleted or de-prioritised.
• Re-classify tiers after every label change. A label change that moves a rule between tiers must trigger a re-run of the coverage check.
• Reject PRs that shrink coverage. A code-review check that fails when a scenario is removed without a matching rule change keeps the discipline over time.

Verification

You should now be able to answer:

• What are the four rule tiers, and what minimum scenario count does each carry?
• Why is “rules with at least one test” the wrong way to measure coverage, and what does it hide?
• What three layers enforce coverage in a healthy rule pipeline, and what does each layer catch?
• What is the per-tier scenario count tool, and what command-line invocation runs it?
• Why does the quarterly review matter when the PR-time check is wired in?

Quiz