Rule Lint and Review

A team inherits 240 rules from a contractor who left two years ago. Half have no runbook URL. A third reference metrics that no longer exist. None have unit tests. The team spent six weeks auditing the rule set by hand, and burned three quarters of that auditing rules that turned out to be wrong in obvious ways a static check would have caught. The lesson is that the discipline of linting and reviewing rules is the same discipline as linting and reviewing code: cheap to automate, expensive to skip.

What it is

Rule lint is the set of static and dynamic checks that confirm a rule is syntactically valid, semantically correct, and operationally complete before it reaches production. The three layers:

• Static check — promtool check rules parses the YAML, validates the PromQL expression, and reports parse errors. This is the cheap, fast layer that catches typos and structural mistakes.
• Unit test — promtool test rules evaluates the rule against synthetic time series and asserts the expected alert state at specified timestamps. This is the layer that catches semantic mistakes: a rule that parses but never fires, or fires under conditions it should not.
• Review — a human reviewer checks the operational completeness: runbook URL, dashboard URL, severity label, team label, for: value, threshold justification. This is the layer that catches the omissions no tool can detect.

A team that adopts all three layers catches the rule problems that one or two of the layers miss.

Why a sysadmin cares

Every alert rule is a piece of production code. It runs every 30 seconds. Its output reaches the on-call rota. A bug in a rule is a production incident waiting to happen: the rule either misses a real outage or pages the rota for a non-event. The cost of either mistake is measured in minutes per page and in credibility with the rota.

The review cadence matters because rules decay. The threshold that was correct two years ago is too low now (traffic grew) or too high (the codebase changed). The runbook URL that pointed at a Confluence page now points at an archive. The team label that matched the team two years ago now matches a different team. A scheduled review catches the decay; an ad-hoc review lets it accumulate.

How it works

The three layers fit into a CI pipeline:

  pull request opened
        |
        v
  promtool check rules      <-- static parse, PromQL validate
        |
        v
  promtool test rules       <-- synthetic series, expected alerts
        |
        v
  human review              <-- runbook, dashboard, severity
        |
        v
  merge to main
        |
        v
  deploy + SIGHUP Prometheus
        |
        v
  recurring review (quarterly) -- check stale thresholds, dead runbooks

A pull request that fails any of the first three layers does not merge. The recurring review is separate from the PR workflow; it catches problems that grew over time, not problems that were introduced in the latest change.

How to configure it

Two halves: the rule file and the test fixture.

The rule (a worked example, abridged):

groups:
  - name: orders-api.slo
    interval: 30s
    rules:
      - alert: OrdersApiHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: checkout
        annotations:
          summary: 'orders-api 5xx ratio above 5% in {{ $labels.region }}'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

The test fixture, in test/orders-api_test.yml:

rule_files:
  - ../orders-api.yml

evaluation_interval: 30s

tests:
  # Test 1: a 4-minute breach should NOT fire (within for: dwell).
  - interval: 30s
    input_series:
      - series: 'http_requests_total{service="orders-api",region="eu-west-1",status="200"}'
        values: '0 100 100 100 100 100 100 100 100'
      - series: 'http_requests_total{service="orders-api",region="eu-west-1",status="500"}'
        values: '0 1 1 1 1 1 1 1 1'
    alert_rule_test:
      - eval_time: 4m
        alertname: OrdersApiHighErrorRate
        exp_alerts: []

  # Test 2: a 6-minute breach SHOULD fire.
  - interval: 30s
    input_series:
      - series: 'http_requests_total{service="orders-api",region="eu-west-1",status="200"}'
        values: '0 100 100 100 100 100 100 100 100'
      - series: 'http_requests_total{service="orders-api",region="eu-west-1",status="500"}'
        values: '0 10 10 10 10 10 10 10 10'
    alert_rule_test:
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        exp_alerts:
          - exp_labels:
              severity: critical
              team: checkout
              service: orders-api
              region: eu-west-1
            exp_annotations:
              summary: 'orders-api 5xx ratio above 5% in eu-west-1'
              runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

Three things to notice:

• The input_series: block defines synthetic time series. The first test has a 1% error rate (1 in 100), which is below the 5% threshold; the alert should not fire at 4m. The second test has a 10% error rate (10 in 100), which is above; the alert should fire at 6m.
• The alert_rule_test: block asserts the alert state at a specific eval_time. exp_alerts: is the expected list; an empty list means no alert should exist.
• The exp_labels: and exp_annotations: blocks confirm the rule produces the labels and annotations the Alertmanager routes expect.

The CI pipeline (GitHub Actions, abridged):

name: rules
on:
  pull_request:
    paths:
      - 'rules/**'
      - '.github/workflows/rules.yml'

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: promtool check
        uses: prometheus/promtool-github-action@v0.1.0
        with:
          arguments: check rules rules/

      - name: promtool test
        run: |
          for f in rules/test/*.yml; do
            echo "Testing $f"
            promtool test rules "$f"
          done

Two checks. The first uses a published action that installs promtool and runs check rules. The second walks every test file under rules/test/ and runs promtool test rules against each. A failure on either step blocks the merge.

How to validate it

Three checks. The first is the static check; the second is the unit test; the third is the live confirmation.

# 1. Static check. Runs in milliseconds.
promtool check rules /etc/prometheus/rules/orders-api.yml

Expected output:

Checking /etc/prometheus/rules/orders-api.yml
  SUCCESS: found 1 rules, 1 alerts

A non-zero exit with a parse error means the rule will not load. Fix the YAML and rerun.

# 2. Unit test. Runs in milliseconds against synthetic data.
promtool test rules /etc/prometheus/rules/test/orders-api_test.yml

Expected output:

SUCCESS

A non-zero exit with a test failure names the failing test (Test 2: 6m breach should fire) and the actual alert state versus the expected. Fix the rule or the test until they match.

# 3. Live confirmation after deploy. Same checks as lesson 01.
curl -s http://prometheus:9090/api/v1/rules \
  | jq '.data.groups[].rules[] | select(.name == "OrdersApiHighErrorRate")
        | {state, lastEvaluation}'

Expected output:

{
  "state": "inactive",
  "lastEvaluation": "2026-08-13T04:00:00.000Z"
}

state: inactive is fine; the rule is loaded and the expr returned no series.

How it can fail

Six failure modes:

1. promtool check rules passes but the rule never fires in production. Symptom: rule loaded, expr returns data, but /api/v1/alerts is empty. Cause: the rule’s for: is larger than the longest window in the unit test, so the test never observed the firing state. Lengthen the test window or set for: shorter for the test.

2. Unit test passes but the rule misroutes. Symptom: the alert fires but the Alertmanager catch-all picks it up. Cause: the rule’s labels: block was not asserted in exp_labels:. Add severity, team, and service to exp_labels: so the test fails if the labels regress.

3. CI runs promtool from a different version than production. Symptom: check rules passes in CI but the rule fails to load in production. Cause: the CI image pins promtool 2.54 and production runs Prometheus 2.55. Pin the same version (or a version known to be compatible) in both.

4. No unit test for a rule that mutates labels. Symptom: a rule that adds a label in labels: goes un-reviewed because the test file does not cover the labels block. Cause: the rule author skipped the test. Add a CI step that fails when a rule is added without a matching test file.

5. Review checklist is not enforced. Symptom: rules reach production without a runbook URL, because the checklist is a wiki page nobody reads. Cause: the checklist is not enforced by tooling. Encode the checklist in a linter (mixin, custom script) so a missing runbook URL fails the build.

6. No recurring review. Symptom: a rule that was correct two years ago has a threshold that no longer matches the traffic shape. Cause: nobody re-reads the rules after the initial merge. Schedule a quarterly review and assign owners.

How to troubleshoot it

In order:

1. Did promtool check rules pass? If not, the rule does not parse. Fix the YAML and rerun.
2. Did promtool test rules pass? If not, the test failed. The error message names the failing test and the actual versus expected state. Adjust the rule or the test until they match.
3. Did the rule load in Prometheus? /api/v1/rules. Missing rule: fix the glob and SIGHUP.
4. Does the expr return data in production? Compute it in Grafana Explore. Empty result: the metric is missing or the selector is wrong.
5. Does the alert reach the right receiver? Check Alertmanager /api/v2/alerts for the receivers[] field.
6. Is the recurring review happening? Confirm the review calendar entry exists; if it does not, schedule it.

Security implications

Rule lint and review are not security-sensitive on their own. The risks are operational and indirect:

• A unit-test fixture that includes real-looking secrets (an API key used in a runbook_url) commits those secrets to the repo. Use placeholder values and confirm with a secret scanner.
• A review that approves a rule without checking the runbook_url can leak the existence of an incident to a third-party wiki. Confirm the URL points at trusted infrastructure.

Performance implications

Lint and unit tests run in milliseconds. The cost is in CI minutes, not in production CPU. A pipeline that runs promtool check rules and promtool test rules on every pull request adds seconds to the CI run, not minutes. The cost is worth it for the mistake prevention.

A unit-test fixture that uses a very long input_series (more than a few hundred samples) slows the test. Keep test fixtures small: enough samples to exercise the rule, no more.

Production guidance

• Adopt the three layers. Static check first; unit test second; human review third. Do not merge without all three.
• Pin promtool in CI to the same version as production. A two-version drift catches you out.
• Encode the human-review checklist in a linter so missing runbook URLs fail the build, not just the reviewer.
• Schedule a recurring review (quarterly is a common cadence) and assign owners per rule group.

Verification

• What is the difference between promtool check rules and promtool test rules?
• What four blocks make up a promtool test rules fixture?
• What is the minimum CI step to add to a pull request pipeline for an alert rule change?
• Why does a static check not catch every rule mistake, and what is the role of the human review?

Quiz