Alert Rules as Code

The new on-call rotation arrives. The first review of the rule files is a directory listing of 23 YAML files across two repositories. A team has been adding alerts.rules.yml blocks every quarter; the average file is 380 lines; many of them have duplicated alert names with subtly different thresholds. Three HighErrorBudgetBurn rules exist with for: 5m, for: 1h, and for: 6h, in that order. None of them is wrong in isolation; all three together mean an error-budget burn fires three pages for the same incident.

This is the failure shape that rules-as-code is designed to prevent. The rule files belong in one place; they pass a linter in CI; they are unit-tested against fixture data; they are deployed through the same discipline as the rest of the configuration.

What it is

Rules-as-code is the discipline of treating alert and recording rules as a single versioned artefact that passes through code review, CI validation, unit-test fixtures, and canary deploy before it reaches production. In Prometheus 2.55.x, the artefact is a set of YAML files under rule_files: in prometheus.yml. Each file contains one or more groups: and each group contains a list of rules: (alerts and recordings).

A rule file is the operative definition of “what does Prometheus page on?”. A change to the file is a change to the behaviour of the on-call rotation. That change goes through review like any other production code.

Why a sysadmin cares

The alert rules are the loudest part of observability. When they are wrong, two distinct symptoms appear:

1. Spam. A rule that fires and stays firing for a transient condition. The on-call engineer pages out, finds nothing, and the rule still fires again the next morning. The fix is a for: clause and a unit test that asserts the rule does not fire on a single-sample spike.
2. Silence. A rule that does not fire when it should. The on-call rotation never pages because the threshold is wrong, the metric name has drifted, or the label selector is too narrow. The fix is a unit test that asserts the rule does fire on the expected input, plus a canary that runs the new rule against a copy of production.

Both shapes are preventable. Both are amplified when the rule files are not under configuration management.

How it works

The mental model is “rules are data, evaluated every evaluation_interval, with linter + unit-test gate before deploy”:

  rules/*.yml   (in Git)
       |
  promtool check rules  (CI gate)
       |
  promtool test rules   (unit-test gate; runs fixtures)
       |
  canary Prometheus   (smoke test against production data)
       |
  rule_files glob in prometheus.yml   (prod)
       |
  Prometheus evaluates every group every evaluation_interval
       |
  Alerts fire -> Alertmanager -> on-call

Two details matter. First, promtool check rules parses every rule file with the same loader Prometheus uses at runtime. CI must run it on every pull request that touches a rule file. A non-zero exit blocks merge.

Second, promtool test rules runs unit-test fixtures that synthesise samples and assert the expected alerts. A fixture file is YAML; it is checked into Git alongside the rules.

How to configure it

A working rule-file layout under rules/:

rules/
  recording.rules.yml    # pre-aggregations; SLI building blocks
  alerts.sli.rules.yml   # service-level alerts (latency, error rate, throughput)
  alerts.slo.rules.yml   # error-budget burn alerts
  alerts.host.rules.yml  # host-level alerts
  rules-test/            # fixture files, one YAML per rule group
    alerts.sli.test.yml

A recording rule:

groups:
  - name: api.recording
    interval: 30s           # override the global evaluation_interval for this group
    rules:
      - record: api:request_error_rate:ratio_5m
        expr: |
          sum(rate(http_requests_total{job="api",code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="api"}[5m]))
        labels:
          team: payments
          sli: error_rate

An alert rule that depends on the recording rule:

groups:
  - name: api.alerts
    rules:
      - alert: HighErrorRate
        expr: api:request_error_rate:ratio_5m > 0.01
        for: 5m
        labels:
          severity: critical
          team: payments
          sli: error_rate
        annotations:
          summary: 'api error rate above 1 percent'
          description: 'See {{ $labels.instance }} — 5xx ratio is {{ $value | humanizePercentage }}.'

Three disciplines in the YAML above:

• for: 5m means “the expression must be true for five minutes before firing”. Without it, a single spike pages out.
• Labels under labels: are routing labels. Alertmanager matches on them. Putting severity: critical and team: payments here is how routing trees select the right team.
• Annotations under annotations: are the only template the Alertmanager can render. The placeholders {{ $labels.* }} and {{ $value }} are mandatory for a useful page.

The CI gate:

.PHONY: rules
rules:
  promtool check rules rules/*.yml
  promtool test  rules rules-test/*.yml

The full unit test for the alert above:

rule_files:
  - ../alerts.sli.rules.yml
evaluation_interval: 1m

tests:
  - interval: 1m
    # synthetic series that crosses the 1 percent threshold at t=5m
    input_series:
      - series: 'http_requests_total{job="api",code="500"}'
        values: '0+0x10'
      - series: 'http_requests_total{job="api"}'
        values: '100+0x10'
    alert_rule_test:
      - eval_time: 6m
        alertname: HighErrorRate
        exp_alerts:
          - exp_labels:
              severity: critical
              team: payments
              job: api
            exp_annotations:
              summary: 'api error rate above 1 percent'

Run it:

promtool test rules rules-test/alerts.sli.test.yml

How to validate it

Three validations, each in a different stage.

# 1. CI: rule file parses with the runtime loader
promtool check rules rules/*.yml

# 2. CI: unit-test fixtures pass
promtool test rules rules-test/*.yml

# 3. Prod: rule_files on disk matches the live config
ssh prom-prod-01 curl -fsS http://localhost:9090/api/v1/status/config \
  | jq -r .data.yaml.original \
  | grep -A2 "rule_files"

For canary, run a second Prometheus pointed at the new rules against a read-replica of the production TSDB (or a remote-write target that mirrors the same metrics). Watch the canary /api/v1/rules for the expected alert name and confirm the lastEvaluation advances.

How it can fail

Six concrete failure modes appear repeatedly.

1. Duplicate alert names across files. Two rules with the same alert: name in different files. Prometheus loads both; Alertmanager receives both; the on-call sees two pages for the same condition. The fix is a CI regex that asserts alert names are unique across the rules directory.
2. for: too short. A for: 10s on an error-rate rule fires on a single second of network noise. The fix is a CI rule that asserts for: is at least 1m for any alert that derives from a rate() over a 5m window.
3. expr: label selector too narrow. A rule scoped to a single instance rather than a job. When the instance rotates, the rule goes silent. The fix is a fixture that asserts the alert fires on multiple instances.
4. Recording rule expression drift. A recording rule that produces a metric name different from what the alert consumes due to a copy-paste typo. The alert never fires. The fix is unit tests that assert the recording rule produces the expected series name at the expected labels.
5. Rule file in Git but not in rule_files glob. A new file is added to rules/; the glob picks it up; everything is fine. The failure shape is the reverse: the file is added to the directory but is excluded by a stricter glob. The fix is to always glob the directory and never a single file.
6. for: blocking the rule from firing during recovery. Setting for: 0 rather than removing the clause causes different behaviour from the absence of for: in some clients. The fix is to omit the clause when no hold-down is intended.

Security implications

Rule files do not expose new attack surface to the running service. The labels and annotations in a rule can contain user-supplied data only if the underlying metric labels contain it. The discipline:

• Do not interpolate untrusted text into annotations. The $value and $labels.* placeholders are safe; raw text from an HTTP query parameter is not.
• Reviewers must confirm that label names used in alert routing (e.g. team) are sourced from a fixed set rather than from runtime cardinality. Otherwise, an alert can route to a team that does not exist.
• *.rules.yml files belong under the same CODEOWNERS discipline as prometheus.yml. Treating them as second-class configuration is the failure mode.

Performance implications

Rule evaluation is a CPU and memory cost on the Prometheus host. The big knobs:

• evaluation_interval — halving it doubles the rule-evaluation cost. Recompute the cost when adding a group.
• The complexity of expr: — a rate() over [5m] evaluated over a high-cardinality label set can dominate rule-evaluation cost. Limit label cardinality in the metric side; refactor long expressions into recording rules.
• Recording rules reduce repeated evaluation cost. A recording rule that aggregates once per interval and is referenced by five alert rules is cheaper than five alert rules each running the same expression.

The file size itself is not the bottleneck. A 100 KiB rules file parses in milliseconds.

Production guidance

• One directory for rules. Globs in rule_files.
• promtool check rules in CI on every change.
• promtool test rules in CI for every rule group.
• Assert in CI: alert names are unique; for: is at least 1m on rate-based rules; expressions reference existing metric names.
• Deploy rules through the same GitOps pipeline as prometheus.yml.
• Canary Prometheus runs the new rules against a copy of the workload before the canary is promoted to production.

Verification

You should now be able to answer:

• What is the role of promtool check rules and promtool test rules in a CI pipeline?
• Why is for: 1m (or longer) the right default for an alert that derives from a rate(...) expression?
• What is the failure shape when two rule files both declare the same alert name?
• What does a unit-test fixture file contain, and what does it assert?

Quiz