promtool check rules

A team adds a new alerting rule to app-checkout.yml. The expr references a histogram bucket the team recently renamed from http_request_duration_seconds_bucket to checkout_request_duration_seconds_bucket. The merge happens at 11:32. At 11:38 the configuration is reloaded, and Prometheus accepts the file — the YAML is valid, the rule parses — but the new alert never fires because the source metric does not exist. The team does not notice for two days; during that time, the original checkout failure that the alert was meant to catch happens, the alert does not page, and a real outage is diagnosed twenty minutes later than it should have been. The rule parse was fine. The expression was fine. The metric was missing. The static check cannot see missing metrics, but it can see a related class of mistake, and a careful CI wiring catches the related class before any of the other costs are paid.

What it is

promtool check rules <path> parses the named files (or every .yml and .yaml under the named directory) as Prometheus rule files. It walks each groups: block, parses every record or alert entry, validates the YAML schema, parses the PromQL expression, and validates the annotation template strings. The command exits 0 if every rule passes, and exits non-zero with a file name, line number and error message if any rule fails.

Two properties:

1. It uses the same parser as the running daemon. Just like promtool check config, the tool links against the same rule-loading code Prometheus invokes at reload time. If promtool check rules accepts a rule file, Prometheus will accept it at reload.
2. It is static. The check parses PromQL and validates the template strings, but does not evaluate the expression against any data. A rule whose expression references a metric that does not exist is valid syntax and passes the check.

What the check does catch:

• Unknown keys inside a rule: block.
• Wrong type for a known key (for: "5m" instead of for: 5m).
• A record: name that does not match the [a-zA-Z_][a-zA-Z0-9_]* regex.
• A PromQL expression that fails to parse.
• A template string that fails to expand (a {{ $lables.foo }} typo).
• A duplicate record: name across files (with --lint-fatal).

What the check does not catch:

• A PromQL expression that parses but never returns data (missing source metric, wrong selector).
• A for: value that is correct semantically but wrong operationally (5s instead of 5m for a flap-prone metric).
• A threshold that is too tight or too loose for the actual traffic shape.
• A label dimension that is missing from labels: or annotations:.

The semantic failures are the job of promtool test rules, covered in the next lesson.

Why a sysadmin cares

Every alert rule is a piece of production code that runs on the configured evaluation_interval and whose output reaches the on-call rota. A bug in a rule is a production incident waiting to happen. The static check catches the cheap class of bug — typos, schema drift, PromQL parse errors — and the dynamic check (promtool test rules) catches the next class. Without both, the team ships rules that look correct in code review and misbehave in production.

The four most operationally painful failure shapes the static check catches:

1. PromQL parse error. A reference to a metric that does not exist in this version of the codebase. The rule loads; the expression fails to evaluate; the alert never fires. The page that should have happened does not.
2. Template typo. {{ $lables.instance }} parses fine but fails to expand at evaluation time. The alert’s summary becomes literal $lables.instance text, which the Alertmanager template discards or sends as empty.
3. Wrong-type value for for:. for: "5m" (string) instead of for: 5m (duration). The daemon rejects the whole file; the rule, and every other rule in the same file, fails to load.
4. Duplicate record: name across files. Two groups write the same series at the same evaluation timestamp. The daemon rejects the duplicate; the older record wins in the TSDB; the newer record’s downstream consumers read staleness.

Each of these is invisible to code review without the static check. Each is caught in 300 milliseconds per file.

How it works

  promtool check rules observability/prometheus/rules/
        |
        v
  YAML parser reads every *.yml and *.yaml in the directory
        |
        v
  For each groups: block:
    for each rule:
      validate YAML schema
      parse PromQL expression (syntax only)
      parse and validate template strings in annotations
        |
        v
  With --lint-fatal: check for duplicate record: names
  across files; check for empty globs; check for
  unsupported fields
        |
        v
  On any failure: print file:line and error, exit 1
  On success: print "found N rules, M alerts" per file,
  exit 0

The order matters: a YAML schema error short-circuits the PromQL parse for that rule. The tool does not promise to report every error in one pass; one parse failure per file is the common case.

How to configure it

The check is a command. The two common wirings:

Locally, as a pre-commit check:

#!/usr/bin/env bash
# .git/hooks/pre-commit
set -e
promtool check rules observability/prometheus/rules/ --lint-fatal

In a GitHub Actions workflow:

# .github/workflows/promtool-rules.yml
name: promtool-check-rules

on:
  pull_request:
    paths:
      - 'observability/prometheus/rules/**'
      - '.github/workflows/promtool-rules.yml'

permissions:
  contents: read

jobs:
  check:
    name: promtool check rules
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: promtool check rules
        uses: prometheus/promtool-github-action@v0.1.0
        with:
          arguments: >-
            check rules
            observability/prometheus/rules/
            --lint-fatal

Two things to notice:

• --lint-fatal is passed. Without it, a duplicate record: name prints a warning and exits 0; with it, the same finding exits 1. CI should always pass --lint-fatal.
• Path trigger. The job runs only when files under observability/prometheus/rules/** change. Saving CI minutes on unrelated PRs.

For teams that prefer to install promtool explicitly and target multiple directories:

      - uses: actions/checkout@v4
      - name: Install promtool
        run: |
          PROMTOOL_VERSION=2.55.1
          curl -sSL \
            "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/prometheus-${PROMTOOL_VERSION}.linux-amd64.tar.gz" \
            | tar xz -C /tmp
          sudo mv \
            /tmp/prometheus-${PROMTOOL_VERSION}.linux-amd64/promtool \
            /usr/local/bin/
      - name: promtool check rules
        run: |
          set -e
          for d in observability/prometheus/rules \
                   observability/remote-write/rules; do
            echo "Checking $d"
            promtool check rules "$d" --lint-fatal
          done

The directory walk covers all rule file globs the production daemon loads.

How to validate it

Three checks confirm the gate is wired correctly.

1. The check accepts a known-good rule file.

promtool check rules observability/prometheus/rules/ --lint-fatal

Expected output, exit 0:

Checking observability/prometheus/rules/app-checkout.yml
  SUCCESS: found 6 rules, 2 alerts

Checking observability/prometheus/rules/platform.yml
  SUCCESS: found 4 rules, 0 alerts

The first line is the per-file report; the tool prints a line per file in the directory. The “found N rules, M alerts” count must match the file’s actual count.

2. The check rejects a known-bad rule file.

# observability/prometheus/rules/bad.yml
groups:
  - name: bad
    rules:
      - alert: BadRule
        expr: sum(rate(missing_metric[5m]))
        for: "5m"   # string instead of duration
        labels:
          severity: critical
        annotations:
          summary: '{{ $lables.instance }} is bad'

promtool check rules observability/prometheus/rules/bad.yml --lint-fatal

Expected output, exit 1:

FAILED: parsing YAML file observability/prometheus/rules/bad.yml:
  yaml: line 7: cannot unmarshal !!str `5m` into time.Duration

The error names the file and line. Fix the type, rerun, and the check passes.

3. The lint pass catches a duplicate record name.

echo '
groups:
  - name: g1
    rules:
      - record: job:up:avg5m
        expr: avg by (job) (up)
' > rules/a.yml

echo '
groups:
  - name: g2
    rules:
      - record: job:up:avg5m
        expr: avg by (job) (up)
' > rules/b.yml

promtool check rules rules/   # exit 0, prints warning
# warning: found duplicate record name "job:up:avg5m"

promtool check rules rules/ --lint-fatal
# exit 1
# FAILED: rules/b.yml: duplicate recording rule "job:up:avg5m"

The --lint-fatal flag turns the duplicate finding into a non-zero exit.

How it can fail

Six failure modes specific to the rule check:

1. PromQL expression references a metric that does not exist. Symptom: the check passes; the rule loads; the alert never fires. Cause: the metric was renamed or removed in a recent exporter upgrade. The check is static and cannot see missing metrics. Add a unit test that asserts the rule produces data.

2. Template typo. {{ $lables.instance }} instead of {{ $labels.instance }}. Symptom: the check passes (the template syntax is valid Go template syntax); the alert’s summary becomes literal $lables.instance text at evaluation time. Cause: the typo is invisible to the Go template parser. Add an exp_annotations: assertion in a unit test that confirms the expanded text.

3. Wrong-type value for for:. Symptom: the check rejects the file with a type error. Cause: a recent copy from a different YAML source pasted for: "5m" instead of for: 5m. The error message names the field and line; fix the type.

4. Duplicate record: name across files. Symptom: the daemon rejects one of the duplicate files at reload, and the older record wins in the TSDB. Cause: two teams wrote a recording rule with the same name. Run the check with --lint-fatal in CI; the duplicate becomes a build failure.

5. promtool version mismatch with production. Symptom: the check passes in CI, the rule fails to load in production. Cause: CI pins promtool v2.54.0 while production runs v2.55.1, which enforces stricter PromQL validation. Pin CI to the same version.

6. The path glob is too narrow. Symptom: a new rule file is added to observability/prometheus/rules/ but the CI job’s paths: trigger does not include the new path. Cause: the trigger was set to observability/prometheus/rules/checkout.yml (the only file at the time) and never broadened. Use a directory trigger, not a per-file trigger.

How to troubleshoot it

In order:

1. Read the tool’s error. The error names the file, the line, and the field. The fix is in the message.
2. Confirm --lint-fatal is set. Without it, lint findings are warnings, not errors. CI must pass --lint-fatal.
3. Run on one file at a time. When the check rejects a directory, isolate the offending file with a single path argument and rerun.
4. Confirm the promtool version. promtool --version must match the production daemon’s version.
5. Check the lint output even on success. Without --lint-fatal, warnings print to stdout; capture them in the CI log and review manually.

Security implications

• The check prints file content on parse error. A rule whose annotations: block contains a secret (a credential, a runbook URL with an embedded token) will leak the secret into the CI log on parse error. Use a secret manager or a template variable for secrets in annotations.
• The check does not connect to the live daemon. It is read-only and does not expose any new endpoint. It is safe to run from any workstation.
• The check is bounded by the rule files in scope. A check that runs only against a partial directory will miss duplicate record: names across the un-scanned files. Always scan the entire rule tree.

Performance implications

The check is fast. A directory of fifty rule files with two hundred rules validates in well under a second on commodity hardware. The cost is dominated by the PromQL parse for each rule and the lint pass for the directory; CI budgets the check at 300 ms to 1 s per file. There is no production cost from running the check; it is purely a pre-merge gate.

Production guidance

• Run promtool check rules --lint-fatal on every pull request that touches the rule files.
• Pin the promtool version in CI to the same version as production.
• Treat lint findings as errors. --lint-fatal is the difference between a polite warning and a failed build.
• Run the check against the entire rule tree, not a subset. A subset misses cross-file duplicates.
• Wire the check as a complement to promtool check config, not a replacement. The two checks catch different classes of mistake.

Verification

You should now be able to answer:

• What does promtool check rules validate that promtool check config does not?
• What does --lint-fatal change in the output?
• Why does a rule whose expr references a missing metric pass the static check?
• Why must the CI’s promtool version match the production daemon’s version?
• What is the difference between a parse error and a semantic error in a rule?

Quiz