Rule Tests in CI

A team writes two hundred unit tests for its rules. The developers run them locally before every commit. The CI is not wired up. Six months later, a junior engineer refactors an alerting rule and merges without running the tests locally. The rule reaches production. The alert fires for the wrong reason. The on-call rota is paged at 02:00. The investigation finds the bug is one the test suite would have caught. The suite exists. The CI does not run it. The discipline of “tests that do not run are not tests” was violated by a single missing workflow file.

This is what the rule-tests-in-CI lesson exists to prevent. Tests that live on developer laptops are not enforcement; they are suggestions. The CI is the enforcement mechanism. A workflow that runs promtool check rules and promtool test rules on every pull request, configured as a required status check, is the discipline that converts the test suite from a suggestion into a gate.

What it is

Rule tests in CI is the GitHub Actions (or equivalent runner) workflow that:

1. Installs a pinned version of promtool matching the production Prometheus version.
2. Runs promtool check rules against every rule file in the repository.
3. Runs promtool test rules against every fixture under the test directory.
4. Reports a non-zero exit on any failure.
5. Is configured as a required status check so a failing run blocks the PR from merging.

The workflow is the enforcement mechanism. Without it, the test suite is a developer-local artefact; the team depends on the discipline of each developer to run the tests before commit. With it, the workflow is mechanical; a failing fixture blocks the merge regardless of the developer’s discipline.

Two common shapes exist for the workflow:

• Action-based. Use prometheus/promtool-github-action, which installs promtool and runs a single command. Simple; one action per check.
• Manual install. Install promtool from the official release tarball, then run the checks in a run: step. More control over the version and the install location.

Both shapes produce the same enforcement. The discipline is to wire the workflow into the branch protection so a failing run blocks the merge.

Why a sysadmin cares

A rule test suite without a CI integration is a documentation artefact. It documents the team’s intent for the rule; it does not enforce it. The failure mode is the junior engineer who merges without running the tests, the contractor who does not know the test suite exists, or the production hot-fix that bypasses the workflow because the branch protection was not configured for the hot-fix path.

The CI integration is the difference between “we have tests” and “tests are enforced”. A team that adopts the integration catches the bug at PR time, before merge, before the rule reaches production. The cost is the workflow file and the branch protection configuration. The benefit is the bug never reaches production.

How it works

The pipeline has four steps:

  pull request opened or updated
        |
        v
  checkout repository
        |
        v
  install promtool (pinned to production version)
        |
        v
  promtool check rules rules/*.yml
        |
        v
  promtool test rules rules/test/*.yml
        |
        v
  report status to GitHub
        |
        v
  branch protection evaluates:
        required status check "rules" passes?
        |
   yes  v       v  no
   merge          block merge, surface failure

The branch protection configuration makes the workflow a required status check. The configuration lives in the repository settings (or in a GitHub App / Terraform / Pulumi declaration). The discipline is to add the workflow name exactly as the workflow reports it; a mismatch means the check runs but does not block.

A useful addition is a fixture-presence step (covered in the coverage lesson) that fails when a rule file is added without a matching fixture. That step belongs in the same workflow so the gate is the union of all four checks.

How to configure it

A worked GitHub Actions workflow that wires rule tests into every pull request.

# .github/workflows/rules.yml
name: rules

on:
  pull_request:
    paths:
      - 'rules/**'
      - '.github/workflows/rules.yml'
  push:
    branches:
      - main
    paths:
      - 'rules/**'

jobs:
  check-and-test:
    runs-on: ubuntu-latest
    steps:
      - name: checkout repository
        uses: actions/checkout@v4

      - name: install promtool
        # Pin to match production Prometheus.
        run: |
          set -euo pipefail
          PROMETHEUS_VERSION=2.55.1
          curl -sSL \
            "https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz" \
            | sudo tar -xz -C /opt/
          sudo ln -sf \
            /opt/prometheus-${PROMETHEUS_VERSION}.linux-amd64/promtool \
            /usr/local/bin/promtool
          promtool --version 2>&1 | head -1

      - name: promtool check rules
        run: |
          set -euo pipefail
          for rule_file in rules/*.yml; do
            echo "Checking $rule_file"
            promtool check rules "$rule_file"
          done

      - name: promtool test rules
        run: |
          set -euo pipefail
          for fixture in rules/test/*_test.yml; do
            echo "Testing $fixture"
            promtool test rules "$fixture"
          done

      - name: fixture presence check
        run: |
          set -euo pipefail
          for rule_file in rules/*.yml; do
            base=$(basename "$rule_file" .yml)
            fixture="rules/test/${base}_test.yml"
            if [ ! -f "$fixture" ]; then
              echo "::error file=$rule_file::missing fixture $fixture"
              exit 1
            fi
          done

Five things to notice:

• on.pull_request.paths scopes the workflow to rule and workflow changes. A PR that touches only documentation does not run the workflow; the CI minutes are saved.
• on.push.branches runs the workflow on every push to main. A merge that bypasses the PR path (force-push, admin override) still runs the check.
• PROMETHEUS_VERSION=2.55.1 pins the version. The workflow fails fast if the release tarball is unavailable.
• The promtool test rules step walks every fixture. The pattern rules/test/*_test.yml matches the convention.
• The fixture presence step fails when a rule file is added without a matching fixture. This is the coverage discipline from the previous lesson, wired into the same workflow.

The branch protection configuration that turns the workflow into a gate:

Settings -> Branches -> Branch protection rules -> main
  -> Require status checks to pass before merging
  -> Status checks found in the last week for this repository
     -> select "rules / check-and-test"
  -> Require branches to be up to date before merging
  -> Do not allow bypassing the above settings

The discipline is the last checkbox. Without it, an admin can override the protection and merge a failing PR. The override is the failure shape; the discipline is to remove the override.

How to validate it

Three checks. The first confirms the workflow file is syntactically valid; the second confirms the workflow runs on a test PR; the third confirms the branch protection is configured.

# 1. The workflow YAML parses.
# Use any YAML linter; for example:
yamllint .github/workflows/rules.yml

Expected output: no errors. A parse error in the workflow file means the workflow cannot run.

# 2. The workflow runs on a test PR.
# Open a PR that touches a rule file; confirm the "rules"
# check appears in the PR's checks list and reports success.

Expected: a check named rules / check-and-test with a green tick. A red cross or a missing check means the workflow did not run or the branch protection is not configured.

# 3. The branch protection is configured.
# Use the GitHub CLI:
gh api repos/:owner/:repo/branches/main/protection \
  | jq '.required_status_checks.checks[] | select(.context=="rules / check-and-test")'

Expected output:

{
  "context": "rules / check-and-test",
  "app_id": null
}

A missing entry means the branch protection does not require the check; a failing workflow would not block the merge.

How it can fail

Six CI-specific failure modes.

1. CI uses a different promtool version than production. Symptom: a rule change passes check rules in CI but fails to load in production. Cause: the action or the manual install pinned an older version. Fix: pin the version to match production; update both at the same time when upgrading.
2. The workflow is not a required status check. Symptom: a PR with a failing rules check merges anyway. Cause: the branch protection does not list the check as required. Fix: add the check to the branch protection; confirm with the GitHub CLI.
3. The workflow runs only on push to main, not on pull_request. Symptom: a rule change is merged through a PR without the workflow ever running; the check runs only after the merge. Cause: the on: block misses the pull_request trigger. Fix: add pull_request with the same path scope.
4. The fixture glob in CI does not match the developer’s local glob. Symptom: a fixture exists locally but the CI step does not pick it up. Cause: the CI glob rules/test/*.yml does not match the convention *_test.yml. Fix: align the glob; or rename the fixtures.
5. The admin override is enabled. Symptom: a failing PR is merged by an admin. Cause: the “Do not allow bypassing the above settings” checkbox is not set. Fix: enable the checkbox.
6. The action or version is silently upgraded. Symptom: a CI run on Tuesday uses promtool 2.54; the same workflow on Wednesday uses 2.55 because the action was bumped. Cause: the action’s tag tracks the latest release, or the manual install uses latest instead of a pinned version. Fix: pin the version explicitly.

How to troubleshoot it

In order:

1. Did the workflow run? Open the PR; look at the “checks” list. A missing check means the workflow did not trigger.
2. Did the workflow pass? A red cross on the check means the check failed. Open the workflow log and find the failing step.
3. Does the failing step match the local failure? A mismatch between local and CI output means the CI environment differs from the local environment (different promtool version, different working directory, different fixture glob).
4. Is the branch protection configured? Use the GitHub CLI to confirm the check is required. A missing entry means the check runs but does not block the merge.
5. Is the admin override enabled? Check the “Do not allow bypassing” setting. If the override is enabled, an admin can merge a failing PR.
6. Is the version pin current? Confirm the PROMETHEUS_VERSION in the workflow matches the production Prometheus version.

Security implications

The CI workflow reads the rule directory and the fixture directory. There is no network exposure beyond the GitHub Actions runner’s outbound connections (which fetch the promtool tarball). Three risks matter.

1. The workflow exposes the promtool version. The pinned version is visible in the workflow YAML. For most teams this is acceptable; for security-sensitive environments, version disclosure can inform a targeted attack. Treat the workflow YAML with the same disclosure posture as rule files.
2. The fixture content reveals internal topology. A fixture that asserts on http_requests_total{job="payments-prod-eu-west-1"} commits service names and regions to the repository. Treat fixture content with the same disclosure posture as dashboards.
3. The CI runner executes code from the workflow YAML. A workflow that uses unpinned actions or unverified downloads can be a supply-chain attack vector. Pin actions by SHA, not by tag; pin the promtool tarball by version; verify the checksum if the team is security-sensitive.

Performance implications

A workflow that installs promtool, runs check rules on every rule file, and runs test rules on every fixture finishes in under a minute for a typical repository with hundreds of rules. The cost is in CI minutes, not in production CPU. The bottleneck is the install step; using the GitHub Actions cache for the tarball brings the cost down to seconds.

The hidden cost is the cumulative time developers wait for the CI to finish on every PR. A workflow that runs unnecessarily (because the path scope is too broad) slows every PR. Scope the workflow to rule changes; let other PRs run their own checks.

Production guidance

• Pin the promtool version to match production. A two-version drift catches you out. Update both at the same time when upgrading Prometheus.
• Make the workflow a required status check. Without the branch protection, the workflow runs but does not gate. Add the check to the branch protection and confirm with the GitHub CLI.
• Disable the admin override. The “Do not allow bypassing the above settings” checkbox is the discipline that prevents an emergency merge from bypassing the gate.
• Scope the workflow to rule changes. A workflow that runs on every PR wastes CI minutes. Scope the paths: to rules/** and the workflow YAML itself.
• Add a fixture-presence check to the same workflow. The coverage discipline belongs in the same gate; a rule change without a fixture is a failing CI.
• Use the GitHub Actions cache for the promtool tarball. The install step is the bottleneck; caching brings the cost from minutes to seconds.
• Pin actions by SHA, not by tag. Tag-based pinning allows a tag to be moved; SHA-based pinning does not.

Verification

You should now be able to answer:

• What are the two common shapes for a rule-tests-in-CI workflow, and what is the trade-off between them?
• Why must the promtool version in CI match the production Prometheus version, and what is the failure shape when it drifts?
• What branch protection configuration turns a workflow into a gate, and what is the role of the “Do not allow bypassing” checkbox?
• Why does a workflow that runs only on push to main miss the PR-time gate, and what trigger must be added?
• What fixture-presence check belongs in the same workflow, and why is it part of the same gate?

Quiz