Alert Test Cadence

A team adopts the four-tier alert testing discipline. The unit test runs on every pull request. The synthetic series runs nightly. The end-to-end test runs nightly. The scheduled canary runs every hour. Every CI badge is green. The team is confident the chain is healthy. Six months later, a real production outage occurs. The alert does not fire. The team investigates. The rule is loaded. The expression is correct. The for: dwell is correct. But the metric it references was renamed in an exporter upgrade four months earlier. The synthetic series exporter was never updated to match the new metric name. The nightly end-to-end test passed because it tested a different rule. The unit test passed because the fixture still references the old metric name — and the fixture was never reviewed after the exporter upgrade.

The team’s test cadence did not catch the rotted fixture. The unit test was not running against the production rule. The end-to-end test was not exercising the renamed metric. The canary was firing on a different rule entirely. The test cadence looked green; the discipline was silent.

The lesson is that a test cadence is not just “run the tests on a schedule.” The cadence must include a recurring human review that walks every fixture, every staging config, and every credential and asks “is this still right?” Without the review, the cadence rots.

What it is

Alert test cadence is the rhythm at which the four tiers of alert testing run. The cadence is layered; each tier runs at the frequency that matches its operational tempo.

  +---------------------------------------------+
  |  Tier 1: Unit test (promtool test rules)    |
  |    Cadence: on every pull request that      |
  |    touches the rule file or the fixture.    |
  |    Owner: rule author (CI blocks merge).    |
  +---------------------------------------------+
                     |
                     v
  +---------------------------------------------+
  |  Tier 2: Synthetic series (staging)         |
  |    Cadence: nightly, plus on every          |
  |    change to the rule's upstream exporter   |
  |    or to the staging Prometheus.            |
  |    Owner: observability team.               |
  +---------------------------------------------+
                     |
                     v
  +---------------------------------------------+
  |  Tier 3: End-to-end (staging)               |
  |    Cadence: nightly, plus on every          |
  |    change to the Alertmanager config or     |
  |    to the receiver templates.               |
  |    Owner: observability team.               |
  +---------------------------------------------+
                     |
                     v
  +---------------------------------------------+
  |  Tier 4: Scheduled canary (production)      |
  |    Cadence: continuous (the rule fires on   |
  |    a fixed schedule; the on-call rota       |
  |    expects the canary page).                |
  |    Owner: observability team.               |
  +---------------------------------------------+
                     |
                     v
  +---------------------------------------------+
  |  Quarterly review (human, in person)        |
  |    Cadence: every 90 days.                  |
  |    Owner: observability team lead.          |
  |    Reviews every fixture, every staging     |
  |    config, every credential, every runbook. |
  +---------------------------------------------+

The right approach is to run the four tiers on the cadence that matches their operational tempo, and to add a quarterly human review that catches what automation cannot: a fixture that asserts the wrong thing, a staging config that drifted, a credential that has been rotated, a runbook URL that points at a deleted page.

The most common shape is:

• Tier 1: every PR. A CI job that runs promtool test rules on every pull request that touches the rule files or the fixtures. The job blocks the merge on failure.
• Tier 2: nightly. A scheduled GitHub Actions job (or a cron-driven container) that runs the synthetic series test against the staging Prometheus. The job posts a failure to the observability chat channel.
• Tier 3: nightly. A scheduled job that runs the end-to-end test against the staging Alertmanager and receiver. The job posts a failure to the observability chat channel.
• Tier 4: continuous. A scheduled canary rule that fires every hour and pages the on-call via a dedicated receiver. A missing canary pages the on-call through the meta-alert.
• Quarterly review. A 90-minute meeting where the observability team lead walks every fixture, every staging config, every credential, and every runbook and confirms each is still correct. Findings are filed as issues.

Why a sysadmin cares

The test cadence is the discipline that keeps the alerting test discipline alive. A team that adopts the four tiers but never reviews them finds, six months later, that the fixtures assert the wrong metric names, the staging configs have drifted, and the credentials have been rotated three times since the test was last run. The cadence looks green; the discipline is silent.

The quarterly review is the layer that catches what automation cannot. Automation can run the test and report green or red. Automation cannot tell whether the fixture asserts the right thing, whether the staging config matches production, whether the runbook URL is still valid. A human reviewer walks every fixture, every config, every credential, every runbook, and asks “is this still right?” The reviewer is the discipline’s immune system.

The cost of skipping the review is paid in the next real outage. The cost of running the review is 90 minutes every 90 days per team lead. The trade-off is the right one.

How it works

The cadence is implemented as a set of scheduled jobs in the CI system (GitHub Actions, GitLab CI, Jenkins) and a recurring calendar entry for the quarterly review.

  PR (developer push)
    |
    v
  Tier 1: promtool test rules (CI job, blocks merge)
    |
    v
  Nightly 02:00 (cron)
    |
    +--> Tier 2: synthetic series test (CI job)
    |
    +--> Tier 3: end-to-end test (CI job)
    |
    v
  Hourly (continuous, production)
    |
    +--> Tier 4: scheduled canary fires; meta-alert
                 catches a missing canary
    |
    v
  Every 90 days (calendar entry)
    |
    +--> Quarterly review meeting (human)

Each tier posts its result to the observability chat channel. The on-call rota is expected to acknowledge a failing test within 24 hours. A failing canary is a production incident; the meta-alert pages the on-call.

How to configure it

The cadence is configured in the CI system, the production Prometheus, and the team’s calendar.

The Tier 1 CI job (GitHub Actions):

# .github/workflows/promtool-test.yml
name: promtool-test-rules

on:
  pull_request:
    paths:
      - 'observability/prometheus/rules/**'
      - '.github/workflows/promtool-test.yml'

permissions:
  contents: read

jobs:
  test:
    name: promtool test rules
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install promtool
        run: |
          PROMTOOL_VERSION=2.55.1
          curl -sSL \
            "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/prometheus-${PROMTOOL_VERSION}.linux-amd64.tar.gz" \
            | tar xz -C /tmp
          sudo mv \
            /tmp/prometheus-${PROMTOOL_VERSION}.linux-amd64/promtool \
            /usr/local/bin/
      - name: Run rule tests
        run: |
          set -e
          for f in observability/prometheus/rules/test/*.yml; do
            echo "Testing $f"
            promtool test rules "$f"
          done

The job runs on every pull request that touches the rule files or the workflow itself. A failure blocks the merge.

The Tier 2 + 3 nightly CI job:

# .github/workflows/alert-test-nightly.yml
name: alert-test-nightly

on:
  schedule:
    # Every night at 02:00 UTC
    - cron: '0 2 * * *'
  workflow_dispatch:

permissions:
  contents: read

jobs:
  synthetic-series:
    name: Tier 2: synthetic series test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run synthetic series test
        run: |
          docker compose -f observability/test/synthetic/docker-compose.yml up -d
          observability/test/synthetic/test.sh
      - name: Notify on failure
        if: failure()
        run: |
          curl -s -X POST "${{ secrets.OBSERVABILITY_CHAT_WEBHOOK }}" \
            -H 'Content-Type: application/json' \
            -d '{"text": "Tier 2 (synthetic series) test failed."}'

  end-to-end:
    name: Tier 3: end-to-end test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run end-to-end test
        run: observability/test/e2e/test.sh
      - name: Notify on failure
        if: failure()
        run: |
          curl -s -X POST "${{ secrets.OBSERVABILITY_CHAT_WEBHOOK }}" \
            -H 'Content-Type: application/json' \
            -d '{"text": "Tier 3 (end-to-end) test failed."}'

The job runs every night at 02:00 UTC. A failure posts to the observability chat channel.

The Tier 4 production canary (already covered in lesson 04).

The quarterly review calendar entry:

  Subject: Quarterly alert test review
  Cadence: every 90 days
  Duration: 90 minutes
  Attendees: observability team lead, on-call rota
            representative, one rule author
  Agenda:
    1. Walk every fixture in observability/prometheus/rules/test/
       - Confirm the metric name still exists in production
       - Confirm the label set still matches the rule's expr
       - Confirm the synthetic series is still in the staging
         Prometheus
    2. Diff the staging Alertmanager config against production
       - Confirm routes match
       - Confirm receivers point at test receivers
       - Confirm inhibit_rules match
    3. Walk every receiver credential
       - Confirm the webhook URL is valid
       - Confirm the API key has not expired
       - Confirm the secret scanner does not flag the fixture
    4. Walk every runbook URL
       - Confirm the URL is reachable
       - Confirm the page describes the current alert
       - Confirm the on-call response is current
    5. Walk the canary alert
       - Confirm the canary fired on schedule in the last 90
         days
       - Confirm the meta-alert has not fired
       - Confirm the synthetic exporter is healthy
  Output: a list of issues filed against the responsible
          owner with a 30-day deadline

The review meeting is the discipline’s immune system. The reviewer walks every fixture, every config, every credential, every runbook, every canary. The output is a list of issues with deadlines. The team addresses the issues before the next quarterly review.

How to validate it

Three checks confirm the cadence is in place.

1. The CI badge for the unit test is green.

gh run list --workflow=promtool-test-rules \
  --limit=1 --json conclusion,createdAt

Expected output, exit 0:

[{"conclusion": "success", "createdAt": "2026-08-13T01:00:00Z"}]

A success conclusion confirms the unit test ran recently and passed.

2. The nightly synthetic and end-to-end tests ran in the last 24 hours.

gh run list --workflow=alert-test-nightly \
  --limit=2 --json conclusion,createdAt

Expected output, exit 0:

[
  {"conclusion": "success", "createdAt": "2026-08-13T02:00:00Z"},
  {"conclusion": "success", "createdAt": "2026-08-12T02:00:00Z"}
]

Two recent successful runs confirm the nightly cadence is running.

3. The canary fired on schedule in the last 90 days.

amtool alert query alertname=SyntheticCanaryDown \
  --start=$(date -d '90 days ago' --iso-8601=seconds) \
  --end=$(date --iso-8601=seconds)

Expected output: a list of firings, one per hour (or per the canary’s configured cadence). An empty list or a single firing indicates the canary is not firing on schedule.

How it can fail

Six failure modes appear repeatedly when teams adopt a test cadence.

1. The fixture references a renamed metric. Symptom: the unit test passes but the production rule does not fire. Cause: the rule was updated to a new metric name and the fixture was not. Fix: the quarterly review walks every fixture and confirms the metric still exists.
2. The staging Alertmanager config drifted from production. Symptom: the end-to-end test passes but a production route is broken. Cause: a route was added to production and the staging config was not updated. Fix: generate the staging config from production with a substitution step; the quarterly review confirms the diff is empty.
3. A credential was rotated in production but not in staging. Symptom: the end-to-end test passes but the production receiver rejects the POST. Cause: the staging credential is the old production credential. Fix: rotate the staging credential in lockstep with production; the quarterly review confirms the credentials are in sync.
4. A runbook URL points at a deleted page. Symptom: the on-call rota opens the URL during a real incident and finds a 404. Cause: the runbook page was moved or deleted. Fix: the quarterly review walks every runbook URL and confirms it is reachable.
5. The canary fires constantly because the test harness does not restore the metric. Symptom: the on-call rota is paged every hour. Cause: the test harness crashed before restoring the metric to 1. Fix: the test harness restores the metric in a trap or finally block; the quarterly review confirms the canary is firing on schedule, not constantly.
6. The quarterly review is skipped because the team is busy. Symptom: the cadence rots; a real outage reveals the gap. Cause: the review meeting is the lowest-priority meeting on the calendar and gets bumped. Fix: the review is a recurring calendar entry that cannot be bumped without explicit team lead approval; findings are filed as issues with deadlines.

How to troubleshoot it

In order:

1. Did the test run? gh run list --workflow=<name> --limit=1. An empty list means the workflow never ran; check the schedule trigger.
2. Did the test pass? gh run view <run-id> --log-failed. The log shows the failure shape; fix the underlying issue.
3. Is the cadence still right? Walk the four tiers and confirm each is running at the configured cadence. A tier that has stopped running (e.g., the nightly job was disabled) is a silent failure.
4. When was the last quarterly review? Check the calendar. A review that was skipped or bumped is a signal that the cadence is rotting.
5. What changed since the last successful run? git log --since=<last-run-date> shows the changes to the rule files, the staging configs, and the credentials. Walk each change and confirm the test was updated.

Security implications

• The CI workflow runs in the team’s CI environment. The CI environment has access to the rule files, the fixture files, the staging Prometheus, and the staging Alertmanager. The CI environment does not have access to production credentials by default. Restrict the CI environment’s secrets to the staging credentials.
• The chat webhook for failure notifications is a secret. The webhook URL must be stored in the CI system’s secret store, not in the workflow file. A webhook URL committed to git leaks the channel’s identity and may allow attackers to post messages.
• The quarterly review may surface sensitive information. A reviewer who finds a credential committed to git must rotate the credential and remove the commit from history. The review’s output is filed as issues; the issues are private to the observability team.

Performance implications

• The unit test is cheap. A fixture with ten tests runs in milliseconds. The CI job’s cost is dominated by the runner startup time, not the test.
• The nightly synthetic and end-to-end tests are slow. The synthetic test waits for the rule’s for: dwell; the end-to-end test waits for the rule’s for: dwell plus the Alertmanager’s group_wait. The nightly job budget is 5-10 minutes per tier.
• The canary is cheap. The canary Prometheus and Alertmanager handle one alert per hour; the resource cost is trivial.
• The quarterly review is human time. 90 minutes per quarter per team lead. The cost is the team’s most scarce resource; the benefit is the discipline’s immune system.

Production guidance

• Adopt a layered cadence. The four tiers run at different frequencies: every PR (unit), nightly (synthetic + end-to-end), continuous (canary), quarterly (review). The cadence matches each tier’s operational tempo.
• Assign owners to each tier. The unit test’s owner is the rule author (CI blocks the merge). The nightly tests’ owner is the observability team. The canary’s owner is the observability team. The quarterly review’s owner is the observability team lead.
• Wire the cadence into the team’s calendar. A recurring calendar entry for the quarterly review that cannot be bumped without explicit team lead approval. The calendar entry names the agenda and the attendees.
• Treat the cadence as code. The CI workflows are committed to the repository. The canary rule is committed to the repository. The quarterly review’s agenda is committed to the repository. The cadence is reviewable in code review.
• Review the cadence itself at the quarterly review. The review walks the four tiers and asks: is the cadence still right? A team that adds new alert tiers (e.g., a Tier 5 for SLO burn-rate alerts) updates the cadence accordingly.

Verification

You should now be able to answer:

• What is the right cadence for each of the four tiers of alert testing?
• What does the quarterly review walk, and what failure shapes does it catch?
• Why must the cadence be assigned an owner for each tier, and what is the consequence of an unowned tier?
• What are the five ways a test cadence rots, and which of them does the quarterly review catch?
• Why is the quarterly review the discipline’s immune system, and what is the cost of skipping it?

Quiz