Alert Testing Basics

An on-call engineer is paged at 03:00 for a 5xx error rate. The error rate is real. The alert fires. The receiver receives. So far so good. Three hours of investigation later, the team discovers the alert that should have fired two hours earlier — the one on queue depth — never did. The rule loaded. The expression was correct. But the rule referenced a metric that had been renamed in an exporter upgrade eight months earlier. The rule evaluated to no data. No alert. No page. No detection.

A unit test would have caught the missing-metric reference. An alert canary would have caught the dead rule. The team had neither. The lesson is not “add tests after the fact.” The lesson is that alerting is code that wakes humans, and the discipline of testing alerting is the same discipline as testing any other production artefact: cheap to automate, expensive to skip.

What it is

Alert testing is the discipline of verifying that the alerting chain — rule expression, for: dwell, Alertmanager routing, receiver delivery — does what it claims to do, before the incident the rule was written to catch. There is no single type of alert test. There are four tiers, each catching a different class of mistake.

  Tier 1: Unit test (promtool test rules)
    |     Catches: wrong expression, wrong for:, missing
    |              metric reference, wrong label set.
    |     Runs offline. Milliseconds.
    |
  Tier 2: Synthetic series test
    |     Catches: rule that loads but never evaluates
    |              against the live TSDB (no data, wrong
    |              join, missing label, schema drift).
    |     Runs against a non-prod Prometheus. Seconds.
    |
  Tier 3: End-to-end alert test
    |     Catches: rule fires but Alertmanager does not
    |              route, receiver does not receive, or
    |              notification is malformed.
    |     Runs against a staging Prometheus +
    |              Alertmanager. Tens of seconds.
    |
  Tier 4: Scheduled canary (in production)
          Catches: dead rule, dead exporter, dead
                   Alertmanager, dead receiver, dead route,
                   dead paging integration. The rule itself
                   is correct but the chain is broken.
          Runs in production on a fixed schedule. Continuous.

The four tiers are not alternatives. They are layers. A team that runs only Tier 1 catches the wrong expression but not the dead exporter. A team that runs only Tier 4 catches the dead exporter but cannot tell whether the alert expression is correct without a real incident. A team that runs all four catches the full failure surface.

Why a sysadmin cares

Every alerting rule that reaches production is code that wakes a human at some point in the future. The cost of the rule being wrong is paid in three currencies:

• Minutes of sleep. The on-call rota is paged for a non-event. The engineer wakes, opens a laptop, opens Grafana, finds nothing wrong, closes the laptop, attempts to sleep. Repeat for every false positive.
• Minutes of incident response chasing a false alarm. The alert fires correctly but the runbook is wrong, the service is fine, and the engineer spent twenty minutes confirming that. The real incident, on a different rule, is still burning.
• Credibility. The rota learns to ignore a page because it has been wrong before. The next real page gets triaged at the bottom of the queue. The blast radius of the ignored page is larger than the blast radius of the rule that should have been tested.

The unit test catches the rule-level mistakes. The synthetic series test catches the data-level mistakes. The end-to-end test catches the routing mistakes. The scheduled canary catches the chain-level mistakes. None of them catches all four. The discipline is to run all four and to know which tier catches which shape.

How it works

The four tiers map to four questions about the alerting chain.

  Q1: Does the rule expression do what we claim?
        |
        v
  Tier 1: Unit test
          - Synthetic input_series at controlled timestamps
          - Asserts alert state, labels, annotations
          - No Prometheus required
          - Catches: wrong expr, wrong for:, missing metric

  Q2: Does the rule evaluate correctly against the live TSDB?
        |
        v
  Tier 2: Synthetic series test
          - Inject a synthetic series into a non-prod Prometheus
          - Wait for the rule to evaluate over the synthetic data
          - Assert the rule fires as expected
          - Catches: label drift, schema drift, join mistakes,
            missing series after exporter upgrades

  Q3: Does the alert reach the receiver?
        |
        v
  Tier 3: End-to-end test
          - Trigger a known condition in staging
          - Confirm Alertmanager fires
          - Confirm the receiver (webhook, Slack, PagerDuty)
            receives the notification with the expected
            payload
          - Catches: routing tree mistakes, receiver
            credentials expiry, template mistakes, network
            reachability

  Q4: Is the chain still alive in production?
        |
        v
  Tier 4: Scheduled canary
          - A rule that fires on a fixed schedule (for
            example, an `up == 0` rule on a synthetic
            canary target, or a `time() - alertmanager_last_run`
            rule on a known-good alert)
          - Confirms that the chain is alive end-to-end
          - Catches: dead exporter, dead Alertmanager, dead
            receiver, dead route, dead paging integration

The right per-tier approach is to ask “what class of mistake would I miss without this tier?” and to allocate the budget accordingly.

How to configure it

The four tiers are configured in different layers of the stack. A worked example for the same OrdersApiHighErrorRate rule:

Tier 1 — unit test (committed alongside the rule):

# observability/prometheus/rules/test/checkout_test.yml
rule_files:
  - ../checkout.yml

evaluation_interval: 30s

tests:
  - interval: 30s
    name: error ratio at 10 percent fires after dwell
    input_series:
      - series: 'http_requests_total{service="orders-api",status="200"}'
        values: '0 100 100 100 100 100 100 100 100 100 100 100 100'
      - series: 'http_requests_total{service="orders-api",status="500"}'
        values: '0 10 10 10 10 10 10 10 10 10 10 10 10'
    alert_rule_test:
      - eval_time: 6m
        alertname: OrdersApiHighErrorRate
        exp_alerts:
          - exp_labels:
              severity: critical
              team: checkout
            exp_annotations:
              summary: 'orders-api 5xx ratio above 5%'
              runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

Tier 2 — synthetic series (injected via a test exporter):

# observability/prometheus/test-exporter/docker-compose.yml
services:
  prometheus-test:
    image: prom/prometheus:v2.55.1
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    ports:
      - "9091:9090"

  synthetic-app:
    image: your-registry/synthetic-orders-api:1.0.0
    environment:
      - ERROR_RATIO=0.10
      - DURATION_SECONDS=420
    ports:
      - "9101:9100"

# observability/prometheus/test-exporter/prometheus.yml
scrape_configs:
  - job_name: synthetic-orders-api
    static_configs:
      - targets: ['synthetic-app:9100']
    metrics_path: /metrics

The synthetic exporter publishes http_requests_total series that drive the rule over threshold. The test waits 6 minutes (5 minutes for: dwell + 1 minute for evaluation) and then asserts the alert fired.

Tier 3 — end-to-end (Prometheus + Alertmanager + receiver):

# observability/alertmanager/test/alertmanager.yml
route:
  receiver: test-webhook
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
receivers:
  - name: test-webhook
    webhook_configs:
      - url: 'http://test-receiver:8080/alerts'

The test receiver is a small HTTP service that records every POST. The test asserts the receiver got a POST with the expected alertname and severity.

Tier 4 — scheduled canary (in production):

# observability/prometheus/rules/canary.yml
groups:
  - name: alert-pipeline-canary
    interval: 1m
    rules:
      - alert: AlertPipelineCanary
        expr: |
          time() - alertmanager_last_evaluation_timestamp_seconds
            > 300
        for: 0m
        labels:
          severity: warning
          team: observability
        annotations:
          summary: 'Alertmanager has not evaluated alerts in 5 minutes'
          runbook_url: 'https://runbooks.example.com/observability/alertmanager-stalled'

The canary fires whenever Alertmanager stops evaluating. It is itself an alert that goes through the full chain. If the chain is broken, the canary does not page; the absence of the canary page is the signal. This is a meta-alert; the discipline of meta-alerts is a separate lesson.

How to validate it

Three checks confirm the four-tier discipline is in place.

1. Each rule file has a unit test.

find observability/prometheus/rules \
  -name '*.yml' -not -path '*/test/*' \
  | while read -r rule; do
      test="observability/prometheus/rules/test/$(basename "${rule%.yml}")_test.yml"
      if [ ! -f "$test" ]; then
        echo "MISSING: $rule has no unit test at $test"
      fi
    done

Expected output, exit 0, no MISSING lines. A non-empty output names every rule file without a fixture.

2. The unit test passes in CI.

promtool test rules observability/prometheus/rules/test/checkout_test.yml

Expected output, exit 0:

SUCCESS

A failing fixture exits non-zero and prints the diff. The CI job fails the merge.

3. The canary fires on a known-good alert path.

amtool alert query alertname=AlertPipelineCanary

Expected output (after the canary has had at least one scheduled firing):

Alertname          State     Active Since
AlertPipelineCanary firing    2026-08-13T03:00:00Z

A firing state on the canary means the chain is alive. If the canary is not firing when expected, the chain is broken.

How it can fail

Six failure modes appear repeatedly in teams that adopt alert testing without the four-tier discipline.

1. Only Tier 1 is wired. Symptom: a rule with a perfect unit test ships and never fires in production because the metric it references was renamed. Cause: Tier 1 tests the rule expression, not the live data. Fix: add Tier 2 (synthetic series test) against the actual metric name in the live TSDB.
2. Tier 3 is run against the production Alertmanager with a test receiver. Symptom: the test receiver is wired to the production Slack channel, and the test fires a real page. Cause: the staging Alertmanager is not a separate process. Fix: run Tier 3 against a staging Alertmanager with a test webhook URL; never point a test at the production receiver.
3. Tier 4 canary fires constantly. Symptom: the on-call rota is paged every five minutes for the canary alert. Cause: the canary’s for: is 0m and the expression evaluates true on every tick when Alertmanager pauses briefly for a config reload. Fix: set for: 5m on the canary so a single brief pause does not fire.
4. Tier 1 fixture is too weak. Symptom: the unit test passes but the rule fires under the wrong conditions in production. Cause: exp_alerts: [] does not exercise the firing path, or exp_series: omits value:. Fix: assert the full label set and numeric value.
5. Tier 2 synthetic series drift. Symptom: the synthetic exporter publishes a metric name that the rule’s expr: no longer matches after a refactor. Cause: the synthetic exporter is maintained separately from the rule. Fix: generate the synthetic exporter’s metrics from the rule’s expr: (parse the label selectors from the rule file and emit a matching metric).
6. Tier 4 canary is on the same Prometheus as the alert. Symptom: the Prometheus that evaluates the canary dies, so the canary stops evaluating, so the canary does not fire, so the missing page is not detected. Cause: the canary shares the failure domain it is supposed to monitor. Fix: run the canary on a separate Prometheus (a different host, a different region) that has an independent view of the chain.

How to troubleshoot it

In order:

1. Confirm the tier exists. ls observability/prometheus/rules/test/ and confirm a fixture for the rule. curl -s http://alertmanager:9093/api/v2/alerts | jq and confirm the canary rule is loaded.
2. Confirm the tier ran. Check the CI pipeline for the last unit test run. Check the staging Prometheus for the last synthetic series run. Check the staging receiver for the last end-to-end run. Check the production Alertmanager for the canary’s last firing time.
3. Confirm the tier passed. promtool test rules exit code is 0. The synthetic series assertion passed. The end-to-end receiver got the POST. The canary fired.
4. Confirm the tier covers the right failure shape. A Tier 1 fixture that asserts exp_alerts: [] does not cover the firing path; a Tier 3 test that uses the production receiver does not cover the staging path. Re-read the tier’s purpose and confirm the fixture exercises it.

Security implications

• Tier 3 receivers must not point at production channels. A staging webhook that posts to the production Slack channel turns every test into a real page. Run Tier 3 against a dedicated test receiver.
• Tier 1 fixtures may reference realistic-looking credentials. A runbook_url with an embedded token commits that token to the repo. Use placeholder values; run a secret scanner in CI.
• Tier 4 canary exposes the alerting chain’s health. A canary that pages for “Alertmanager stalled” reveals that Alertmanager exists and is monitored. Treat the canary’s existence as a discoverable artefact; restrict the canary’s annotations to non-sensitive runbook URLs.

Performance implications

• Tier 1 cost: milliseconds per fixture, dominated by CI minutes. No production cost.
• Tier 2 cost: seconds per test, dominated by the staging Prometheus scrape interval and the rule’s for: dwell. No production cost.
• Tier 3 cost: tens of seconds per test, dominated by the staging Alertmanager’s group_wait and the test receiver’s POST handling. No production cost.
• Tier 4 cost: a single rule evaluating once per minute on the production Prometheus. Trivial cost; the rule’s expr: is a single subtraction. The canary is cheap to run continuously.

The total budget for the four-tier discipline is a few minutes of CI per pull request, a few minutes of staging per day, and a single rule evaluating once per minute in production. The cost is bounded; the benefit is bounded only by the size of the production failure surface.

Production guidance

• Adopt all four tiers, not one. Tier 1 catches the cheapest class of mistake and the fewest production failure shapes. Tier 4 catches the most expensive class of mistake. The four tiers together cover the full surface.
• Pin the promtool version in CI to match production. A two-version drift catches you out. Pin to the production Prometheus version.
• Run Tier 4 on a Prometheus that is not the Prometheus being monitored. The canary shares a failure domain with what it monitors otherwise. Run it on a separate host with an independent Alertmanager view.
• Have a second pair of eyes review the fixture against the runbook. A unit test that asserts the rule does the wrong thing is worse than no test at all. The human review layer is the discipline no test tier replaces.
• Treat alert testing as code. The fixtures live in the repository. The CI step runs on every pull request. The canary is a rule file with its own fixture. The discipline is enforceable in code review.

Verification

You should now be able to answer:

• What are the four tiers of alert testing, and what class of failure shape does each tier catch?
• Why does a unit test alone not catch the most expensive production alert failures?
• What is the difference between Tier 2 (synthetic series) and Tier 3 (end-to-end), and which failure shapes does each catch?
• Why must Tier 4 (the canary) run on a Prometheus that is not the Prometheus it monitors?
• What is the right per-tier cadence: continuous, on every PR, daily, weekly, or quarterly?

Quiz