End-to-End Alert Test

A team adopts the unit test and synthetic series tiers. The unit test passes. The synthetic series drives the rule over threshold in staging. The rule fires. The on-call rota is happy. Three weeks later, an actual production outage occurs. The rule fires. The on-call rota is paged. The page says “checkout error rate elevated.” The receiver is Slack. The PagerDuty integration that the team thought was configured was deleted in a config refactor. The page lands in Slack, where nobody is watching at 03:00. The on-call rota wakes two hours later when a customer tweets.

The unit test caught the wrong expression. The synthetic series caught the live data drift. Neither caught the Alertmanager route deletion. The team had no end-to-end test. The lesson is that the alerting chain has four links — rule evaluation, Alertmanager routing, receiver delivery, human acknowledgement — and the test tiers catch the first two. The end-to-end test catches the third and the fourth.

What it is

An end-to-end alert test exercises the full alerting chain under controlled conditions and asserts that the receiver received the expected notification. The chain has four links:

  +-----------------------------------------+
  |  1. Rule evaluation                     |
  |    Prometheus evaluates the rule;       |
  |    if firing for the configured dwell,  |
  |    the alert is sent to Alertmanager.   |
  +-----------------------------------------+
                  |
                  v
  +-----------------------------------------+
  |  2. Alertmanager routing                |
  |    Alertmanager applies the routing     |
  |    tree (group_by, group_wait,          |
  |    group_interval, repeat_interval,     |
  |    routes, inhibitors, silencers);      |
  |    the alert is grouped, inhibited,     |
  |    and routed to one or more receivers. |
  +-----------------------------------------+
                  |
                  v
  +-----------------------------------------+
  |  3. Receiver delivery                   |
  |    The receiver (Slack webhook,         |
  |    PagerDuty integration, email,        |
  |    custom webhook) accepts the          |
  |    notification and delivers to the     |
  |    on-call channel.                     |
  +-----------------------------------------+
                  |
                  v
  +-----------------------------------------+
  |  4. Human acknowledgement               |
  |    The on-call rota sees the            |
  |    notification, opens the runbook,     |
  |    and acknowledges the alert.          |
  +-----------------------------------------+

The end-to-end test verifies links 1, 2, and 3. Link 4 is human; it cannot be automated. The test asserts that the receiver received a notification with the expected alertname, severity, and labels.

The right approach is to run the test against a staging Alertmanager with the same routing tree as production and a dedicated test receiver that records incoming POSTs. The test never touches the production receiver. The notification reaches a sandbox; the production notification chain is not affected.

The most common shape is:

1. A staging Prometheus evaluates the rule against a synthetic series.
2. A staging Alertmanager with the production routing tree receives the alert.
3. A staging test receiver (a small HTTP service) records the incoming POST.
4. The test asserts the receiver got the POST with the expected alertname, severity, and labels.

Why a sysadmin cares

Three failure shapes appear at the chain level that no single-tier test catches:

1. Alertmanager route deleted in a config refactor. The rule fires. Alertmanager has no route to the production receiver. The alert is dropped silently.
2. Receiver credential expired. The webhook URL or API key for Slack/PagerDuty is rotated; the old credential is in the Alertmanager config; the receiver returns 401; the alert is dropped silently (or Alertmanager retries until repeat_interval expires).
3. Routing template mistake. The Alertmanager template references a label that the rule does not set; the rendered notification is empty; the receiver accepts the POST; the on-call rota sees an empty page; the alert is effectively lost.

The end-to-end test catches all three. The unit test catches none of them. The synthetic series test catches only the rule-evaluation link.

The cost of the chain-level failure is paid in incident response. The team discovers the failure when the alert that should have fired for a real outage does not fire. The post-mortem names the missing link. The fix is to add the end-to-end test to the alerting discipline.

How it works

The end-to-end test is a controlled re-run of the production alerting chain against a staging environment. The chain is the same; only the destination is different.

  Synthetic series (staging Prometheus)
        |
        |  scrape
        v
  Staging Prometheus
        |
        |  rule evaluation; alert firing
        v
  Staging Alertmanager (same routing tree as production)
        |
        |  route match; group; inhibit
        v
  Test receiver (HTTP POST)
        |
        v
  Test assertion (POST payload matches expected)

The staging Prometheus evaluates the rule against the synthetic series. The rule fires. Alertmanager receives the alert, applies the routing tree, and routes the alert to the test receiver. The test receiver records the POST. The test asserts the POST payload matches the expected alertname, severity, and labels.

The test runs on a schedule (nightly) or on every pull request that touches the rule files, the Alertmanager config, or the receiver templates. A failing test posts to the observability chat channel and fails the CI job.

How to configure it

The end-to-end test has three components: the staging Alertmanager config, the test receiver, and the test harness.

The staging Alertmanager config (mirror of production):

# observability/alertmanager/test/alertmanager.yml
global:
  resolve_timeout: 5m

route:
  receiver: test-webhook
  group_by: ['alertname', 'severity']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  routes:
    - matchers:
        - severity = "critical"
      receiver: test-webhook-critical
    - matchers:
        - severity = "warning"
      receiver: test-webhook-warning
    - matchers:
        - test = "synthetic-canary"
      receiver: test-silence
      # Synthetic canary alerts go to a sink that drops them
      # so the on-call rota does not see them in staging.

receivers:
  - name: test-webhook
    webhook_configs:
      - url: 'http://test-receiver:8080/alerts'
        send_resolved: true
  - name: test-webhook-critical
    webhook_configs:
      - url: 'http://test-receiver:8080/alerts/critical'
        send_resolved: true
  - name: test-webhook-warning
    webhook_configs:
      - url: 'http://test-receiver:8080/alerts/warning'
        send_resolved: true
  - name: test-silence
    webhook_configs:
      - url: 'http://test-receiver:8080/alerts/sink'

inhibit_rules:
  - source_matchers:
      - severity = "critical"
    target_matchers:
      - severity = "warning"
    equal: ['alertname', 'service']

The routing tree mirrors production. The receivers point at the test receiver, not at Slack or PagerDuty. The synthetic-canary route sends canary alerts to a sink so they do not clutter the test output.

The test receiver (a small HTTP service):

# observability/alertmanager/test/test-receiver.py
from flask import Flask, request, jsonify
from datetime import datetime
import json

app = Flask(__name__)
received = []

@app.post("/alerts/<path:path>")
def alerts(path):
    payload = request.get_json(force=True)
    received.append({
        "path": path,
        "payload": payload,
        "received_at": datetime.utcnow().isoformat(),
    })
    with open("/tmp/test-receiver.json", "w") as f:
        json.dump(received, f, indent=2)
    return jsonify({"status": "ok"}), 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

The test receiver records every incoming POST to /tmp/test-receiver.json. The test harness reads the file after the dwell and asserts the expected alert is present.

The test harness:

#!/usr/bin/env bash
# observability/alertmanager/test/test.sh
set -euo pipefail

RECEIVER_LOG=/tmp/test-receiver.json
TEST_LABEL="test=e2e-$(date +%s)"

echo "Driving synthetic series to fire rule"
# Trigger the synthetic series via the test exporter's
# admin endpoint. The synthetic series carries the
# test label so the rule picks it up.
curl -s -X POST http://synthetic-app:9101/admin/fire \
  -d "label=${TEST_LABEL}"

echo "Waiting 6 minutes for rule to evaluate and route"
sleep 360

echo "Reading receiver log"
python3 - <<EOF
import json
with open("${RECEIVER_LOG}") as f:
    received = json.load(f)

matched = [
    r for r in received
    if any(
        a.get("labels", {}).get("alertname") == "OrdersApiHighErrorRate"
        and a.get("labels", {}).get("severity") == "critical"
        for a in r["payload"].get("alerts", [])
    )
]

if not matched:
    print("FAIL: alert not delivered to receiver")
    exit(1)

alert = matched[0]["payload"]["alerts"][0]
expected_summary = "orders-api 5xx ratio above 5%"
if expected_summary not in alert["annotations"].get("summary", ""):
    print(f"FAIL: summary mismatch: {alert['annotations']}")
    exit(1)

print("PASS")
EOF

echo "Cleaning up"
curl -s -X POST http://synthetic-app:9101/admin/clear \
  -d "label=${TEST_LABEL}"
rm -f /tmp/test-receiver.json

The test harness drives the synthetic series, waits for the rule’s for: dwell plus the Alertmanager’s group_wait, reads the receiver log, asserts the expected alert is present, and cleans up.

How to validate it

Three checks confirm the end-to-end test is wired correctly.

1. The staging Alertmanager has the same routing tree as production.

diff <(yq '.route' observability/alertmanager/production/alertmanager.yml) \
     <(yq '.route' observability/alertmanager/test/alertmanager.yml)

Expected output: no diff. A non-empty diff names the routes that differ. The fix is to update the staging config to match production (or to update production and re-run the diff to confirm the staging config is up to date).

2. The test receiver accepts the POST.

curl -s -X POST http://test-receiver:8080/alerts \
  -H 'Content-Type: application/json' \
  -d '{"alerts": [{"labels": {"alertname": "TestAlert"}}]}'

Expected output, exit 0:

{"status": "ok"}

A non-2xx response means the receiver is misconfigured or the JSON shape is wrong.

3. The end-to-end test passes against a known-good alert.

observability/alertmanager/test/test.sh

Expected output, exit 0:

Driving synthetic series to fire rule
Waiting 6 minutes for rule to evaluate and route
Reading receiver log
PASS

A PASS confirms the rule fired, Alertmanager routed, and the receiver received the expected payload.

How it can fail

Six failure modes appear repeatedly in end-to-end alert tests.

1. The staging Alertmanager config drifts from production. Symptom: the test passes in staging but a production route change does not propagate. Cause: the staging config is maintained separately. Fix: generate the staging config from the production config with a sed substitution that replaces receiver URLs with the test receiver URL.
2. The test receiver is wired to the production Slack channel. Symptom: every test fires a real production page. Cause: the test receiver’s webhook URL is the production Slack webhook. Fix: point the test receiver at a sandbox Slack channel or a local HTTP sink.
3. The group_wait is too short for the test. Symptom: the test reads the receiver log before Alertmanager has sent the alert; the assertion fails. Cause: the test harness waits for: dwell but not group_wait. Fix: add group_wait to the test harness’s sleep duration.
4. The rule’s for: dwell is not honoured in staging. Symptom: the rule fires in the unit test but not in staging. Cause: the staging Prometheus evaluates the rule at a different interval than production. Fix: set the rule group’s interval: to match production.
5. The receiver template references a label that the rule does not set. Symptom: the rendered notification is empty; the receiver accepts the POST; the on-call rota sees an empty page. Cause: the template uses {{ .Labels.foo }} but the rule’s expr: does not include foo. Fix: update the rule’s labels: block to set foo, or remove {{ .Labels.foo }} from the template.
6. The Alertmanager config fails to load. Symptom: the test harness times out; the receiver log is empty. Cause: the staging Alertmanager config has a YAML error. Fix: run amtool check-config observability/alertmanager/test/alertmanager.yml and fix the reported error.

How to troubleshoot it

In order:

1. Did the rule fire? curl -s http://prometheus-staging:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.name=="OrdersApiHighErrorRate") | \{state, health, lastError\}'. A non-firing state means the rule did not fire; check the synthetic series.
2. Did Alertmanager receive the alert? amtool alert query alertname=OrdersApiHighErrorRate. An empty result means Prometheus did not send the alert to Alertmanager; check the alerting: URL in the Prometheus config.
3. Did the route match? amtool alert query alertname=OrdersApiHighErrorRate shows the receiver the alert was routed to. An empty receiver means the routing tree dropped the alert; check inhibit_rules and route matchers.
4. Did the receiver accept the POST? Check the receiver access log for the incoming POST. A 4xx or 5xx response means the receiver rejected the POST; check the template.
5. Did the rendered notification contain the expected fields? Read the receiver’s recorded payload and compare against the expected alertname, severity, and labels.

Security implications

• The test receiver accepts unauthenticated POSTs. A staging Alertmanager that sends to a public receiver URL leaks alert payloads. The test receiver must be on a private network with no public ingress.
• The staging Alertmanager config may contain production credentials. A copy-paste from production to staging can include the Slack webhook URL or PagerDuty API key. Use a substitution step that replaces production credentials with placeholder values; verify with a secret scanner.
• The end-to-end test fires synthetic alerts. A test that uses realistic-looking alert payloads may include customer-impact descriptions. Use synthetic payloads; do not include real customer identifiers.

Performance implications

• The end-to-end test is slow. A test that waits for the rule’s for: dwell (5 minutes), Alertmanager’s group_wait (10 seconds), and the receiver’s POST handling (sub-second) takes roughly 5.5 minutes per scenario. A nightly run with ten scenarios takes 55 minutes; budget the CI step accordingly.
• The staging Prometheus and Alertmanager are lightweight. A staging Prometheus evaluating one rule per minute against a synthetic series uses negligible resources. The staging Alertmanager handling a handful of alerts per hour is similarly trivial. The cost is the test duration, not the resource usage.

Production guidance

• Generate the staging config from production. A substitution step (typically sed or gomplate) replaces production receiver URLs with test receiver URLs. The staging config is a derivative and cannot drift.
• Run the test on a schedule and on PRs. A nightly run catches drift. A PR-time run catches changes before merge.
• Assert the receiver got the expected payload. A test that asserts only “the receiver got a POST” is too weak. Assert the alertname, severity, and the rendered summary or description.
• Never point the test receiver at the production notification chain. The test receiver is a sandbox. Production notifications are not a test fixture.
• Include the Alertmanager config in the same repository as the rule files. The rule files and the Alertmanager config are one alerting artefact. They belong in the same review and the same CI job.

Verification

You should now be able to answer:

• What are the four links of the alerting chain, and which links does the end-to-end test cover?
• Why must the staging Alertmanager config be derived from production (not maintained separately)?
• What is the most common shape of an end-to-end alert test, and what three subsystems does it exercise?
• Why must the test receiver never point at the production notification chain?
• What is the failure shape of a test that asserts only “the receiver got a POST” without checking the payload?

Quiz