RunBook AcademyObservability

← All checklists in Observability

Before deploymentalerting-readiness

Alerting Readiness

21 items ·15 critical ·5 warn ·1 info

Run this before an alerting stack carries its first production page, and again before any change to the routing tree, the receivers, the inhibit rules or the set of people the pages reach.

This checklist is about the chain, not the rules. It asks whether an alert that fires reaches a human who can act, and whether you would find out if it stopped doing so. The content of the rules themselves — labels, annotations, thresholds, tests, burn-rate windows — is the Alert Review checklist, and the question of whether the rules that fired last quarter earned their pages is the Alert Quality Review. Running this one alone gives you a chain that reliably delivers rules nobody has audited.

What a failure means

Every critical item here describes a way a real page disappears without an error anywhere. That is the defining property of this class of fault: the rule evaluates, the alert appears in the Alertmanager list, the config is valid, the process is healthy, and no notification arrives. Each of the individual components reports success, because each of them did succeed at its own job.

The two items with the widest blast radius are the inhibit-rule scope and the catch-all receiver. Both convert a working alerting system into one that silently discards a subset of alerts, and both look correct in review.

Access this needs

amtool and promtool on a host that can reach Alertmanager on 9093 and Prometheus on 9090, plus read access to alertmanager.yml and the rule files. The amtool alert add items push synthetic alerts into the live Alertmanager; they will notify real receivers, so agree the window with the on-call rotation first and expire the test alerts afterwards. Everything else is a read.

Four items — the out-of-band canary path, the staffed rotation, the agreed page budget, and the record of the review — are attested by a person. No command can answer them, and they are the four most likely to be assumed rather than checked.

Sign-off

Reviewer: ________________ Date: ___________
Platform owner: ___________ Date: ___________
On-call lead: _____________ Date: ___________

Critical15 items

The running Alertmanager is serving the configuration you think it is. `amtool check-config` parses the file on disk; the reload timestamp proves the process accepted it. A failed reload leaves the previous configuration in place and the process healthy, so a broken routing tree can be deployed, look green, and route by last week's rules.
```
amtool check-config /etc/alertmanager/alertmanager.yml && curl -s http://alertmanager:9093/metrics | grep alertmanager_config_last_reload_success_timestamp_seconds
```
Every rule file parses before Prometheus is asked to load it. `promtool check rules` is the only step in the chain that costs nothing and runs offline, which makes it the cheapest place to catch a typo that would otherwise take the whole rule group out of evaluation.
```
promtool check rules /etc/prometheus/rules/*.yml
```
For every severity you ship, the route tree resolves to the receiver you intended. Test the label set, not the file: the tree is ordered and a matching route with `continue: false` ends the walk, so a route somebody added above yours silently takes your pages.
```
amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml severity=page team=payments service=checkout
```
The receiver at the root of the tree is somewhere a human reads, not a null receiver. An alert whose `team` label is missing or misspelled falls through every child route to the root; if the root discards it, the alert was accepted, grouped, and thrown away with no error in any log.
```
amtool config routes show --config.file=/etc/alertmanager/alertmanager.yml
```
Each receiver has been proven by a notification that actually arrived, in this environment, since the last credential rotation. A rotated PagerDuty routing key or a regenerated Slack webhook leaves the configuration valid and every notify attempt returning 401; nothing about that state is visible from the Alertmanager alert list.
```
amtool alert add alertname=ReadinessProbe severity=page team=payments --alertmanager.url=http://alertmanager:9093
```
A rule watches `alertmanager_notifications_failed_total`. Delivery failure is the one fault the alerting system cannot page you about through the path that is failing, so it has to be a rule somebody wrote deliberately. The `reason` label separates a misconfigured receiver from a rejected request from an upstream outage.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | .query' | grep -c alertmanager_notifications_failed_total
```
A rule watches the age of `alertmanager_config_last_reload_success_timestamp_seconds`. Without it, a config that stopped loading is indistinguishable from one that loads fine, and the divergence between the file in Git and the tree in memory grows silently until someone tests a route.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | .query' | grep -c alertmanager_config_last_reload_success_timestamp_seconds
```
A canary alert exercises the production chain on a fixed cadence. Unit tests prove the expression, staging tests prove the routing tree, and neither notices that the production webhook was rotated six weeks ago. The canary is the only tier that runs against the chain that will carry the real page.
```
amtool alert query --output=json --alertmanager.url=http://alertmanager:9093 | jq -r '.[] | select(.labels.alertname | test("Canary|Watchdog")) | "\(.labels.alertname) \(.startsAt)"'
```
The absence of the canary reaches a human by a path that does not traverse the production Alertmanager. A canary whose missing-page alert routes through the component it is testing tells you nothing: the failure it is designed to detect is exactly the failure that suppresses the report of it.
Something outside this failure domain scrapes Prometheus, Alertmanager and the rest of the platform, and pages through its own Alertmanager. A Prometheus scraping a second Prometheus on the same host is co-monitoring, not meta-monitoring: one power failure takes the observer and the observed together.
```
curl -s http://meta-prometheus:9090/api/v1/targets | jq -r '.data.activeTargets[] | "\(.labels.job) \(.health)"'
```
Every `inhibit_rule` carries an `equal:` clause. Without it the rule matches across the whole estate: a critical on the edge router suppresses warnings on the database cluster, and the suppression is invisible because the target alert is still listed as firing. Compare the two counts this prints; they must match.
```
awk '/^inhibit_rules:/{f=1} f' /etc/alertmanager/alertmanager.yml | grep -c source_matchers; awk '/^inhibit_rules:/{f=1} f' /etc/alertmanager/alertmanager.yml | grep -c "equal:"
```
The scope of each inhibit rule has been demonstrated with two synthetic alerts in different scopes, not reasoned about from the YAML. Add the source in one cluster and the target in another; the second must still notify. This is the test that catches a missing label in `equal:`, which reads correctly and behaves fleet-wide.
```
amtool alert add alertname=ClusterDown cluster=prod-eu-1 severity=critical; amtool alert add alertname=HostDown cluster=prod-us-1 severity=critical
```
Page and ticket are different labels routed to different receivers, and somebody has looked at how many rules carry each. A rule file where every alert is page-severity has not made the decision; it has deferred it to the on-call engineer at 03:00, who makes it by muting the channel.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | .labels.severity' | sort | uniq -c
```
Every paging receiver terminates at a named rotation that is staffed for the hours the page can fire, with a written escalation path when the first responder does not acknowledge. A receiver pointing at a service with nobody on the schedule delivers successfully and wakes no one, and every metric in the chain reports success.
The team has agreed, before go-live, what a tolerable page volume per rotation per week looks like, and who reviews it when the number is exceeded. Without a number, alert fatigue has no threshold, so it is only ever noticed retrospectively in a post-incident review.

Warning5 items

If Alertmanager runs as a cluster, each instance sees the peers you configured. A peer that never joined behaves as a working single instance until the moment you rely on the other one, and the deduplication that clustering exists to provide is quietly absent.
```
curl -s http://alertmanager:9093/metrics | grep -E '^alertmanager_cluster_(peers|members)'
```
The `group_by` labels are the ones shared by every alert in a single incident, and `group_wait`, `group_interval` and `repeat_interval` are deliberate values rather than defaults nobody read. Adding a per-instance label to `group_by` turns a thirty-host outage into thirty notifications, each individually correct.
```
grep -nE 'group_by|group_wait|group_interval|repeat_interval' /etc/alertmanager/alertmanager.yml
```
Every active silence has an end time you would defend and a comment naming why it exists and what to do when it expires. A silence scoped only by `alertname` covers every instance of that alert across the estate, and a comment reading "ok" is operationally invisible six months later when the real incident is the one being muted.
```
amtool silence query --output=json | jq -r '.[] | "\(.endsAt) \(.createdBy) \(.comment)"'
```
Recurring maintenance windows are declared as time intervals in the configuration rather than re-created as ad-hoc silences by whoever remembers. A declared window is reviewable in Git; a silence created at 02:00 is a decision nobody else can see and nobody expires.
```
grep -nE 'mute_time_intervals|active_time_intervals|time_intervals' /etc/alertmanager/alertmanager.yml
```
The dispatcher aggregation-group limit is watched. When a group exceeds the limit the dispatcher drops alerts, which looks from the outside exactly like alerts that never fired, and the counter is the only place the loss is recorded.
```
curl -s http://alertmanager:9093/metrics | grep alertmanager_dispatcher_aggregation_group_limit_reached_total
```

Info1 item

The review, its findings, the owner of each open item and the agreed date are recorded. A readiness review nobody can produce afterwards is indistinguishable from one that did not happen, and the items with no command are exactly the ones an auditor asks about.