RunBook AcademyObservability

← All checklists in Observability

Quarterlyalert-review

Alert Quality Review

20 items ·11 critical ·8 warn ·1 info

Run this at least quarterly, on a calendar slot that is not negotiable, and treat the deferral of the review as a finding in its own right. Quarterly is the floor rather than the target: the course lesson on the alert quality review argues for monthly, on the grounds that bad alerts accumulate faster than a quarterly cadence can remove them. If your page volume is growing, run it monthly and shorten the windows in the commands below to match.

This is the outcome review. The Alerting Readiness checklist asks whether the chain can deliver; the Alert Review checklist asks whether each rule is a correct statement about a system that exists. This one asks a question neither of those can: given everything that happened last quarter, is the alerting stack earning the attention it costs.

Where the evidence comes from

Alertmanager is not the history. It holds the alerts that are currently active plus a short notification log, so a quarter-long audit reads either the ALERTS series that Prometheus generates for every firing rule, or a copy of the notification stream shipped somewhere that retains it. Establish which of those you have as the first item; every ranking below is only as long as that window.

Two of the most important numbers — pages actually delivered, and what a human did next — are not in the observability stack at all. They live in the paging system and the incident record. If those two cannot be joined to an alert name, the action rate cannot be computed, and that gap is the finding.

Where an item names a path or a Prometheus address, substitute the ones your estate uses. The 30d windows assume Prometheus retention covers the review period; widen or narrow them to match.

What this review produces

A pull request against the rule files, not a document. Each rule that fired gets one of three dispositions — keep, redesign, delete — and the disposition is the diff. The most suspicious possible outcome of an audit is an empty diff.

Sign-off

Reviewer: ________________ Date: ___________
On-call lead: _____________ Date: ___________
Platform owner: ___________ Date: ___________

Critical11 items

You can see the firing history for the whole review window before the review starts. Alertmanager holds current and recently resolved state only — its notification log is garbage-collected on a retention measured in days — so an audit that reaches back a quarter has to read the `ALERTS` series in Prometheus or a shipped copy of the notification stream. This prints how far back the live Alertmanager view reaches; if that is hours, the rest of this review must come from the long-term store.
```
amtool alert query --output=json | jq -r '[.[] | .startsAt] | min'
```
The rules are ranked by how much of the window they spent firing, and the top of that list has been read. `ALERTS` emits a sample per evaluation while a rule is firing, so this measures firing duration rather than page count — which is the right question for fatigue, because a rule that fires for six days straight costs the rotation more than one that fires six times.
```
curl -s -G http://prometheus:9090/api/v1/query --data-urlencode 'query=topk(20, sum by (alertname) (count_over_time(ALERTS{alertstate="firing"}[30d])))' | jq -r '.data.result[] | "\(.value[1]) \(.metric.alertname)"'
```
The number of pages each rotation received in the window is a number somebody has looked up in the paging system, and it is compared against the budget the team agreed. Delivered pages are not the same as fired alerts — grouping, inhibition and silences all sit between them — so the paging system is the only source that answers this. A rotation above roughly two hundred pages a month is being trained to ignore the pager.
Every rule that fired has an action rate: the fraction of its firings that led to a human doing something. Below roughly thirty percent the rule is a candidate for redesign or deletion. The number comes from joining the firing history with the incident record, and a rule whose action rate cannot be computed at all is itself a finding — it means nothing links the page to what happened next.
The false-positive rate is measured and is under five percent. A false positive is specifically a page where a human investigated and found no incident, which is not the same as a noisy alert the team has learned to dismiss. Fewer than one page in twenty should end that way; above that the rules are describing a system that does not exist.
Every rule that never fired in the window has been looked at, and each is either genuinely rare or dead. A renamed metric, a decommissioned service and a well-designed rare-event alert are indistinguishable by their firing count — the difference only shows up during the incident, when two of the three stay quiet. This lists rules Prometheus has loaded that produced no `ALERTS` series at all.
```
comm -23 <(curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | .name' | sort -u) <(curl -s -G http://prometheus:9090/api/v1/query --data-urlencode 'query=count by (alertname) (count_over_time(ALERTS[30d]))' | jq -r '.data.result[].metric.alertname' | sort -u)
```
No rule has been silenced continuously through the window. A long-lived silence is a deletion nobody wrote down: it removes the alert without removing the rule, so the rule file still claims coverage the estate does not have, and the silence is still in force during the incident it was created to mute. Read the end time, the creator and the comment on every active silence.
```
amtool silence query --output=json | jq -r '.[] | "\(.endsAt) \(.createdBy) \(.comment)"'
```
The canary arrived on schedule for the whole window, and every gap was investigated. Without it, "a quiet quarter" and "a quarter in which no page could be delivered" produce identical evidence, and every conclusion in this review rests on the assumption that the chain was alive.
```
curl -s -G http://prometheus:9090/api/v1/query --data-urlencode 'query=count_over_time(ALERTS{alertname=~".*Canary.*|Watchdog", alertstate="firing"}[30d])' | jq -r '.data.result[] | "\(.metric.alertname) \(.value[1])"'
```
Notification failures over the window have been counted and explained. A non-zero count with `reason="config"` is a receiver that was never able to send; `client` is a rejected request, typically a rotated credential; `server` is an upstream that was down. Any of the three means some subset of the pages in this review never reached anybody, and the alert list will not say which.
```
curl -s -G http://prometheus:9090/api/v1/query --data-urlencode 'query=increase(alertmanager_notifications_failed_total[30d])' | jq -r '.data.result[] | "\(.metric.integration) \(.metric.reason) \(.value[1])"'
```
The decisions from the previous review are visible in the rule files. A review whose output was a document and not a merged change has not altered what pages the rotation, and the same rules appear at the top of the firing-time ranking next quarter. The commit log is the evidence that the review is a control loop rather than a report.
```
RULES_REPO=/srv/observability; git -C "$RULES_REPO" log --since='3 months ago' --oneline -- prometheus/rules/
```
Any deletion of a paging rule was agreed by two people. A rare-but-real alert has a low firing count and a low action rate, which is exactly the profile this review flags for deletion — and it is also the profile of the alert that covers the failure nobody has seen yet. The second reviewer exists to ask what the rule was written for.

Warning8 items

Time from page to mitigation is measured, and it comes from the incident record rather than from Alertmanager. Alertmanager counters describe notifications, not human response; deriving a mitigation time from them produces a number that looks precise and means nothing.
The pages that arrived outside working hours have been listed and each one is a condition that genuinely could not wait until morning. Out-of-hours volume is the part of the fatigue cost that the rotation feels most and that the aggregate page count hides.
Take the largest multi-host incident in the window and count the notifications the rotation actually received. One incident should have produced one conversation, not one notification per instance. If the count matches the host count, the `group_by` labels include something per-instance.
```
grep -nE 'group_by|group_wait|group_interval' /etc/alertmanager/alertmanager.yml
```
For that same incident, the dependent alerts were suppressed and the cause alert was not. Both directions are failures: no suppression is the thirty-page flood, and suppression of the parent is an incident that produced no page at all. The nflog and the alert list from the incident are the evidence.
For incidents that ran longer than `repeat_interval`, the re-notification was useful rather than noise. A four-hour repeat during a six-hour incident that the team is already working delivers a page into a room where everyone is already awake, which teaches the rotation that repeats can be ignored.
```
grep -n repeat_interval /etc/alertmanager/alertmanager.yml
```
Every rule added since the last review went through the alert review checklist before it shipped. Unreviewed growth is how a rule file gains forty percent in two quarters, and it arrives one plausible-looking rule at a time. Tag the rule tree at each review so this diff is a single command.
```
RULES_REPO=/srv/observability; PREV_TAG=alert-review-2026-Q2; git -C "$RULES_REPO" diff --stat "$PREV_TAG" -- prometheus/rules/
```
This review ran when it was due, and any deferral is recorded with a reason. Deferral is the normal failure mode: the rotation is busy precisely when the alerting is worst, so the quarter that most needs the review is the quarter it gets skipped, and two skipped quarters is a rule file nobody can read in an afternoon.
Somebody asked the on-call rotation whether they still trust the pager. A muted notification channel, a phone on do-not-disturb, a Slack keyword filter — none of these appear in any metric, and all of them mean the delivery numbers in this review describe a chain that ends before the human.

Info1 item

The deletions, redesigns and open findings are written down with an owner and a date, and a short summary goes to the team. Keeping the audit visible is what makes the next one easier to schedule, and it is the only record that explains, six months later, why a rule the team half-remembers is no longer there.