RunBook AcademyObservability

← All checklists in Observability

Quarterlyalert-review

Alert Review Checklist

21 items ·13 critical ·7 warn ·1 info

Run this quarterly against the rule files as they exist in Git and as Prometheus has actually loaded them, and additionally after any exporter upgrade, instrumentation change or service decommission.

This checklist reads the rules. Whether an alert that fires reaches a human is the Alerting Readiness checklist; whether the alerts that fired last quarter deserved their pages is the Alert Quality Review. This one asks a narrower and more mechanical question: is each rule still a correct statement about a system that still exists.

Read the loaded rules, not only the files

Most of the commands here query the Prometheus rules API rather than grepping the repository, and the difference is the point. The repository is what somebody intended; the API is what is evaluating. A rule file that failed to load, a group that was renamed, a change that was merged but never reloaded — all of them leave the two views disagreeing, and every one of those disagreements is a rule that is not doing what its author believes it is doing.

Where an item names a path such as /etc/prometheus/rules/, substitute the path your estate uses.

What a failure means

The failure shape this review exists to catch is the rule that has stopped working without breaking. A renamed metric, a deleted service, a rotted fixture, a broken recording-rule name: in every case the rule still parses, still loads, still reports healthy, and simply never fires again. Nothing in the alerting chain reports a rule that has gone quiet, because a quiet rule is the normal state of a rule.

That is why so many items here are worded as “print the ones that do not” rather than as a pass/fail command. The finding is a list, and the list is the review.

Sign-off

Reviewer: ________________ Date: ___________
Service owner: ____________ Date: ___________
Platform owner: ___________ Date: ___________

Critical13 items

Every metric name the rule files reference still exists in the TSDB. A rule whose metric was renamed in an exporter upgrade parses, loads, evaluates to an empty vector and never fires again — and it is indistinguishable from a working rule by reading the file. This compares the names with conventional suffixes in the rule tree against the names Prometheus actually holds; anything printed is a rule watching a metric that is gone.
```
comm -23 <(grep -rhoE '[a-z_][a-z0-9_]*_(total|seconds|bytes|count|sum|bucket|info)' /etc/prometheus/rules/ | sort -u) <(curl -s http://prometheus:9090/api/v1/label/__name__/values | jq -r '.data[]' | sort -u)
```
Every alerting rule carries the labels the route tree matches on — at minimum an owner and a severity. A rule missing either does not fail; it falls through every child route to the catch-all, which is how a database alert ends up waking the platform rotation with no context about who owns the host.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | select((.labels.severity // "") == "" or (.labels.team // "") == "") | .name'
```
Every rule that can page carries `summary`, `description`, `runbook_url` and `dashboard_url`. The annotations are the only context the responder receives without effort, and the ten minutes spent loading it by hand comes out of the mitigation window, not out of the responder's day.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | select((.annotations.runbook_url // "") == "" or (.annotations.dashboard_url // "") == "" or (.annotations.summary // "") == "") | .name'
```
Every runbook URL in the rule tree returns a page. A 404 behind a page is worse than no link: the responder clicks it at 03:00, waits, gets nothing, and now has to find the procedure while the incident runs. Links rot silently because nothing in the alerting chain ever fetches them.
```
for url in $(grep -rh "runbook_url:" /etc/prometheus/rules/ | awk -F"'" '{print $2}'); do echo "$(curl -s -o /dev/null -w '%{http_code}' "$url") $url"; done
```
The rules that page describe a failure a user can feel — error ratio, latency against the SLO, a failing availability probe. Cause-based thresholds such as host CPU fire before impact, often by hours, and frequently resolve on their own; they belong in the ticket route. Read the expression of every paging rule and ask whether a user was affected when it last fired.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | select(.labels.severity=="page" or .labels.severity=="critical") | "\(.name): \(.query | gsub("\\s+"; " "))"'
```
Every SLO page is the conjunction of a short and a long burn-rate window, not a single window. A lone short window fires on the transient dip that recovers, which is the alert the rotation learns to mute; a lone long window is blind to the sharp incident. Each expression printed here should show two windows joined by `and`.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | select(.query | test("ratio_rate|burn")) | "\(.name): \(.query | gsub("\\s+"; " "))"'
```
The burn-rate multipliers match the SLO the team actually publishes and the window that SLO is measured over. The familiar 14.4x/6x page and 3x/1x ticket pairs are derived from a 30-day budget; carrying them onto a 7-day or 90-day SLO without redoing the arithmetic produces an alert whose threshold means something nobody intended.
Every rate-based rule that can page has an `absent()` companion. A threshold rule compares a value that is not there against a number and stays quiet, so a pipeline that stopped reporting looks exactly like a pipeline with nothing to report. The absence of the series is the only signal that failure emits, and only `absent()` sees it. This command inverts the convention used elsewhere in this list: it prints the rules that already have the companion, so read it against the list of paging rules from the symptom-based item and the gap between the two lists is the finding. An empty result is the worst case, not a pass — it means nothing in the estate alerts on a series going away.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | select(.query | test("absent")) | .name'
```
Every rule has a `for:` dwell, or the omission is a decision somebody defends in this review. Without it a single scrape across the threshold moves the rule straight to firing and it clears on the next evaluation, so the page arrives every interval for a condition that was never sustained.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | select((.duration // 0) == 0) | .name'
```
The rule unit tests run and pass against the current rule files. This is the one tier of alert testing that runs offline in milliseconds, which makes a failing or missing unit test a choice rather than a cost.
```
promtool test rules /etc/prometheus/rules/test/*.yml
```
The test fixtures have been read since the last exporter or instrumentation change. A fixture that still asserts the old metric name passes forever while the production rule it claims to cover matches nothing — the test cadence stays green and the discipline is silent. Check when the fixture tree was last touched against when the exporters were last upgraded.
```
RULES_REPO=/srv/observability; git -C "$RULES_REPO" log -1 --format='%ad %an %s' -- prometheus/rules/test/
```
No rule watches a service that no longer exists. Rules outlive the systems they were written for, and a rule for a decommissioned service either never fires — hiding the fact that nothing covers its replacement — or fires forever into a rotation that cannot act on it.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | .labels.service // "UNSET"' | sort -u
```
Every distinct owner label maps to a rotation that exists in the paging system and has people on it this quarter. A team label that was correct at the last reorganisation routes pages at a group that has been dissolved, and the delivery succeeds every time.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | .labels.team // "UNSET"' | sort | uniq -c
```

Warning7 items

The `summary` of every paging rule names what to do, not what the metric did. "Checkout 5xx above 2%" makes the responder infer the action; "Roll back the checkout deploy or scale payments-svc" is the action. This is the field read in the first five seconds, and it is the difference between mitigating and orienting.
Each threshold has been compared against the last quarter of the metric it watches. A threshold calibrated for last year's traffic fires daily against this year, and the rotation resolves that by ignoring it rather than by filing a finding. Record the number you compared against, not just the conclusion.
The severity values on the rules are spelled the way the route tree matches them. The course uses both `page`/`ticket` and `critical`/`warning`; an estate that mixes the two has rules that look correctly labelled and route to the catch-all. Compare the distinct values on the rules against the matchers in the Alertmanager configuration.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | .labels.severity // "UNSET"' | sort | uniq -c; grep -n severity /etc/alertmanager/alertmanager.yml
```
Non-production rules carry an environment label and the route tree filters on it. Staging alerts on the production rotation are the cheapest possible source of alert fatigue, and they are invariably introduced by copying a rule file rather than by anyone deciding it.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | .labels.environment // .labels.env // "UNSET"' | sort | uniq -c
```
Recording rules keep the colon-separated naming convention and no name changed in this review. The recording rule name is the contract between the SLI and everything that reads it; renaming one breaks every dashboard panel and every burn-rate alert built on it, and both fail by returning nothing rather than by erroring.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="recording") | .name' | grep -v ":"
```
No rule group takes longer to evaluate than its own interval. A group that overruns delays every alert inside it, including the ones that page, and the delay appears in the incident timeline as slow detection rather than as a monitoring fault. Anything printed here is a group running close to or past its budget.
```
curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[] | select(.evaluationTime > (.interval * 0.8)) | "\(.name) eval=\(.evaluationTime)s interval=\(.interval)s"'
```
No rule group has recorded an evaluation failure in the review window. A failing rule stops producing its series without removing the alert that reads it, so the alert simply never fires again and nothing in the alert list says so.
```
curl -s -G http://prometheus:9090/api/v1/query --data-urlencode 'query=increase(prometheus_rule_evaluation_failures_total[7d])' | jq -r '.data.result[] | "\(.metric.rule_group) \(.value[1])"'
```

Info1 item

Every finding leaves this review with a named owner and an agreed date, and the changes land as a pull request against the rule files. A review whose output is a document changes nothing about what pages the rotation next quarter, and the same findings reappear in the next one.