How to use this review
Quarterly, and it requires causing conditions rather than reading configuration.
Monitoring fails in three ways and this review is organised around them: an alert that does not exist, an alert that exists and has never been shown to fire, and a metric that reports health it has not observed. The third is the worst, because it is confidently wrong.
The failure mode that hid six weeks
Test every alert by causing its condition
Each of these takes a minute and none of them harms anything:
| Alert | How to make it fire |
|---|---|
| Replication lag | SELECT pg_wal_replay_pause(); on the standby |
| Standby absent | Stop the standby |
| Connection utilisation | Open connections in a test until the threshold |
| Backup failure | Break the backup destination once |
| Archive failure | Point archive_command at an unwritable path briefly |
| Collection failure | Stop the exporter |
Metrics that report starts, not completions
Thresholds in units that mean something
Disk: hours of headroom at the measured write rate, not a percentage. Ninety-five percent of a 20 GB volume was sixteen minutes.
Replication: bytes behind, not seconds since the last commit. The time-based metric read 17 seconds on a perfectly healthy pair and normal during a 3.1 GB lag.
Connections: 75 percent of max_connections, not 99. A threshold
that fires when the slots are gone leaves no time, and may leave the
responder locked out.
Review the alerts that did fire
An alert firing weekly and being right once a year is not monitoring; it is training. The team learns to acknowledge it without reading it, and then mutes it, and then the real one arrives.
Include the false-positive rate in this review, and treat a muted alert as an open item with an owner and an expiry.
Where the numbers come from
Coverage comes from the alert rules themselves, checked against the failure list this course teaches rather than against what happens to be graphed. Thresholds come from the estate’s own measured baseline — retained series, not a default someone copied.
Whether an alert works comes from it having fired, in this environment, at least once: deliberately during a test or genuinely during an incident. An alert that has never fired is untested, and the review records it as such rather than as passing.
Access this needs
Read access to the monitoring system: the metric series, the alert rules, and the record of which alerts have fired and what was done about them.
A role holding pg_monitor on the cluster to confirm that each metric
the alerts depend on is actually collectable by the exporter’s own role,
which is a different question from whether it is collectable by yours.
Read access to the on-call rotation and to the runbook each alert points at, because an alert with no runbook is an unfinished alert and that is an item here.
What the review produces
A dated record naming the reviewer and the disposition of every item, with the alert inventory attached: rule, threshold, the series it reads, the runbook it points at, and the date it last fired.
Two findings outrank the rest and go to the service owner: an alert firing routinely and being ignored, which is a defect in the alert; and a failure mode with no alert at all, which is a gap the estate is currently carrying without knowing it.
Sign-off
- Reviewer: ________________ Date: ___________
- Database owner: ___________ Date: ___________
- Service owner: ____________ Date: ___________
Every critical item must pass. A failing critical item is a blocker, not a note for the next sprint: record the date, the reviewer, the disposition of every item that did not pass, and the name of whoever accepted the residual risk.