Alert Quality Review

A team adopts a quarterly alert review in March. By June the review has been deferred three times because the on-call rotation is busy. By September the review has not run for two quarters and the rule file has grown by forty percent. By December the on-call engineer has muted the page channel and the alerting system has become a cost centre rather than a control system.

The cost of “we will get to it” is the cost of letting bad alerts accumulate. The discipline of the alert quality review is that the review runs on cadence regardless of how busy the team is. The review is the only mechanism that keeps the rule file small and the page channel useful.

What it is

The alert quality review is a scheduled, recurring audit of every alert that has fired in the review window. The review inspects each alert for owner, runbook URL, false-positive rate, and action rate. The audit output is a PR that deletes, redesigns, or certifies each alert. The PR is reviewed by the team; the team’s collective decision is the documentation.

The review is monthly, not quarterly. Bad alerts accumulate faster than a quarterly cadence can remove them. A quarterly cadence is too slow.

Why a sysadmin cares

The cost of skipping the review is alert fatigue, missed incidents, and an exhausted error budget. The cost of running the review is roughly four hours per month of auditor time and a PR review per finding. The cost-benefit is asymmetric: the cost of the review is bounded; the cost of skipping it is unbounded.

How it works

The review is a four-step process. Each step has a clear input and a clear output.

  Alert quality review (monthly)
  ==============================

  Step 1: Inventory
    Input:  rule files in Git, Alertmanager alert history
    Output: list of every alert that fired in the last 30 days

  Step 2: Per-alert audit
    Input:  each alert's firing history, action rate,
            owner label, runbook URL
    Output: KEEP / REDESIGN / DELETE decision per alert

  Step 3: PR
    Input:  the decisions
    Output: PR against rule_files/ that deletes or
            modifies the affected rules

  Step 4: Document
    Input:  the PR
    Output: short summary posted to the team channel
            with the deletions and redesigns

The audit decision is rule-based. Each alert is judged against the same four criteria; the criteria do not change between reviews. Consistency is what makes the audit fair.

  Per-alert decision criteria
  ===========================

  Owner label present?
    NO  -> DELETE or assign owner (no anonymous rules)

  Runbook URL present?
    NO  -> REDESIGN (every page needs a runbook)

  Action rate above 30%?
    NO  -> DELETE or REDESIGN (the alert produces no
           operational outcome)

  Threshold and for: clause reviewed in the last quarter?
    NO  -> REDESIGN (the rule has drifted)

A rule that fails any one criterion is a candidate for deletion or redesign. A rule that fails two or more is deletion-default.

How to configure it

The Alertmanager retention extension and the Loki shipping are the production-grade configuration for the audit:

# alertmanager.yml (excerpt)
# Extend retention to 30 days for audit support
--cluster.retain-period: 720h

# Grafana Alloy or promtail config: ship Alertmanager
# notifications to Loki for long-term audit storage
loki.source.file:
  filename: /var/log/alertmanager/notifications.log
  labels:
    job: alertmanager
    component: notifications
  relabel_rules:
    - source_labels: ['filename']
      regex: '.*notifications.*'
      target_label: 'log_type'

The auditor queries Loki for the 30-day notification history:

# SEVERITY: READ-ONLY
logcli query --since=30d \
  '{job="alertmanager"} |= "firing"' | jq -r .alertname

The output is every alert name that fired in the last 30 days. The auditor joins this list with the incident history to compute the action rate per alert.

A minimal Grafana dashboard for the review:

# provisioning/dashboards/alert-quality.yaml
apiVersion: 1
providers:
- name: alert-quality
  orgId: 1
  folder: 'Observability'
  type: file
  options:
    path: /etc/grafana/dashboards/alert-quality.json

The dashboard panels:

  Alert Quality Dashboard
  =======================

  Panel 1: Monthly page volume (Stat)
    Query: count(ALERTS{severity="page"})
    Time range: last 30 days

  Panel 2: Mean time to mitigate (Stat)
    Query: alertmanager_notifications_total{state="resolved"}
           - alertmanager_notifications_total{state="firing"}
    Time range: last 30 days

  Panel 3: Action rate per alert (Bar gauge)
    Query: count by (alertname) (ALERTS)
           / count by (alertname) (incidents)
    Time range: last 30 days

  Panel 4: Top 10 highest-firing alerts (Table)
    Query: topk(10, count by (alertname) (ALERTS))
    Time range: last 30 days

How to validate it

Run the audit commands. The commands are READ-ONLY against the Alertmanager state, the Git history, and the incident management system.

# SEVERITY: READ-ONLY
# 1. Inventory: alerts that fired in the last 30 days
amtool alert query --start=-30d \
  --output=json \
  | jq -r '[.[] | .labels.alertname] | unique | .[]'

# 2. Per-alert firing count
amtool alert query --start=-30d \
  --output=json \
  | jq -r '.[] | .labels.alertname' \
  | sort | uniq -c | sort -rn | head -20

# 3. Active silences
amtool silence query --output=json \
  | jq -r '.[] | {alertname: .matchers.alertname,
                  createdBy: .createdBy,
                  duration: .duration}'

# 4. Routes per alert name
for alert in $(amtool alert query --start=-30d \
  --output=json \
  | jq -r '[.[] | .labels.alertname] | unique | .[]'); do
  amtool config routes test \
    --config.file=/etc/alertmanager/alertmanager.yml \
    alertname="$alert" \
    severity=page team=$( \
      amtool alert query --alertname="$alert" \
        | jq -r '.[0].labels.team')
done

The output of step 4 is a list of every alert with the receiver it should route to. The auditor verifies that page-severity alerts route to PagerDuty and ticket-severity alerts route to Jira. Any mismatch is a routing bug; the auditor files a PR to fix the labels in the rule file.

Verify the audit PR is reviewable:

# SEVERITY: READ-ONLY
git diff --stat rules/

The output should show deletions and edits in the rule files directory. If the diff is empty, the audit produced no findings, which is the audit’s most suspicious outcome.

How it can fail

Four failure modes appear in teams that run the review:

1. The review is skipped because the team is busy. Three months of bad alerts accumulate. Symptom: monthly page volume grows 3x in one quarter; on-call rotation is burned out. The fix is to treat the review as a calendar block, not a backlog item.
2. The review proposes deletions but the deletions are blocked. A senior engineer insists the rule is needed because “we added it last year for a reason.” Symptom: the rule stays; the audit produces no PR. The fix is to require evidence (action rate, runbook execution) for the rule to stay; absence of evidence is deletion.
3. The review proposes redesigns but the redesigns break. A threshold change causes a real incident to fall below the line. Symptom: missed incident; error budget consumed. The fix is to canary new thresholds in a recording rule before replacing the production rule.
4. The audit findings are not converted to PRs. The auditor writes a Google Doc; the rules are not changed. Symptom: the same firing counts reappear next month. The fix is to make the PR the audit output, not the report.

How to troubleshoot it

When the audit produces findings the team disagrees with, the order is:

1. Inspect the firing history with amtool alert query. Show the team the firing count, the action rate, and the silence duration.
2. If the rule has high firing count and low action rate, the audit’s recommendation is supported by data. Push for the PR.
3. If the rule has high firing count and high action rate, the audit’s recommendation may be wrong. Investigate why the rule fires often and why the action rate is high; the rule may be tuning its threshold rather than being deleted.
4. If the disagreement cannot be resolved, defer the decision to the next monthly review with a comment in the rule file. The deferral is not a deletion; it is a documented postponement.

Security implications

The audit touches the rule file, which is in Git. The rule file contains expressions, labels, and annotations. Annotations can include dashboard URLs and runbook URLs that may carry internal hostnames. If the rule file is in a public repository, the hostnames leak the internal topology. Production rule files should live in a private repository, and the audit PRs should be reviewed by engineers with read access to the internal network.

Alertmanager’s API exposes the alert history. If the API is exposed to a third-party integration without authentication, the third party can read which alerts have fired and which services are degraded. Restrict the API to operators with a known role, and audit the API access log.

Performance implications

The audit query against Alertmanager is bounded by the retention of Alertmanager’s --cluster.retain-period (default 120 hours). A 30-day query is not possible against a default Alertmanager deployment; the audit must rely on a separate store. The common approach is to ship Alertmanager’s notification history to a long-term store (Loki, a database, or a SIEM) and to query that store for the audit. The cost is the storage of every notification; the benefit is the ability to compute action rate over months.

The audit PR itself is a normal Git workflow and has no performance cost beyond the reload of Prometheus and Alertmanager. The reload is a service-impact operation; it is described in the lesson on alertmanager as code.

Production guidance

• The review runs monthly, on a fixed calendar slot. The auditor does not negotiate the slot; the slot is a calendar block.
• The audit output is a PR, not a report. Findings that are not converted to code changes do not improve the system.
• The auditor requires two reviewers for any deletion of a page-severity rule. A rare-but-real alert must not be deleted by a single auditor.
• The action rate is the audit’s north-star metric. A rule with low action rate is the candidate for deletion or redesign; a rule with high action rate is the candidate for keep.
• The team sees the audit findings. A monthly Slack post with the deletions and redesigns keeps the discipline visible.

Verification

You should now be able to answer:

• What four steps make up the monthly alert quality review, and what is the output of each step?
• What four per-alert criteria does the audit decision tree check, and what is the disposition when a rule fails two or more?
• What three operational metrics does the review compute, and where do they come from (Alertmanager state, Git, incident system)?
• Why does a 30-day audit window require extending Alertmanager retention or shipping notifications to Loki, and what is the default retention?

Quiz