Alert Fatigue

Six months into a new alerting stack, the on-call engineer has muted the PagerDuty channel on the phone, set the Slack notification sound to “only mentions,” and trained themselves to open Grafana only when a colleague mentions an incident in passing. The platform still pages. The platform no longer communicates. The alerting system has become a cost centre rather than a control system, and the cost is paid in the next incident that nobody notices until a customer support ticket arrives.

This is alert fatigue. It is the predictable failure shape of any alerting stack that is not actively pruned. The remediation is not training, or discipline, or “better on-call processes.” The remediation is to fix the alerts.

What it is

Alert fatigue is the condition in which an operator stops responding to alerts because the alerts have repeatedly failed to represent real, actionable conditions. The operator has learned that pages are usually optional. The operator has stopped treating pages as urgent.

The fatigue is not a property of the operator; it is a property of the alerting system. A team of disciplined engineers will develop fatigue in any stack that pages too often for non-incidences. A team of less disciplined engineers will not develop fatigue in a stack where every page represents a real, actionable condition. The difference is the stack.

Why a sysadmin cares

The cost of fatigue is not the muted notifications; it is the incident the team misses while the notifications are muted. Three months of bad alerts have taught the team that pages are optional. The fourth month brings a real incident - a customer- facing service is degraded - and the page is delivered to the muted channel. The engineer reads email. The error budget is consumed. The SLO breach is the first signal anyone sees.

How it works

Three root causes produce fatigue. Each has an observable signal that a quarterly review can detect without reading a single rule file.

  Alert fatigue: three root causes
  ================================

  1. TOO MANY ALERTS
     - Symptom: monthly page volume above 200 per on-call
     - Diagnosis: amtool alert query for last 30 days
     - Fix: delete or downgrade rules with no action rate

  2. LOW SIGNAL
     - Symptom: false-positive rate above 30%
     - Diagnosis: action_rate / firing_rate per rule
     - Fix: redesign expression, add for:, raise threshold

  3. NO ACTION
     - Symptom: alerts routinely acknowledged but never
       linked to an incident or a runbook execution
     - Diagnosis: cross-reference alert labels with
       incident-management tickets
     - Fix: convert to dashboard panel or delete

The three causes compound. A stack with too many alerts has low signal because the signal is buried in noise. A stack with low signal trains the operator to ignore it, which makes “no action” the dominant outcome. The cycle is self-reinforcing; the only discipline that breaks it is the audit.

How to configure it

The configuration of alert fatigue is the audit, not the rule. A weekly review process inspects the firing history and decides each alert’s fate. The decision tree:

  Weekly alert review
  ===================

  For each alert that fired in the last 7 days:

  Was the alert linked to an incident or runbook execution?
    - YES -> keep, possibly tune threshold
    - NO  -> was the alert acknowledged with no action?
        - YES, < 3 times -> add to "watch" list
        - YES, 3+ times  -> delete or convert to dashboard

  Was the alert acknowledged within the repeat interval?
    - YES -> keep
    - NO  -> page is being missed; investigate channel

  Is the alert firing more than 10 times per week?
    - YES -> diagnose threshold or for: clause
    - NO  -> keep, log in review notes

  Result: a list of deletions, redesigns, and keeps.
  Filed as a PR against the rule_files directory.

The audit produces a PR. The PR deletes rules, redesigns rules, or moves rules from severity: page to severity: ticket / severity: info. The PR is reviewed by the team; the team’s collective decision is the documentation.

How to validate it

Compute the three metrics the audit cares about. The commands are READ-ONLY against the Alertmanager state and the incident system.

# SEVERITY: READ-ONLY
# Total alerts fired in the last 30 days
amtool alert query --start=-30d \
  --alertname=~'.*' \
  --output=json \
  | jq '[.[] | .labels.alertname] | group_by(.) |
        map({alertname: .[0], count: length}) |
        sort_by(.count) | reverse'

The output is a list of every alert name with its 30-day firing count. The auditor walks the list from top to bottom and asks the decision-tree questions for each rule that fired more than ten times per week.

Compute the action rate per rule:

# SEVERITY: READ-ONLY
# Cross-reference with the incident-management system
amtool alert query --start=-30d \
  --output=json \
  | jq -r '.[] | .labels.alertname' \
  | sort | uniq -c | sort -rn

This is the firing count per rule. Compare against the incident count for each rule in the incident-management system. The ratio is the action rate. A rule with firing count 50 and incident count 2 has an action rate of 4%, well below the 30% threshold the audit considers healthy.

Verify the audit PR is staged for review:

# SEVERITY: READ-ONLY
git diff --stat rules/

The output should show deletions and edits in the rule files directory. The audit is incomplete if the diff is empty for two consecutive weeks.

How it can fail

Five failure modes appear in teams that run an audit cadence:

1. The audit is skipped for a quarter. Three months of bad alerts accumulate; the page channel fills with noise; the on-call mutes. Symptom: monthly page volume grows 3x in one quarter.
2. The audit deletes good alerts. The auditor mistakes a rare-but-real alert for noise. Symptom: a real incident is missed because the alert was deleted. The fix is to require two reviewers for any deletion of a page-severity rule.
3. The audit produces a report but no PR. The findings sit in a Google Doc; the rules are not changed. Symptom: the same firing counts reappear next quarter.
4. The audit proposes redesigns but the redesigns break. A threshold change causes a real incident to fall below the line. Symptom: missed incident; error budget consumed. The fix is to canary new thresholds in a recording rule before replacing the production rule.
5. The action rate is undefined. No link between alert labels and incident-management tickets. Symptom: every alert looks like “no action” because there is no record of what action was taken. The fix is to enforce a pagerduty_incident label on every page that links to the incident ID.

How to troubleshoot it

When fatigue is suspected (muted channels, missed pages, low action rate), the order is:

1. Pull the 30-day firing count per rule with amtool alert query. Identify the top 10 highest-firing rules.
2. For each, check the action rate in the incident-management system. The rules with low action rate are the candidates for deletion.
3. Check the threshold and for: clause for each candidate. A rule that fires every minute but resolves itself in 90 seconds is a threshold problem, not a deletion candidate.
4. File the deletions and redesigns as a single PR. Require two reviewers for any page-severity rule deletion.
5. Reload Alertmanager and Prometheus after the PR merges. Verify the next week has a lower firing count.

Security implications

The audit touches the rule file, which is in Git. The rule file contains expressions, labels, and annotations. Annotations can include dashboard URLs and runbook URLs that may carry internal hostnames. If the rule file is in a public repository, the hostnames leak the internal topology. Production rule files should live in a private repository, and the audit PRs should be reviewed by engineers with read access to the internal network.

Alertmanager’s API exposes the alert history. If the API is exposed to a third-party integration without authentication, the third party can read which alerts have fired and which services are degraded. Restrict the API to operators with a known role, and audit the API access log.

Performance implications

The audit query against Alertmanager is bounded by the retention of Alertmanager’s --cluster.retain-period (default 120 hours). A 30-day query is not possible against a default Alertmanager deployment; the audit must rely on a separate store. The common approach is to ship Alertmanager’s notification history to a long-term store (Loki, a database, or a SIEM) and to query that store for the audit. The cost is the storage of every notification; the benefit is the ability to compute action rate over months.

The audit PR itself is a normal Git workflow and has no performance cost beyond the reload of Prometheus and Alertmanager. The reload is a service-impact operation; it is described in the lesson on alertmanager as code.

Production guidance

• The audit runs weekly. A quarterly cadence is too slow; bad alerts accumulate faster than they can be removed.
• The audit is a PR, not a report. Findings that are not converted to code changes do not improve the system.
• The auditor requires two reviewers for any deletion of a page-severity rule. A rare-but-real alert must not be deleted by a single auditor.
• The action rate is the audit’s north-star metric. A rule with low action rate is the candidate for deletion or redesign; a rule with high action rate is the candidate for keep.
• The team sees the audit findings. A weekly Slack post with the deletions and redesigns keeps the discipline visible.

Verification

You should now be able to answer:

• What three root causes produce alert fatigue, and what observable signal does each produce?
• What is the action rate, and why is it the audit’s north-star metric?
• What command produces the 30-day firing count per rule, and where does Alertmanager’s retention limit the audit window?
• Why must the audit produce a PR and not just a report, and why does a page-severity deletion require two reviewers?

Quiz