Alert on Every Metric

The on-call rotation has been paged eleven times this week. Three of the pages pointed to a real user-visible incident. The other eight pointed to a CPU threshold that crossed during a backup, a deploy that crossed during a deploy, and a pod that restarted and crossed during the restart. The on-call engineer has started closing pages from the notification preview without opening the runbook. The next real incident arrives at 03:00. The page is closed in the same three-second reflex. The investigation starts forty minutes late.

This is the alert on every metric anti-pattern at work. The volume is not the problem. The signal-to-noise ratio is. An on-call rotation that pages on every threshold crossing will, by construction, page on every threshold crossing, including the ones that do not require action. The reflex that protects the engineer from the noise is the same reflex that misses the real incident.

What it is

The alert on every metric anti-pattern is the practice of configuring an alert for every metric that crosses a threshold. The threshold is usually a static value chosen once and forgotten. The result is a steady-state page rate that is independent of whether anything is broken.

Compare to the alternative: alert on a symptom that requires action. A symptom that requires action is a user-visible change in service level that the operator can do something about, that the operator is uniquely positioned to fix, and that the operator would not have noticed without the page. A CPU-above-eighty threshold is not a symptom that requires action. The CPU will fall when the backup finishes; the operator cannot fix the backup by being paged; the operator did not need the page to notice the CPU spike when they next look at the dashboard.

The discipline is the service-level objective (SLO). An SLO is a target value for a service-level indicator over a window. 99.9% of requests succeed in a 28-day window is the canonical shape. An alert fires when the burn rate of the error budget exceeds a threshold. The alert fires when something is going to miss the SLO unless action is taken. The alert does not fire on every threshold crossing; it fires when the threshold crossing matters to the business.

The trade-off is honest. An SLO-based alerting posture accepts the risk that a metric will spike above a threshold without firing an alert. The mitigated risk is that the same spike, had it fired, would have been one of the eight pages the on-call ignored.

Why a sysadmin cares

Alert fatigue has three measurable symptoms. All three are observed in production at companies of every size.

Acknowledgement time rises. The first signal that a rotation is fatigued is that the median time-to-acknowledge a page creeps up from under a minute to over five. The engineer is no longer at their keyboard when the page arrives; they are asleep, in a meeting, or unwilling to break flow for a notification that has been wrong every time this week.

Pages are closed without action. The second signal is the ratio of closed without comment to closed with comment and action. A healthy rotation closes ten percent of pages without action; a fatigued rotation closes fifty.

Real incidents are missed. The third signal is the longest MTTR on record for the past quarter, paired with the largest backlog of unresolved alerts. The real incident arrived; the on-call engineer triaged it as another threshold crossing; the diagnosis started late.

How it works

The mental model is that an alert is a contract between the operator and the page. The page costs the operator their sleep, their context switch, and the implicit cost of leaving whatever they were doing. The alert must repay that cost with information the operator could not have gotten from a dashboard or a log query.

Metric or log signal
        |
        v
+-------------------+
| PromQL expression |
|  (the alert rule) |
+-------------------+
        |
        v
+-------------------+
| Alertmanager      |
|  - grouping       |
|  - inhibition     |
|  - routing        |
|  - silencing      |
+-------------------+
        |
        v
+-------------------+
| Receiver          |
|  (PagerDuty,      |
|   Slack, email)   |
+-------------------+
        |
        v
    On-call engineer
        |
        v
    Action or close

The PromQL expression decides whether the alert fires. The Alertmanager decides whether the alert is grouped, suppressed, routed, or silenced before the receiver is contacted. The receiver decides how loud the page is. The on-call engineer’s response decides whether the next alert gets the same respect.

The cheapest leverage is at the PromQL expression. A threshold expression is a one-line expr block. The expression defines what user-visible change triggers the alert. The expression should be tied to the SLI; the SLI should be tied to the SLO; the SLO should be tied to the budget. The alert fires when the budget is being burned faster than the rate the SLO tolerates.

How to configure it

The configuration is two parts. The alert rule defines what fires. The Alertmanager configuration defines what reaches the operator.

# /etc/prometheus/rules/availability.yml
groups:
  - name: availability.slo
    interval: 30s
    rules:
      # Burn-rate alert. The expression is the rate at which
      # the SLO error budget is being consumed. The threshold
      # is the rate at which the budget will be exhausted before
      # the window ends.
      - alert: HighErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{job="checkout"}[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: critical
          slo: checkout-availability
          runbook: https://runbooks.example.com/checkout-availability
        annotations:
          summary: 'Checkout error budget burning at 14.4x'
          description: |
            The checkout service is failing 1.4% of requests
            over the last five minutes. At this rate the
            28-day error budget will be exhausted in 43 hours.

The matching Alertmanager configuration routes by severity and suppresses the noisy threshold alerts:

# /etc/alertmanager/alertmanager.yml
route:
  receiver: default-slack
  group_by: [alertname, slo]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Critical pages go to PagerDuty. Only critical pages.
    - matchers:
        - severity = "critical"
      receiver: pagerduty-critical
      group_wait: 10s
      repeat_interval: 1h

    # Warnings go to Slack only. They inform; they do not page.
    - matchers:
        - severity = "warning"
      receiver: slack-warnings
      repeat_interval: 12h

inhibit_rules:
  # When a critical alert is firing, suppress the warnings
  # for the same service. The warnings are noise during an
  # incident; the operator is already looking.
  - source_matchers:
      - severity = "critical"
    target_matchers:
      - severity = "warning"

receivers:
  - name: pagerduty-critical
    pagerduty_configs:
      - service_key: '<pagerduty-service-key>'
        severity: critical
  - name: slack-warnings
    slack_configs:
      - api_url: '<slack-webhook>'
        channel: '#ops-warnings'
        send_resolved: true
  - name: default-slack
    slack_configs:
      - api_url: '<slack-webhook>'
        channel: '#ops-info'

silences:
  # Silences are managed via the API, not the file. The file
  # entry is a placeholder for the documented silence workflow.
  - matchers:
      - alertname = "PlannedDeploy"
    # duration set at creation time

The two configurations share a discipline: every alert has a severity, a runbook, and a receiver that matches the severity. A page that requires no action is misrouted to the critical receiver; a warning that requires no page is misrouted to the warning receiver.

How to validate it

Four commands confirm the alert posture is healthy.

# 1. Pages per on-call shift. The first signal of fatigue.
# Severity: READ-ONLY
amtool alert query --receiver=pagerduty-critical \
  --start=$(date -d '7 days ago' --iso-8601=seconds) \
  --end=$(date --iso-8601=seconds) \
  | jq -r '.[] | .startsAt' | wc -l

A number above ten per week is the signal that the rotation is fatigued.

# 2. Time-to-acknowledge distribution. The second signal.
# Severity: READ-ONLY
amtool alert query --receiver=pagerduty-critical \
  --start=$(date -d '7 days ago' --iso-8601=seconds) \
  | jq -r '.[] | "\(.startsAt) \(.acknowledgedAt // "unack")"'

The median column should be below sixty seconds for critical pages. A median above five minutes is the signal that the rotation is treating the pages as notifications.

# 3. Alert rules with no runbook label. The hygiene audit.
# Severity: READ-ONLY
find /etc/prometheus/rules -name '*.yml' \
  -exec grep -L 'runbook:' {} \;

A rule file without a runbook label is a release blocker until the runbook exists.

# 4. Inhibitions firing. Confirms the suppression is in place.
# Severity: READ-ONLY
amtool alert query --inhibited \
  --start=$(date -d '24 hours ago' --iso-8601=seconds) \
  | jq -r '.[] | .labels.alertname' | sort | uniq -c

A non-zero count confirms that warnings are being suppressed during critical incidents.

How it can fail

Five shapes recur when the alert posture is not disciplined.

1. The threshold-without-action alert. “CPU above 80% for 5 minutes” pages the operator. The operator cannot fix the CPU. The alert fires during every backup, every deploy, every batch job. The rotation treats it as noise. The rotation ignores all CPU alerts, including the real one.
2. The missing-severity alert. Every alert is severity: critical. The Alertmanager routes every alert to the critical receiver. Every alert is a page. The operator cannot distinguish fire now from fire next week.
3. The missing-runbook alert. The alert fires; the operator opens the page; the runbook is empty or missing. The operator invents the response. The next operator invents a different response. The MTTR variance is unbounded.
4. The flapping alert. The alert fires, resolves, fires, resolves. The Alertmanager’s repeat_interval is too short. The operator is paged five times for the same condition. The rotation silences the alert permanently. The next real occurrence is silenced too.
5. The duplication alert. Two teams have written two alert rules for the same symptom. The operator is paged twice for one incident. The Alertmanager’s group_by does not include the symptom label. The rotation treats one of the two as the “real” alert and silences the other. The silenced one is eventually the real one.

How to troubleshoot it

1. Quantify the fatigue: pages per shift, ack-time
   distribution, close-without-action ratio
        |
        v
2. Triage the alert list: which alerts have required action
   in the past 90 days?
        |
        +-- none -> silence, delete, or convert to a
        |           dashboard panel
        |
        +-- some -> keep, ensure severity and runbook are set
        |
        v
3. Add inhibitions between critical and warning alerts on
   the same service
        |
        v
4. Set repeat_interval to 4h for critical, 12h for warning
        |
        v
5. Document the alert budget per shift; alert on the budget
   itself
        |
        v
6. Re-measure in two weeks; expect the page rate to drop

Security implications

The Alertmanager API is the highest-privilege surface in the observability stack. The silences endpoint can suppress any alert; the recording rules endpoint can rewrite history. The API must be on an authenticated, audit-logged channel. The receiver credentials (PagerDuty service key, Slack webhook URL) are secrets in their own right and must be sourced from a secret manager, not committed to the configuration file.

Performance implications

The Alertmanager holds the alert state in memory. A misconfigured rule that fires on a label set with millions of distinct values produces millions of alerts, each holding a memory entry until the resolution. The rule group interval should be tuned so that the rule evaluates less often than the alert’s for window; otherwise the rule evaluates more often than the operator can respond. The PromQL expression’s cost is paid once per evaluation; a high-cardinality expression multiplies the cost by the cardinality.

Verification

You should now be able to answer:

• What is the difference between an alert and a notification, and why does the distinction matter operationally?
• What are the three measurable symptoms of alert fatigue?
• What does a burn-rate alert expression evaluate, and why is it tied to the SLO rather than the SLI?
• What is the role of Alertmanager inhibitions in the alert posture?

Quiz