Good Alerts, Bad Alerts

A good alert wakes someone who can do something about it. A bad alert wakes someone who cannot. The discipline of writing alerts is the discipline of admitting that on-call sleep is a finite resource and that every page is a withdrawal from it. Withdrawals that do not produce an operational outcome are bad withdrawals.

This lesson establishes the checklist. Five properties - named, scoped, actionable, owned, reviewed - separate a useful page from decoration. We then walk through the seven anti-patterns that produce bad alerts in real Prometheus / Alertmanager deployments, and finish with a short self-test you can apply before shipping any rule to production.

What it is

A good alert is a rule that, when it fires, produces a useful operational outcome: someone acts on it, and that action resolves the underlying problem within the SLO mitigation window. A bad alert produces no useful outcome. It fires, an engineer reads it, the engineer either cannot act (no context) or need not act (no impact). The definition is operational, not aspirational. A rule whose expression is correct and whose threshold is mathematically sound is still bad if it has no owner. A rule whose annotations are rich is still bad if it pages for a condition nobody can fix.

Why a sysadmin cares

The cost of a bad alert is not the alert itself; it is the next bad alert after it, and the one after that. Three pages an hour for a week teaches the on-call engineer to mute notifications. When the real incident fires at 03:00, the engineer is reading email, not Grafana. The bad alert has reduced the signal-to-noise ratio of the entire platform. In the limit, a team that has been paged too often silently starts to treat every page as a possible false positive, and the alerting system has become a cost centre rather than a control system.

How it works

Five properties separate good alerts from bad alerts. The checklist is short enough to memorise; the discipline is to apply it before every rule ships.

       Good Alert Checklist
       =====================

  1. NAMED     - Alert name reads as a sentence.
                 "CheckoutErrorBudgetBurn", not "alert_42".

  2. SCOPED    - Labels identify the affected instance, region,
                 and service. No "CPU high somewhere" pages.

  3. ACTIONABLE - Annotations state what to do. Runbook URL.
                  Severity matches urgency.

  4. OWNED     - A team / rotation / channel is in the labels.
                 Owner label. No anonymous pages.

  5. REVIEWED  - Unit-tested. Reviewed quarterly. Has a
                 last-review date and a removal plan if it
                 fires to no action.

A rule that fails any one property is bad. A rule that fails two or more is decoration, and should be deleted, not silenced. Silencing a bad alert is not remediation; it is a confession that the rule should not have shipped.

How to configure it

A good alert in Prometheus rule syntax looks like this. The rule below is for the checkout service; it pages when the 5xx error rate exceeds 2% of request volume for five minutes.

groups:
- name: checkout.rules
  interval: 30s
  rules:
  - alert: CheckoutErrorBudgetBurn
    expr: |
      (
        sum(rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
        /
        sum(rate(http_requests_total{job="checkout"}[5m]))
      ) > 0.02
    for: 5m
    labels:
      severity: page
      team: payments
      service: checkout
      slo: availability
    annotations:
      summary: 'Checkout 5xx rate above 2% for 5 minutes'
      description: |
        Burn rate is {{ $value | humanizePercentage }}. Error
        budget will exhaust in roughly two hours at current burn.
        Affected region: {{ $labels.region }}.
      runbook_url: 'https://runbooks.example.com/checkout/5xx'
      dashboard_url: 'https://grafana.example.com/d/checkout'

Reading line by line:

• for: 5m - the condition must hold for five minutes before the alert moves from pending to firing. Single-sample noise does not page. This is the single most important field for avoiding flapping.
• severity: page - routing key. The Alertmanager tree routes severity=page to PagerDuty / OpsGenie; severity=ticket to Jira / Linear. The lesson on the page-vs-ticket decision expands this.
• team: payments - ownership label. Whoever is on the payments rotation owns the page. Alertmanager uses this label to match the team: matcher in routes:.
• service: checkout - service label. Used for dashboard links, per-service SLO burn-rate rules, and Alertmanager inhibition.
• runbook_url and dashboard_url - annotations that the Alertmanager template renders as Markdown links in the page message. The on-call engineer clicks these before doing anything else.

How to validate it

Validate before reload. promtool is the canonical pre-flight check.

# SEVERITY: READ-ONLY
promtool check rules /etc/prometheus/rules/checkout.rules.yml

Expected output:

SUCCESS: /etc/prometheus/rules/checkout.rules.yml
        1 rules found
  CheckoutErrorBudgetBurn  expr ok  for ok  labels ok

Unit-test the rule against fixture metrics:

# SEVERITY: READ-ONLY
promtool test rules test-checkout.yml

Where test-checkout.yml defines synthetic time-series for the http_requests_total{job="checkout",status=~"5.."} expression and asserts the alert fires at the expected timestamp. This catches threshold drift and label-matcher mistakes that lint cannot.

Reload Prometheus without restarting the process:

# SEVERITY: SERVICE-IMPACT (rule reload only; no scrape impact)
curl -X POST http://prometheus:9090/-/reload

Verify the alert is loaded and inert:

# SEVERITY: READ-ONLY
curl -s http://prometheus:9090/api/v1/rules \
  | jq '.data.groups[].rules[] | select(.name=="CheckoutErrorBudgetBurn")
        | {state: .state, health: .health, lastEval: .lastEvaluation}'

The output should report state inactive. If state is firing, the expression matches current production data. Check the threshold before leaving the page; a newly-deployed rule firing on first reload is the most common false positive on a Monday morning.

How it can fail

Seven anti-patterns recur across production Prometheus deployments. Each has an observable symptom that an experienced on-call can identify without reading the rule file.

1. No for: clause. Single scrape crosses threshold, alert fires, condition clears on the next scrape. Symptom: alert flaps every minute; Alertmanager deduplication never kicks in; the team is paged repeatedly for a condition that self-resolves.
2. No runbook_url annotation. On-call engineer receives page with no actionable context. Symptom: mean time to acknowledge (MTTA) above ten minutes for otherwise-known issues; the engineer has to Slack a colleague to ask what to do.
3. No owner / team label. Alertmanager cannot route by ownership, so the alert falls into a default catch-all route. Symptom: pages delivered to a generic on-call rotation that does not own the service. The right team learns about the incident from a third party.
4. Threshold-only rule with no link to user impact. Alert says “CPU above 80%” with no link to latency or error rate. Symptom: pages during known deploy windows that nobody can mitigate without reverting; pages are silenced; the alert becomes wallpaper.
5. Fireworks expression. Alert with expr: vector(1) or expr: up == 0 with no service scope. Symptom: thousands of firing alerts, all useless. Usually appears during rule development accidents and lands in production because CI did not unit-test.
6. Cardinality explosion. expr includes a high-cardinality label such as user_id or request_id. Symptom: the rule evaluator OOMs; the entire alerting pipeline stalls; every alert in the platform goes silent because Prometheus is spending its CPU budget on one rule.
7. Stale rule from a retired service. The service was retired six months ago; the rule was never removed. Symptom: alerts fire on a service that has zero replicas. The on-call acknowledges and closes with no action, every week, forever.

How to troubleshoot it

When a page fires and the alert turns out to be bad, the order is:

1. Confirm you are reading the correct alert. Open the Alertmanager UI and find the firing alert by label set; do not rely on the page message alone.
2. Inspect the description annotation. If it does not say what to do, the rule is missing a runbook. File a follow-up ticket; do not silence.
3. Check whether the rule has an owner. Inspect the team label in Alertmanager. If missing, route manually to the right team and file a follow-up to add the label.
4. Inspect the time series. Run the rule’s expr in Grafana’s Explore view. If the condition is real but the action is not, the rule is symptomatic of a missing playbook; do not silence.
5. If the rule has been silenced for more than 14 days, delete it or fix it. A long-lived silence is a deletion deferred.

Security implications

Alert rules can leak sensitive information through annotations. A description annotation that interpolates a label such as {{ $labels.user_email }} will render the email in the page message, which may be delivered to a phone, a chat, or a third-party incident tool with weaker access controls. The same applies to trace IDs and request URLs. The rule should interpolate only the labels the on-call engineer needs to act: service, region, instance, severity.

Alertmanager itself exposes an HTTP API for silences, acknowledgements, and configuration. The default configuration has no authentication. Production deployments should front Alertmanager with a reverse proxy that requires authentication, restrict the API to operators with a known role, and place the Alertmanager listener on the internal network only. The amtool CLI talks to that API; treat the API token as an on-call credential and rotate it.

Performance implications

The Prometheus rule evaluator is single-threaded per group. A group with many rules, or with rules whose expressions scan many series, can dominate the evaluation budget. Three mitigations:

1. Use recording rules to pre-aggregate expensive expressions, then alert on the recording rule output. Alert evaluation becomes a trivial comparison and the rule evaluator stays cheap.
2. Keep interval: longer than the scrape interval for non-urgent alerts. A five-minute for: on a 30-second scrape is wasteful; use interval: 1m and for: 5m. The rule fires no more often than once every five minutes either way.
3. Bound label cardinality in the expr. count by (job)(...) is cheap. sum by (service, instance, pod, container)(...) can be expensive if there are thousands of pods. If the rule fans out across all pods, consider aggregating in a recording rule first.

Production guidance

• The checklist is not aspirational. A rule that fails any one of the five properties should not ship. Add the checklist to the PR review template; require it in CI.
• promtool check rules and promtool test rules belong in CI. A rule that fails to load silently is the most expensive failure mode; both checks catch it before it reaches production.
• The owner label is non-negotiable. If you do not know who owns the rule, the rule is not ready.
• Silences have a maximum lifetime. Seven days for a deploy window; thirty days for a planned remediation; beyond that, fix or delete. The lesson on alert fatigue expands this.
• A monthly review (covered later in the module) inspects every firing alert and either certifies it as good or removes it. The discipline is the audit, not the rule.

Verification

You should now be able to answer:

• What are the five properties of a good alert, and how do you check each one in a rule file?
• What are the seven recurring anti-patterns in production alert rules, and what observable symptom does each produce?
• Why is silencing a bad alert operationally worse than deleting it, and what is the long-lived-silence rule?
• How does the for: clause change alert behaviour at the rule evaluator state-machine level?

Quiz