Skip to main content
RunBook Academy

← All runbooks in Observability

high riskservice affecting~30 min

Runbook: Investigate an Alert Not Firing

1 · Prerequisites

Confirm every item is in place before any state change.

  • Alerting rule configuration
  • Prometheus and Alertmanager

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Determine the alert
  • · Determine the SLO / condition
  • · Verify production is broken

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Check Prometheus: up{job=...} == 1
  2. 2Check the alert rule: promtool test rules rules.yml
  3. 3If rule is correct: check label matching and selector
  4. 4If rule fires: check Alertmanager routing
  5. 5Check Alertmanager: amtool silence list, amtool alert list
  6. 6Check receiver: PagerDuty / Slack / email status
  7. 7Verify the alert reaches the receiver by sending a test notification

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓The alert fires in Prometheus
  • ✓Alertmanager routes correctly
  • ✓The receiver receives the notification
  • ✓Production impact is captured in the next alert

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If rule is wrong: fix the rule
  • ↶If Alertmanager is wrong: fix the routing
  • ↶If receiver is wrong: fix the integration

6 · Escalation

When the runbook isn't enough, contact:

  • · Escalate after 30 minutes of investigation

Purpose

Investigate an Alert Not Firing

When to use this runbook

Use this runbook when the operator needs a guided procedure to handle the situation described above.

Pre-checks

Before starting the procedure, confirm the prerequisites and pre-checks are met. The structured lists are rendered from the frontmatter by the page layout.

Procedure

Follow the steps from the frontmatter procedure steps. The page layout renders the steps as a checklist with copy-to-clipboard affordances.

Verification

After the procedure, the structured verification items from the frontmatter are rendered as a checklist.

Rollback

If the procedure fails or makes things worse, follow the structured rollback steps from the frontmatter.

Escalation

The structured escalation path is rendered from the frontmatter. Use it if the operator cannot complete the procedure safely.

References

  1. Prometheus documentation
  2. Grafana documentation
  3. Loki documentation
  4. Tempo documentation