← All runbooks in Observability
Runbook: Investigate Alertmanager Delivery Failure
1 · Prerequisites
Confirm every item is in place before any state change.
- Alertmanager
- Receiver configuration
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · Confirm alerts reach Alertmanager
- · Determine which receiver is failing
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1amtool alert list
- 2amtool silence list
- 3Check the receiver configuration
- 4If PagerDuty: check PD service key
- 5If Slack: check webhook URL and channel permissions
- 6If email: check SMTP credentials and recipient
- 7amtool config show to verify the configuration
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓amtool test receiver <name>
- ✓Alerts reach the receiver
- ✓Test alert arrives at the receiver within the configured timeout
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶Disable the failing receiver
- ↶Re-enable once fixed
6 · Escalation
When the runbook isn't enough, contact:
- · Escalate to the platform team
Purpose
Investigate Alertmanager Delivery Failure
When to use this runbook
Use this runbook when the operator needs a guided procedure to handle the situation described above.
Pre-checks
Before starting the procedure, confirm the prerequisites and pre-checks are met. The structured lists are rendered from the frontmatter by the page layout.
Procedure
Follow the steps from the frontmatter procedure steps. The page layout renders the steps as a checklist with copy-to-clipboard affordances.
Verification
After the procedure, the structured verification items from the frontmatter are rendered as a checklist.
Rollback
If the procedure fails or makes things worse, follow the structured rollback steps from the frontmatter.
Escalation
The structured escalation path is rendered from the frontmatter. Use it if the operator cannot complete the procedure safely.