high riskservice affecting~30 min

Runbook: Investigate Alertmanager Delivery Failure

1 · Prerequisites

Confirm every item is in place before any state change.

Alertmanager Delivery
Alertmanager Anatomy
The Routing Tree
Receivers and Templates
Silences and Mutes
Read access to the Alertmanager HTTP API, its /metrics endpoint and its service log
A way to reach the receiver's own record of what it did or did not accept — the PagerDuty incident list, the Slack channel, the mail relay log
Knowledge of who is currently on call, because every test you run in this runbook may reach them

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the alert actually reached Alertmanager: amtool alert query --alertmanager.url=http://alertmanager:9093. If it is not there, this is the wrong runbook — the failure is upstream, in the rule or in the Prometheus-to-Alertmanager hop
· Confirm it was not suppressed on purpose: amtool silence query, plus a look at the inhibition rules. A silenced or inhibited alert was withheld by policy, and that is a governance question rather than a delivery fault
· Establish the blast radius before touching anything: is delivery failing for one integration or for all of them? One is a credential or an endpoint; all is Alertmanager itself, its config reload, or its egress path
· Read the failure counter by integration and reason: curl -sf http://alertmanager:9093/metrics | grep alertmanager_notifications_failed_total. The reason label is the triage — config was never sent, client was rejected, server will be retried
· Check whether the running configuration is the configuration you think it is: alertmanager_config_last_reload_success_timestamp_seconds should be recent relative to the last change. A stale timestamp means the fix somebody already applied never took effect
· Establish who is on call and warn them that test notifications may arrive, before you send any. An unannounced synthetic page is indistinguishable from a real one and gets worked as such
· Name a human to watch the condition directly for the duration of the outage, and an end time for that arrangement. While delivery is broken there is no automated notification for anything routed through the failing receiver

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Confirm Alertmanager itself is healthy before blaming a receiver: curl -sf http://alertmanager:9093/api/v2/status | jq ".cluster.status, .cluster.peers". A settling cluster or a missing peer changes the whole diagnosis
2Read the attempt and failure counters side by side: alertmanager_notifications_total and alertmanager_notifications_failed_total, both broken down by integration. Failures with no attempts is a routing problem, not a delivery problem
3Triage on the reason label. config means the notification was never sent — a missing URL, a missing credential, a template that will not parse. client means a 4xx from the upstream: revoked token, wrong routing key, channel gone. server means a 5xx and will be retried
4Confirm the alert is being routed to the receiver you think it is: amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml with the alert's real labels. An alert delivered to the default receiver is delivered, just not to a pager
5Validate the configuration file on disk: amtool check-config /etc/alertmanager/alertmanager.yml. A parse error means the previous configuration is still running, which is why a change you made an hour ago may be invisible
6Read the log for the failing integration: journalctl -u alertmanager --since "10 min ago" | grep -iE "error|warn|failed". It names the receiver, the integration and the upstream status code
7Check the upstream before rotating anything: the integration's public status page, and whether the outbound path from the Alertmanager host still exists. A firewall change is a much commoner cause than a revoked token
8Apply the smallest fix that matches the reason: correct the URL or credential for config and client; for server, confirm the retry is working and wait rather than changing configuration under a transient upstream failure
9Reload rather than restart: curl -sf -X POST http://alertmanager:9093/-/reload, then confirm alertmanager_config_last_reload_success_timestamp_seconds advanced. A restart drops the in-memory notification state and can re-notify for everything currently firing
10Prove delivery end to end with a synthetic alert, after telling on-call: POST /api/v2/alerts with a distinctive alertname, then watch alertmanager_notifications_total for that integration increment while the failure counter stays flat
11Confirm with a human at the far end that the notification arrived. The counter proves Alertmanager was satisfied with the response; only a person proves the page reached a person
12Close the gap that let this run silently: an alert on rate(alertmanager_notifications_failed_total[5m]) > 0, and a scheduled delivery canary through every receiver

4 · Verification

Confirm the procedure actually fixed the problem.

✓rate(alertmanager_notifications_failed_total[5m]) is zero for the affected integration across a full evaluation window, not merely at the instant you looked
✓alertmanager_notifications_total for that integration is incrementing — a flat attempt counter means nothing is being routed there at all
✓alertmanager_config_last_reload_success_timestamp_seconds is later than the time you edited the configuration
✓amtool config routes test with the real alert labels resolves to the intended receiver
✓curl -sf http://alertmanager:9093/api/v2/status reports cluster status ready with the full configured peer count
✓A synthetic alert sent through the repaired path was received by a human, who confirmed it, and the synthetic was then resolved or expired
✓A standing alert now exists on the delivery-failure counter, and it has been shown to fire — an untested meta-alert is the same defect one layer up

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Revert configuration changes through the repository, then reload: git revert --no-edit HEAD, amtool check-config /etc/alertmanager/alertmanager.yml, curl -sf -X POST http://alertmanager:9093/-/reload
↶Do not simply disable a failing receiver. Removing it makes the alerts routed to it disappear silently, which is a worse version of the failure you are investigating. If the receiver must come out, add a route sending its alerts somewhere a human is watching, in the same change
↶If you added a temporary route or a fallback receiver, record it with an owner and a removal date. Temporary alert routing survives for years unless somebody wrote down when it should die
↶Expire any silence created during the investigation, by ID: amtool silence expire ID. A silence added to stop test noise and forgotten is precisely how the next undelivered page happens
↶Resolve or expire every synthetic alert you injected, and tell on-call the test is over
↶Prefer a reload to a restart when undoing a change. Restarting Alertmanager discards in-memory notification state and can re-notify for everything currently firing, turning a rollback into an alert storm

6 · Escalation

When the runbook isn't enough, contact:

· Escalate immediately if delivery is failing for every integration. That is not a receiver problem: it is Alertmanager, its configuration reload, or the host's egress, and until it is fixed nothing in the estate can page anyone
· Escalate to the integration owner — the PagerDuty, Slack or mail administrator — when reason="client" persists after a credential is confirmed correct. A revoked token or a deleted channel is their change to explain
· Escalate to the network or platform team when the failure is a connection error rather than an HTTP status. That is an egress path question, and it usually arrived with somebody else's firewall change
· Escalate to the platform team if alertmanager_cluster_health_score is below 1.0 or a peer is missing: a partitioned cluster delivers inconsistently, and diagnosing one receiver inside it will mislead you
· Escalate the moment the incident that triggered this investigation is itself customer-visible. Broken detection and a live outage are two problems, and one person handling them in sequence handles the second one late

Every other failure in the alerting chain leaves a mark somewhere a human looks. This one does not. The alert fires, the Alertmanager UI shows it firing and then resolved, the dashboards are green, and the only record of the failure is a counter that increments on a metrics endpoint nobody has put on a dashboard.

That is the defining property of a delivery failure, and it is why the last step of this runbook is not “fix the receiver”. It is “make the next one loud”.

When this runbook applies

The alert reached Alertmanager and did not reach a person. Confirm that first, because it is the boundary between this runbook and the one next to it:

Read-only / Safeis the alert here at all, and was it suppressed

AM=http://alertmanager:9093

amtool alert query --alertmanager.url="$AM"
amtool silence query --alertmanager.url="$AM"

Not in alert query — the failure is upstream. Use the alert-not-firing runbook: the rule, the expression, the dwell, or the Prometheus-to-Alertmanager hop.
Present, and a silence matches it — it was withheld on purpose. The question becomes who created that silence and why it outlived its reason, which is a governance conversation, not a delivery fault.
Present, no silence, no notification — you are in the right place.

The first split: one integration or all of them?

Answer this before anything else. It changes the severity, the owner and the entire shape of the investigation.

Read-only / Safeattempts and failures, per integration

AM=http://alertmanager:9093

curl -sf "$AM/metrics" \
| grep -E '^alertmanager_notifications_(total|failed_total)' \
| sort

What you see	Meaning	Owner
One integration failing, others clean	A credential, an endpoint or an upstream	The integration’s owner
Every integration failing	Alertmanager, its config, or the host’s egress	Platform, and treat as estate-wide
Failures with no attempts	Nothing is being routed there	A routing problem, not a delivery one
Attempts climbing, failures flat, still no page	Delivered — to the wrong receiver	The route tree

That last row is the one people miss. A notification that Alertmanager successfully delivered to a channel nobody watches looks, from the on-call’s side, exactly like no notification at all.

Triage on the `reason` label

Alertmanager splits notification failures into three buckets, and they have three different owners and three different fixes.

`reason`	What happened	Typical cause	What to do
`config`	Never sent	Missing URL, missing credential, template that will not parse	Fix the receiver block; reload; confirm the reload timestamp advanced
`client`	Sent, rejected with a 4xx	Revoked token, wrong routing key, deleted channel, rate limit	Confirm with the integration’s owner before rotating anything
`server`	Sent, rejected with a 5xx	The upstream is having its own incident	Confirm retries are happening, then wait

The config bucket is the silent one. A webhook URL that was rotated away produces config failures on every attempt, forever, with no outbound request ever made and nothing in any upstream’s logs to find.

The server bucket is where people do damage. It is transient by definition and Alertmanager retries it. Changing configuration to “fix” a 5xx usually means you have made an untested change during somebody else’s outage, and you will find out which of the two broke things when the upstream recovers.

Confirm the configuration that is actually running

Read-only / Safethe file on disk, and whether it is the file in memory

AM=http://alertmanager:9093

amtool check-config /etc/alertmanager/alertmanager.yml

curl -sf "$AM/metrics" \
| grep alertmanager_config_last_reload_success_timestamp_seconds

amtool check-config reports the route and receiver counts on success, which is itself a useful sanity check: a config that suddenly has fewer receivers than you expect has lost one to a merge.

The reload timestamp is the question people forget to ask. A parse failure at reload leaves the previous configuration running. If somebody already “fixed” this an hour ago and nothing changed, a stale timestamp is the whole explanation — the fix is on disk and has never been loaded.

Confirm the route, not just the receiver

Read-only / Safereproduce the routing decision with the real labels

amtool config routes test \
--config.file=/etc/alertmanager/alertmanager.yml \
alertname=CheckoutHighErrorRate \
severity=page \
team=checkout

Use the labels from amtool alert query, not the labels the rule file claims to set. External labels are added on the way out and relabelling can rewrite them, so the two can legitimately differ.

A matcher that is one character wrong — severity = "pages" against a rule emitting severity: page — sends every page-worthy alert to the default receiver. The delivery counters look perfect. Nobody is paged.

Read the log for the failing integration

Read-only / Safethe upstream's own words

journalctl -u alertmanager --since "10 min ago" \
| grep -iE 'error|warn|failed|notify'

The log names the receiver, the integration and the status the upstream returned. Before you rotate a credential on the strength of a 401, check two cheaper explanations: the integration’s public status page, and whether the outbound path from this host still exists. A firewall or egress-policy change is a commoner cause of “the token stopped working” than an actually rotated token, and rotating a working credential turns one outage into two.

Testing delivery costs somebody’s attention

Service impact possibledrive one alert through the repaired path

AM=http://alertmanager:9093

curl -sf -X POST "$AM/api/v2/alerts" \
-H 'Content-Type: application/json' \
-d '[{"labels":{"alertname":"DeliveryCanaryManual","severity":"info"},
      "annotations":{"summary":"manual delivery test - ignore"}}]'

curl -sf "$AM/metrics" | grep alertmanager_notifications_total

Watch the attempt counter for the integration increment while the failure counter stays flat. Then go and ask the human at the far end whether anything arrived. The counter proves Alertmanager was satisfied with the response it got; only a person proves the page reached a person, and those are not the same claim.

Reload, do not restart

Configuration changeapply the fix

AM=http://alertmanager:9093

amtool check-config /etc/alertmanager/alertmanager.yml
curl -sf -X POST "$AM/-/reload"
curl -sf "$AM/metrics" | grep alertmanager_config_last_reload_success_timestamp_seconds

Restarting Alertmanager discards its in-memory notification state. Everything currently firing becomes eligible for notification again, so a restart during an incident can turn a delivery fix into an alert storm at exactly the moment the on-call has the least capacity for one. Reload unless you have a reason not to, and validate before you reload so the reload cannot fail.

Holding, and why “disable the receiver” is not it

The obvious lever when a receiver keeps failing is to switch it off. It is the wrong lever. Removing a receiver does not stop the alerts routed to it; it makes them vanish quietly, which is a strictly worse version of the problem you are investigating.

If a receiver genuinely has to come out, the same change must add a route sending its alerts somewhere a human is watching. And whatever you put in place temporarily needs three things written down: what it is, who owns it, and the date it is removed. Temporary alert routing outlives the incident that created it by years unless somebody wrote down when it should die.

The honest hold, while the fix is pending, is a named person watching a named signal until a named time. Not “we’ll keep an eye on it”.

“Nothing arrived” sometimes means “not yet”

Before concluding a notification was lost, check that it was due. Four timers sit between a firing alert and a message, and three of them are Alertmanager’s:

group_wait       first notification for a new group   (default 30s)
group_interval   further notifications for that group (default 5m)
repeat_interval  re-notification while still firing   (commonly 4h)

The one that misleads is repeat_interval. An alert that fired, notified correctly, and is still firing will not notify again until the repeat interval elapses. An on-call who joins mid-incident, sees the alert firing in the UI, and receives nothing has not found a delivery failure — they have found a group that was already notified.

Grouping produces the same illusion from the other direction. With group_by set on alertname alone, twenty alerts differing only by instance collapse into one notification. Nineteen alerts genuinely produced no message, and that is the configuration working as designed. Check group_by and the timers against the notification you expected before you go looking for a fault.

Common failure shapes

Symptom	Cause	Evidence
One integration, `reason="config"`	URL or credential removed from the receiver block	Failures with no outbound request in any upstream log
One integration, `reason="client"`	Token revoked, routing key wrong, channel deleted	4xx in the Alertmanager log
One integration, `reason="server"`	The upstream is having an incident	5xx in the log; upstream status page agrees
All integrations failing	Config never reloaded, or egress blocked	Stale reload timestamp; connection errors rather than statuses
Counters clean, nobody paged	Route matched the default receiver	`amtool config routes test` with the real labels
Delivery intermittent, no pattern	Cluster peer unreachable	`alertmanager_cluster_health_score` below 1.0
Alerts silently dropped in a storm	Dispatcher hit an aggregation group limit	`alertmanager_dispatcher_aggregation_group_limit_reached_total`

Close the gap that let it be silent

Fixing the receiver restores today’s page. It does nothing about the next credential rotation. Two controls do:

# any integration failing delivery
rate(alertmanager_notifications_failed_total[5m]) > 0

# a reload that did not take
time() - alertmanager_config_last_reload_success_timestamp_seconds > 600

# a peer has gone
alertmanager_cluster_health_score < 0.5

The first is one line of PromQL and it catches every delivery-failure shape there is. Route it somewhere that does not depend on the thing it is watching — a delivery alert that can only reach you through the receiver that is broken has told you nothing.

The second control is a scheduled canary: a low-severity alert that fires on a timer and is routed through every receiver. It converts “we hope delivery works” into a signal that appears on a schedule, which is what catches a rotated credential in the week it is rotated rather than during the next incident.

Escalation

Every integration failing: platform team, immediately, estate-wide.
reason="client" after the credential is confirmed correct: the integration’s owner. It was their change.
Connection errors rather than HTTP statuses: network or platform. Egress path.
Cluster health below 1.0 or a peer missing: platform, before you spend any more time on one receiver inside a partitioned cluster.
The condition that triggered this is customer-visible: escalate now and split the work. Broken detection and a live outage are two problems.