Break/Fix: Alertmanager Delivery Failure

Reported symptoms

The payments on-call engineer mentions, at a Thursday standup, that the rota has been unusually quiet. Nobody has been paged in about a fortnight. It has been a good sprint.

Somebody checks. In the same fortnight, Prometheus fired eleven critical alerts, four of them for incidents that were later found and fixed by other means.

What the first thirty minutes of investigation turns up does not obviously belong together:

1. No critical alert has produced a page since the 3rd. Twelve days.
2. Warning-severity notifications arrive in Slack throughout the whole period, on time and correctly formatted. Alertmanager is plainly running and plainly doing work.
3. The Alertmanager UI lists the critical alerts as active, each one showing payments-pagerduty as its receiver. Routing looks right.
4. amtool check-config passes against the configuration file, and the file in git contains the PagerDuty routing key that was rotated thirteen days ago.
5. amtool config routes test with the label set from one of the missed alerts returns payments-pagerduty.
6. An engineer posts a test event to the PagerDuty Events API by hand, using the key from the secret manager. PagerDuty accepts it and an incident appears. The credential works.
7. The PagerDuty service shows zero incidents created by this integration in twelve days.
8. alertmanager_notifications_total{integration="pagerduty"} is incrementing. Notifications are being attempted.

Evidence provided

$ curl -s http://alertmanager:9093/metrics | grep -E '^alertmanager_notifications_(total|failed_total)\{integration="pagerduty"'

alertmanager_notifications_total{integration="pagerduty"} 47
alertmanager_notifications_failed_total{integration="pagerduty",reason="client"} 47

$ curl -s 'http://prometheus:9090/api/v1/query?query=time()%20-%20alertmanager_config_last_reload_success_timestamp_seconds' | jq -r '.data.result[0].value[1]'

3548812

That is about 41 days. The routing key was rotated 13 days ago and the configuration change was merged and deployed 12 days ago.

The two commands that look like they are asking the same question:

$ amtool check-config /etc/alertmanager/alertmanager.yml

Checking '/etc/alertmanager/alertmanager.yml'  SUCCESS
Found 4 routes, 5 receivers, 2 inhibit rules, 1 templates

$ amtool --alertmanager.url=http://alertmanager:9093 config show | grep -A 3 payments-pagerduty

- name: payments-pagerduty
pagerduty_configs:
- routing_key: <secret>
  send_resolved: true

Alertmanager redacts secrets in that output, so the key itself is not comparable directly. Three things are:

• The running configuration reports 3 routes and 4 receivers; the file on disk has 4 and 5. Somebody added a receiver last month.
• The Alertmanager log contains no configuration-reload lines at all since the process started 41 days ago.
• The deployment job that ships this file logs a write step, a permissions step, and a success line. There is no signal step in it.

Work the evidence before reading on

Every check the team ran was green, and every one of them was correct. The question is what each was actually asserting.

1. amtool check-config passed and amtool config routes test returned the right receiver. Which file did each of those two commands read, and which process did each of them ask?
2. The attempt counter and the failure counter are both at 47. What does that pair tell you about where in the delivery path the failure sits — before the HTTP call, during it, or after it?
3. Slack has worked flawlessly for twelve days. What property must the Slack receiver have that the PagerDuty receiver does not, for a single cause to explain both?
4. The running configuration reports one fewer route and one fewer receiver than the file. What single fact explains that, the credential failure, and the 41-day reload timestamp together?

Before continuing: name the one thing that is true of the running Alertmanager and false of the file on disk, and say what the safest next action is — because it is not the obvious one.

Root cause

1. The process has never read the file

Alertmanager loads its configuration at startup and rereads it on SIGHUP or on a POST to its reload endpoint. There is no filesystem watch and no API that mutates the route tree in place. A configuration file that has been written but not signalled is a file the process has never seen.

The deployment job writes the file and exits. It has always written the file and exited. Every change to alertmanager.yml in the last six weeks — a new receiver, a route, and finally the rotated routing key — is correct on disk, correct in git, correct under amtool check-config, and absent from the running process.

The credential rotation did not cause this. It revealed it. The gap had been open since the process last started, and it had simply not yet contained anything that mattered.

2. Why the failure looks partial

The running configuration is not wrong. It is old.

Every integration whose credential did not change still works, because the old configuration still holds a valid one. Slack has been arriving on time for twelve days for exactly that reason, and it has been read throughout as evidence that Alertmanager is healthy — which it is. A healthy Alertmanager faithfully delivering with a revoked credential is the whole shape of this incident.

That is what makes the symptom set look like four problems. Paging is broken, chat is fine, the config is right, the credential is right. Nothing there is contradictory once you accept that the file and the process are two different objects.

3. The two commands that are not the same question

The first two were green for twelve days. They were green about a file that had not been applied. The gap between the top two rows and the bottom two is the entire incident, and it is one command wide.

4. Nothing was watching the counter that knew

alertmanager_notifications_failed_total{integration="pagerduty", reason="client"} has been incrementing on every attempt for twelve days. The reason label was already saying which class of failure it was: client is a 4xx from the far side, which is a credential or an addressing problem rather than a transient upstream one.

The same is true of the reload timestamp, which had been drifting for 41 days. Both signals were present, both were free, and neither had an alert on it.

Resolution

1. Establish which configuration the process is running, before touching anything. amtool config show against the running instance, or the status endpoint, is the only source for this. The file on disk has already been checked and is not the question.
2. Diff the running configuration against the file. This is the change that a reload will apply, and reading it is the difference between a fix and an unreviewed six-week deployment.
3. Decide, explicitly, between applying now and holding. If the delta is small and every hunk is understood, apply it. If it is not, name an engineer to watch the firing alert list directly, give the watch an end time, and schedule the reload as a reviewed change.
4. If applying: validate the file with amtool check-config one more time, then send the reload - SIGHUP to the process or a POST to the reload endpoint - and read the log for the reload result rather than assuming it.
5. Confirm the reload timestamp advances to the present. A reload that failed leaves the process running the old configuration and says so only in the log and in that gauge.
6. Prove delivery on the far side. Send a synthetic alert through the repaired route and confirm an incident appears in PagerDuty. The counters are necessary and not sufficient - Alertmanager can consider a notification delivered and still have sent it somewhere nobody reads.
7. Fix the deployment job so that it signals the process after writing the file, and so that it reads the reload timestamp afterwards and fails when that timestamp has not advanced. A deploy that cannot prove it took effect has not finished.
8. Add the two alerts that were missing: one on the rate of alertmanager_notifications_failed_total per integration, one on the reload timestamp going stale.
9. Write the incident up around the eleven alerts that did not page, and check each one against what was found by other means. Four of them were real, and the response time on those four is the actual cost of this fault.

Verification

1. The running configuration and the file on disk are now identical. Fetch the running configuration again and diff it; this is the check that failed at the start and it is the one that has to pass at the end.
2. The reload timestamp is recent, not merely non-zero. A stale gauge and a missing gauge look similar in a hurry and mean different things.
3. A synthetic alert routed through the repaired receiver produces an incident in PagerDuty. Confirm on the far side, not in the Alertmanager counters.
4. The failure counter for that integration stops advancing while the attempt counter continues. Both halves matter: an attempt counter that also stops means the route is no longer selecting the receiver at all.
5. Slack still works. A reload applies every queued change, so the integrations that were healthy before need re-checking afterwards - this is precisely the risk the diff was reviewed for.
6. The deployment job fails when it should. Point it at a configuration file that cannot load and confirm it reports failure rather than success.
7. The delivery-failure alert fires when it should. Break a receiver deliberately in a non-production Alertmanager and confirm the rule fires and reaches a human. An alert on a failure counter that has never been exercised is the same class of untested control that caused this incident.

Prevention

• Make the deployment own the reload. A job that writes configuration and does not signal the process has not deployed anything. Signal, then read the reload timestamp back, and fail the job if it has not moved.
• Know which of the two questions you are asking. amtool check-config validates a file. amtool config show reports what the process is using. Both belong in the runbook, and they are not interchangeable.
• Alert on the delivery-failure counter. One expression, per integration, with the reason label carried into the annotation so the responder knows whether they are looking at a credential, a template, or an upstream incident.
• Alert on the reload timestamp. A configuration that has not reloaded in twice its expected interval is a silent divergence between what is reviewed and what is running.
• Run a delivery canary on a schedule. A synthetic alert routed through every receiver turns a rotated credential into a routine finding instead of an incident. It costs a low-severity notification per receiver per week.
• Finish credential rotations on the far side. A rotation is not complete when the secret is updated or when the change is merged. It is complete when a notification has been observed arriving with the new credential. Put that step in the rotation runbook.
• Distrust a quiet rota. No page and no incident are indistinguishable from the inside, and the quieter interpretation is the more attractive one. A rota that has gone unusually quiet is worth one query against the delivery counters.