Break/Fix: Alertmanager Routes Wrong

Reported symptoms

At 08:15 the platform team asks, mildly, whether anyone knows why #platform-alerts took 412 notifications overnight. The channel is four days old. Most of what arrived in it concerns clusters the platform team has never been on call for.

Twenty minutes later the same morning gets worse. The DBA on-call discovers that PostgresReplicationLag fired critical on prod-db-1 at 02:40, ran for four hours, and never paged anybody. They found out because the application team asked why reads were stale.

Set against those two, the rest of the picture does not line up:

• Warning alerts for staging-eu-1 reached #oncall-dba all night, on time, exactly as they always have. Half the estate is behaving perfectly.
• The critical pages that did arrive were late - the on-call describes it as “twenty seconds or so” - and a four-hour incident produced one page rather than the hourly reminders the rotation is built around.
• The audit channel that collects severity=info has had nothing for four days. Nobody raised it, because nobody watches it.
• Nothing failed. amtool check-config passes, the last reload logged cleanly, no alerting rule has changed in three weeks, and the paging provider’s status page is green for the whole window.

Two theories are on the table before anyone opens a terminal. The first is that the paging integration is broken, because a page did not arrive. The second is the inhibit rule that shipped in the same commit as the new channel four days ago - suppression is exactly the shape of “the alert exists and nobody was told”.

Evidence provided

$ amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml \
alertname=PostgresReplicationLag severity=critical team=dba cluster=prod-db-1

slack-platform

$ amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml \
alertname=PostgresReplicationLag severity=critical team=dba cluster=staging-eu-1

pagerduty-dba

$ curl -s http://alertmanager:9093/api/v2/alerts \
| jq '.[] | select(.labels.alertname=="PostgresReplicationLag")
      | {state: .status.state, silencedBy: .status.silencedBy,
         inhibitedBy: .status.inhibitedBy, receivers: [.receivers[].name]}'

{
"state": "active",
"silencedBy": [],
"inhibitedBy": [],
"receivers": [
  "slack-platform"
]
}

$ git log --oneline -1 -- alertmanager.yml; git show --stat HEAD -- alertmanager.yml

a1f4c02 feat(alerting): onboard platform team channel
alertmanager.yml | 13 +++++++++++++
1 file changed, 13 insertions(+)

The route it adds is the first entry under the root’s routes: key:

route:
  receiver: 'default-slack'
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Added four days ago. Platform team owns prod-eu-1 and prod-us-1.
    - matchers:
        - cluster =~ "prod-.*"
      receiver: 'slack-platform'

    # Everything below here predates the change and is unmodified.
    - matchers:
        - severity = critical
      receiver: 'pagerduty-oncall'
      group_wait: 10s
      group_interval: 2m
      repeat_interval: 1h
      routes:
        - matchers:
            - team = dba
          receiver: 'pagerduty-dba'
        - matchers:
            - team = infra
          receiver: 'pagerduty-infra'
    - matchers:
        - severity = warning
        - team = dba
      receiver: 'slack-dba'
    - matchers:
        - severity = info
      receiver: 'slack-audit'

The same commit adds the receiver the route names, and an inhibit rule:

receivers:
  # ... existing receivers unchanged ...
  - name: 'slack-platform'
    slack_configs:
      - channel: '#platform-alerts'
        api_url_file: '/etc/alertmanager/secrets/slack-platform.url'

inhibit_rules:
  - source_matchers:
      - alertname = ClusterDown
    target_matchers:
      - alertname =~ "HostDown|ServiceDown"
    equal: ['cluster']

For reference, the clusters in the estate are prod-eu-1, prod-us-1, prod-db-1, prod-edge-1, staging-eu-1 and staging-us-1.

Work the evidence before reading on

Every one of those outputs is correct, and two of them disagree about what the routing does. That is the whole difficulty.

1. The routes test returns a different receiver for prod-db-1 and staging-eu-1 with otherwise identical labels. Which matcher in the tree distinguishes those two label sets, and where does it sit relative to the severity gate?
2. The platform team owns two clusters. How many cluster names in the estate does their matcher select?
3. status.state is active and inhibitedBy is empty. What does that rule out, and why is it a cheaper check than reading the inhibit rule?
4. The new route sets exactly one key. What does a route do about group_wait and repeat_interval when it does not set them, and which parent does it inherit them from now?
5. Staging is fine. Info is silent. Criticals are late. Can one edit produce all three, or do you need three faults?

Before continuing: which check in your change process, run against this file, would have printed slack-platform four days ago?

Root cause

The tree is ordered, and order beats specificity

Alertmanager walks the children of a route in the order they are declared and takes the first one whose matchers all match. continue defaults to false, so that first match is also the last: evaluation stops and the alert is delivered to that node’s receiver. There is no ranking by specificity, no longest-match rule, and no warning when a broad route shadows a narrow one. A route written above the severity gate is not “another option Alertmanager will consider” - for every alert it matches, it is the whole decision.

The platform route matches cluster =~ "prod-.*". The team that wrote it owns prod-eu-1 and prod-us-1, and the matcher is a fair description of their intent. It is not a description of the estate: prod-db-1 and prod-edge-1 match it too, because the naming convention that says “production clusters start with prod-” is older than the routing tree and nobody consulted it when writing the matcher. Every production alert of every severity for every team now terminates on the first child of the root.

That single fact accounts for four of the five symptoms. The DBA critical never reached PagerDuty because the severity gate below is unreachable for production labels. #platform-alerts took 412 notifications because it is now the destination for the entire production estate. The audit channel went quiet because production severity=info alerts stop one node above it. Staging is untouched because staging-eu-1 does not match prod-.*, so those alerts walk past the new route and reach the tree that was always there.

Timers are inherited, so moving a route changes the tempo

The fifth symptom - late pages, one reminder in four hours - comes from the same nine lines by a different mechanism. A route inherits group_by, group_wait, group_interval and repeat_interval from its parent for every key it does not set itself. The platform route sets receiver and nothing else, and its parent is the root.

Production criticals used to be evaluated on the critical branch, which deliberately overrides the tempo: group_wait: 10s so a page goes out quickly, repeat_interval: 1h so a still-firing incident nags the rotation every hour. They are now evaluated on a node that inherits the root’s group_wait: 30s and repeat_interval: 4h. Twenty extra seconds before the first notification, and one reminder in a four-hour incident instead of four. Nobody changed a timer. The alerts changed which timers apply to them.

Why inhibition was the wrong suspect, and how the evidence says so

Suppression and misrouting produce the same complaint - “the alert exists and nobody was paged” - so the inhibit rule in the same commit was a reasonable first theory. It is also the more expensive one to investigate, because reading an inhibit rule means reasoning about source matchers, target matchers and equal: scope across the whole firing set.

The cheap check comes first. An alert suppressed by inhibition reports status.state: "suppressed" and names the suppressing alert in status.inhibitedBy; an alert suppressed by a silence reports suppressed with an entry in silencedBy. This alert reports active with both lists empty, which eliminates inhibition and silences in one call, before anyone has opened alertmanager.yml. The same response carries receivers[], which is Alertmanager stating in its own words where it sent the alert - and it names a receiver the DBA team has never heard of.

Resolution

1. Tell the on-call rotations first. Production has had no paging path for four days and still does not. Until the tree is fixed, every production page depends on a human watching a Slack channel, and the affected rotations need to know that in writing rather than discovering it during the next incident.
2. Establish the real blast radius from the alert state, not from the config. curl -s http://alertmanager:9093/api/v2/alerts | jq -r ".[] | (.receivers[].name)" | sort | uniq -c shows how much of the currently firing set is landing on the new receiver, which is the number the incident note needs.
3. Decide between fix and hold before editing. If the change window is bad - a release in flight, a skeleton on-call - hold is legitimate: leave the route, name one platform engineer as the temporary router for production pages, and set an explicit end time. A hold without a named owner and an end time is just the incident continuing.
4. Move the platform route below the severity gate. Position is the fix; the matcher is the follow-up. A route that sits after the severity branch can no longer shadow it, whatever its matcher says.
5. Narrow the matcher to the clusters the team owns: cluster =~ "prod-eu-1|prod-us-1". Better, if the alerting rules can emit it, match on an owner or team label that the rules set deliberately, so the routing stops depending on a naming convention nobody owns.
6. Set the timers explicitly on the moved route rather than inheriting them from whichever parent it now sits under. Inheritance is what made the tempo change invisible; an explicit group_wait, group_interval and repeat_interval make the next move of this route a no-op for notification speed.
7. Do not use continue: true to make the missing pages come back. It restores the DBA page by also sending every production alert to the platform channel, doubles the notification volume in the next incident, and leaves the ordering defect in the tree for the next person to trip over.
8. Validate the file, then the tree, then reload. amtool check-config for syntax, the routes-test fixture in Verification for behaviour, and only then signal Alertmanager to reload.
9. Expect a burst on reload. Re-evaluating the tree re-routes the alerts that are firing right now, so incidents the DBA rotation already knows about will page again on their correct receivers. Warn the rotation before you reload rather than after.

Verification

1. Every leaf of the tree returns the receiver you intended. Run amtool config routes test once per leaf - a critical per team on a production cluster, a warning per team, an info - and read the receiver name. The DBA critical on prod-db-1 must return pagerduty-dba.
2. The change that started this still works. The platform team's own labels on prod-eu-1 must still return slack-platform. A fix that breaks the onboarding it was repairing will be reverted by whoever notices next, and then you have this incident again.
3. The tempo is back. The receiver the test names identifies the node that claimed the alert, and that node owns the timers; a DBA critical that returns pagerduty-dba is running on the critical branch's group_wait: 10s and repeat_interval: 1h rather than the root's 30s and 4h.
4. A synthetic alert lands where the test says. amtool --alertmanager.url=http://alertmanager:9093 alert add alertname=PostgresReplicationLag severity=critical team=dba cluster=prod-db-1, then confirm the entry in /api/v2/alerts shows receivers[0].name as pagerduty-dba. Resolve it afterwards rather than leaving a synthetic critical in the system.
5. The notification actually arrived in the paging integration - not in a chat channel, and not only in Alertmanager's own view of what it sent. Routing correctly and delivering successfully are two different claims and this step is the only one that tests the second.
6. The negative case holds. A critical on prod-db-1 must not appear in the platform channel, and an alert for a cluster the platform team does not own must not select slack-platform.
7. The quiet channel is no longer quiet. severity=info on a production cluster reaches the audit channel again; four days of silence there was a symptom nobody read.
8. The CI fixture fails when it should. Point it at the broken revision of the file - git show HEAD~1:alertmanager.yml - and confirm the pipeline goes red. A guard nobody has watched reject anything is a comment.

Prevention

• Test the tree, do not review it. A fixture of label sets covering every combination of severity, team and environment the alerting rules can emit, each with its expected receiver, run through amtool config routes test in CI. This is the control that would have caught the defect; check-config structurally cannot.
• Review route insertions for position. In a tree walked in declared order, where a route goes matters more than what it contains. A route added at the top of routes: should oblige the author to state which existing routes it now shadows, and the reviewer to check that list.
• Match on labels the alerting rules own. team, owner and service are set deliberately by a rule author who is thinking about alerting. Cluster naming conventions are set by whoever built the cluster, and they change without anyone consulting the routing tree.
• Set notification timers explicitly on any route that cares about them. Inheritance is a convenience for group_by; for group_wait and repeat_interval it means the tempo of a page depends on where its route currently sits in the file.
• Alert on the alerting. A receiver whose notification volume changes by an order of magnitude overnight, and a receiver that has sent nothing for several days, are both visible without reading any YAML, and both were present here for four days before a human noticed.
• Audit receivers on a schedule. Enumerate the receivers in the config and confirm each still maps to a rotation somebody watches. A channel that nobody reads is indistinguishable from a channel that works.