Alertmanager Routing Wrong

The alert fires. The on-call does not receive a page. The team opens the AM dashboard and sees the alert sitting in /api/v1/alerts with the labels the rule set. They click through to the alert detail page. The receiver field shows default. The team’s paging integration is checkout-pagerduty. The alert that should have paged the right on-call went to the default receiver and the postmortem names the route. The route did not match. The rule was right. The receiver was right. The labels the rule set and the labels the route matched against were not aligned.

This lesson is that alignment. The route matcher is a small piece of YAML that decides which receiver gets the notification. When the matcher is wrong, the alert is either silently misdirected or fall through to default.

What “AM routing wrong” is

AM routing wrong in production terms is an alert that Prometheus fires correctly into AM, that AM has in its /api/v1/alerts list, but that is delivered to the wrong receiver (including the default receiver) because the configured route matchers do not align with the labels Prometheus sent. Three conditions hold:

1. The Prometheus rule fires. The alert appears in ALERTS{alertstate="firing"}.
2. The alert reaches AM. The alert appears in AM’s /api/v1/alerts.
3. AM applies the route tree and the matchers do not select the receiver the operator intended.

Condition one excludes rule-wrong and threshold-wrong defects. Condition two excludes a Prometheus-to-AM handoff defect. Condition three isolates the cause to the route matcher itself.

A routing wrong condition is not the same as a receiver wrong condition. The receiver is the integration AM calls out to; the route is the matching stage that selects which receiver an alert goes to. A route that selects the right receiver but the receiver is broken is lesson 06. A route that selects the wrong receiver is this lesson.

Why a sysadmin cares

The cost of a routing wrong is asymmetric. A page that lands on the wrong on-call is an unwanted wake and an instance of “the on-call that did not need to be woken got woken”. The real cost is in the opposite direction: a rule that should page a specific team’s escalation channel instead lands on the default receiver (chat only, no page). The team finds out from the customer.

Three failure shapes repeat:

• Top-level route catches everything. A rule file ships without explicit child routes, or with continue: false on the top-level route, so every alert falls through to the default. Symptom: the team notices when a paging-tier alert lands in chat only.
• Route matcher typo against the rule’s labels. The rule sets severity: page; the route matches severity = "pages" (extra s). Every alert with severity: page falls through to default.
• Group-wait swallows a transient fire. A group_wait: 5m against a rule whose for: is 30s produces a group that opens but does not notify until the group interval ends; in between, the alert resolved and AM has nothing to send.

The third is less about the matcher and more about the group timing; this lesson treats it because the symptom (“rule fired but did not page”) is the same and the diagnostic overlaps.

How it works

AM applies the route tree to every alert. The tree is written top-down: each child route is checked against the alert’s labels; the first match wins unless continue: true is set. The default route catches anything that did not match.

+------------------------+
| Alert arrives at AM    |
+----------+-------------+
           |
           v
+------------------------+
| Apply silences and     |
| inhibitions            |
+----------+-------------+
           |
           v
+------------------------+
| Apply top-level route  |
| matchers (if any)      |
+----------+-------------+
           |
           v
+------------------------+
| Walk child routes in   |
| declared order         |
|   - match: yes,        |
|     continue: false -> |
|     deliver & stop     |
|   - match: yes,        |
|     continue: true  -> |
|     deliver & continue |
|   - match: no       -> |
|     continue to next   |
+----------+-------------+
           |
           v
+------------------------+
| No child matched;      |
| fall through to        |
| top-level receiver     |
| (default)              |
+----------+-------------+
           |
           v
+------------------------+
| Group by group_by,     |
| apply group_wait,      |
| deliver to receiver    |
+------------------------+

The matcher grammar in AM is the same as Prometheus label matchers, with two extensions:

• match_re: for a regex match on a single label (deprecated in 0.28, use matchers: instead).
• matchers: for an explicit list of label matchers supporting =, !=, =~, !~.

A route can match on the alert’s alertname, severity, team, service, region, or any label Prometheus or the rule attaches. The matcher must match the actual labels present in the alert; a misspelt label key returns zero matches and the alert falls through.

continue: true on a route causes AM to deliver through that route and continue walking the children. This is the mechanism for fanning out one alert to several receivers. Most teams do not need fan-out; the default (continue: false) is the right default.

The most common cause

In roughly half of the AM-routing-wrong investigations the team reviews, the cause is a route matcher that does not match the labels the rule sets. The rule’s labels: block emits severity: page; the route’s matcher is severity = "warn". The alert falls through. The cause is label drift between the rule file and the AM config.

The second most common cause is a top-level route catch that swallows everything. A new team adds a child route inside the existing structure; the parent route still has matchers that all alerts pass through. Symptom: every alert goes through the top-level receiver instead of the intended child.

The third most common cause is match_re: deprecated to matchers: with a malformed regex. The legacy syntax is rejected at config load; the new syntax is more strict. A team mid-migration sees a fraction of routes fail and the rest continue to work.

Under the hood

AM is a Go binary that runs as a single process. The route tree is parsed on SIGHUP; in-memory updates do not happen. This means a config edit must be saved and then signalled to AM to take effect. The signal path matters in container deployments where the PID 1 may not be AM.

The flow on a firing alert:

1. Prometheus POSTs the alert to AM’s /api/v2/alerts/ endpoint (or v1 in older deployments).
2. AM parses the alert, applies silences and inhibitions.
3. AM applies the route tree.
4. AM groups the alert by group_by and starts the group_wait timer.
5. After group_wait elapses, AM delivers to the receiver for the first matching route.
6. Subsequent alerts in the same group wait for group_interval before re-delivery.

The boundary the operator cares about is step 3. If the matchers in the route tree do not select the receiver, the alert lands in the default receiver or never delivers (if the default is a no-op).

How to configure it

An AM configuration that protects against the common routing defects. Annotated example:

global:
  resolve_timeout: 5m
  # SMTP and Slack globals set here, used by all receivers.

templates:
  - '/etc/alertmanager/templates/*.tmpl'

route:
  # Top-level route catches everything that does not match
  # a child. Default receiver posts to chat only.
  receiver: default
  group_by: ['alertname', 'region']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Severity "page" anywhere goes to the paging tier.
    - matchers:
        - severity = "page"
      receiver: pagerduty-default
      continue: false
    # Severity "page" but for the checkout team specifically
    # goes to the checkout pager. Continue is false; this
    # rule wins against the catch-all above because it is
    # checked first.
    - matchers:
        - severity = "page"
        - team = "checkout"
      receiver: checkout-pagerduty
      continue: false
    # Severity "page" but for infra goes to the infra channel.
    - matchers:
        - severity = "page"
        - team = "infra"
      receiver: infra-pagerduty
      continue: false

receivers:
  - name: default
    webhook_configs:
      - url: 'https://hooks.example.com/default-chat'
        send_resolved: true
  - name: pagerduty-default
    pagerduty_configs:
      - routing_key: 'PD_DEFAULT_KEY_REDACTED'
        send_resolved: true
  - name: checkout-pagerduty
    pagerduty_configs:
      - routing_key: 'PD_CHECKOUT_KEY_REDACTED'
        send_resolved: true
  - name: infra-pagerduty
    pagerduty_configs:
      - routing_key: 'PD_INFRA_KEY_REDACTED'
        send_resolved: true

inhibit_rules:
  - source_matchers:
      - severity = "page"
    target_matchers:
      - severity = "warn"
    equal: ['alertname', 'region']

The diagnostic hooks in this config:

• severity = "page" is the matcher. The rule’s labels: block must emit severity: page for the matcher to match. If the rule sets severity: warn by mistake, the matcher does not match; the alert falls through to default.
• team = "checkout" is the team-specific discriminator. Rules that do not set team: checkout fall through to the severity = "page" route above.
• continue: false (explicit) prevents fan-out. Removing this attribute achieves the same; being explicit is defensive.
• The default receiver posts to chat only. A paging-tier alert that falls through is by design not paging; the symptom is observable.

The receive definition uses a routing key per receiver. Rotation is per-receiver, not global. A team that shares the same routing key across several receivers is not isolating their on-call.

How to validate it

Three steps. Run all three.

Step 1: parse-time check.

amtool check-config /etc/alertmanager/alertmanager.yml

Output:

Checking 'alertmanager.yml' SUCCESS
Found 4 routes, 5 receivers, 1 inhibit rules, 1 templates

A parse error includes the offending key:

err: yaml: line 28: did not find expected key
FATAL: route configuration error

Step 2: routes test against a sample alert.

Save the alert to a file:

[
  {
    "labels": {
      "alertname": "CheckoutHighErrorRate",
      "severity": "page",
      "team": "checkout",
      "region": "us-east-2"
    },
    "annotations": {
      "summary": "Checkout error rate above 5% in us-east-2"
    }
  }
]

Then run:

amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --alertmanager.url=http://alertmanager:9093 \
  --source.alertfile=/tmp/sample-alert.json

Output lists the matched route and the receiver:

default -> checkout-pagerduty

If the matched route is default instead of checkout-pagerduty, the rule’s labels: block is wrong against the route’s matcher, or vice versa.

Step 3: live alert inspection.

curl -s http://alertmanager:9093/api/v1/alerts \
  | jq '.data[] | select(.labels.alertname=="CheckoutHighErrorRate") | {receiver: .receiver, status: .status.state}'

A live alert in active state with receiver: default when the team expects receiver: checkout-pagerduty is a route-matcher defect.

How it can fail

Six failure shapes, each tied to a routing defect:

1. Route matcher typo against the rule’s labels. The matcher is severity = "warn" against a rule that emits severity: page. The alert falls through to default. Symptom: the live alert in AM has severity: page but the receiver is default; amtool config routes test confirms the same.
2. Top-level route matches everything. The parent route has matchers: that the alert’s labels happen to pass, and continue: false. Children are walked but the parent has already delivered. Symptom: every alert lands on the parent’s receiver; the children are unreachable.
3. continue: true accidentally set on the parent. A parent’s continue: true causes the alert to deliver to the parent and continue walking; the children try to deliver again, doubling the notification. Symptom: the same alert produces two PagerDuty incidents for one fire.
4. match_re: with a malformed regex. The legacy matcher syntax allows match_re: on a single label. A regex that does not compile fails AM’s reload. Symptom: AM’s log reports the regex compile error on SIGHUP; the previous config continues to run.
5. Receiver name typo against the route’s receiver: field. The route references checkout-pagerduty but the receivers block defines checkout_pagerduty (underscore). AM rejects the config at load. Symptom: amtool check-config reports unknown receiver "checkout-pagerduty".
6. repeat_interval longer than group_interval. A transient fire opens a group; the group interval fires and delivers; the alert resolves; AM’s repeat_interval is 24 hours but the alert needs to re-page if it persists. Symptom: a long-running incident produces only one page.

How to troubleshoot it

Follow the six-step diagnostic.

1. Step 1, AM received the alert. Confirm the alert is in AM’s /api/v1/alerts. If it is not, the Prometheus-to-AM handoff is broken and this lesson does not apply.
2. Step 2, parse AM config. amtool check-config. A parse error fails the whole file; the previous config continues to run. A parse-time pass is necessary for anything else to work.
3. Step 3, route matchers against the alert’s labels. Use amtool config routes test with a sample alert that matches the labels Prometheus is sending. The matched route and receiver are explicit.
4. Step 4, walk the route tree top-down. Find the first route whose matcher matches and has continue: false. The first such route’s receiver is the delivered target.
5. Step 5, check the rule’s labels. If the rule emits severity: warn and the route matches severity = "page", the rule file is the defect, not the AM config. Edit the rule.
6. Step 6, confirm the receiver exists. The receivers block must define the receiver named in the route. amtool check-config catches the typo at load.

Security implications

The routing_key and the webhook URLs in AM are high-value credentials. A compromised routing_key on a PagerDuty receiver allows an attacker to inject alerts into the on-call’s queue. Treat the AM config file with the same controls as the Prometheus config: restricted file permissions, version-controlled, audited changes.

inhibit_rules can be used to suppress alerts from a specific team by another team’s louder rule. This is a useful mechanism but it is also a denial-of-service mechanism. A team that mis-configures inhibitions can silence other teams’ paging. Review inhibitions as part of the AM config review.

Performance implications

AM is single-process; the routing CPU cost is bounded by the number of alerts per second and the depth of the route tree. A deep tree with many continue: true branches causes more matcher evaluations per alert. For a tree that processes a few hundred alerts per minute, the difference is negligible; for a tree that processes thousands, it matters.

group_by directly impacts the cardinality of the notification stream. group_by: ['alertname', 'region'] produces one group per (alertname, region) combination; group_by: ['alertname', 'region', 'instance'] multiplies that by the instance count. A group-by that includes high-cardinality labels floods AM’s group machinery.

Production guidance

• Pin each receiver’s routing key per integration. Do not share a routing key across teams.
• Run amtool check-config in CI for every AM config change. Block merges that fail.
• Run amtool config routes test against a fixture that covers every label the rule files emit. Block merges that route to unexpected receivers.
• Track the receiver distribution over a quarter. A team that sees 80% of alerts landing on default has a matcher-typo class of defect.

Verification

You should now be able to answer:

• What are the three pipeline stages at which a route-matcher defect can fail?
• Which stage accounts for the largest share of routing-wrong cases?
• How do you use amtool config routes test to validate the route tree against a sample alert?
• Why is a default receiver that posts to chat-only the correct safety net?

Quiz