The Routing Tree

A DiskFillingSoon alert fires on a database host. It has labels severity=warning, team=dba, cluster=prod-eu-1. A second alert, HostDown, fires on the same host with labels severity=critical, team=infra, cluster=prod-eu-1. Two alerts, one host, two completely different on-call rotations.

The routing tree is what puts them on the right channels. Get the tree wrong and the DBA team is paged for a switch reboot; the infra team is paged for a runaway query. Get it right and the two alerts go where the runbook lives.

What it is

The routing tree is the structure inside route: of alertmanager.yml. It is a tree of route blocks, each with a matcher expression and zero or more child routes. When an alert arrives, Alertmanager walks the tree top-down, depth-first: the first child route whose matchers all match wins, and the alert inherits that route’s receiver. The root is the catch-all — it matches everything by definition.

The same configuration can be written as a flat list of route: entries (each a peer of the root) or as a deeply nested tree. The flat form is easier to read for a small fleet. The nested form is the only way to express inheritance with override: a child route can change only the keys it cares about (say, receiver) and inherit the rest (say, group_by) from its parent.

Why a sysadmin cares

Three shapes appear repeatedly when the routing tree is treated as a list of rules rather than a tree:

1. The catch-all everything-is-critical. Every alert reaches pagerduty-oncall. The on-call engineer is paged for an expiring certificate in staging. The PagerDuty SLO is “investigate every page in 5 minutes.” The team ignores pages. Real incidents get lost.
2. The by-team tree with no severity floor. A severity=info disk-usage alert routes to the DBA team’s Slack. The DBA team silences it permanently because it is noise. A real disk-full emergency finds a silenced team and an empty inbox.
3. The regex that nearly matched. match_re: team=infra-.* matches infra-prod, infra-staging, and infra-experimental-archive-do-not-touch. The last one was decommissioned two years ago. Its alerts route to nowhere because the receiver name was renamed. The catch-all still picks them up, but the message goes to the wrong channel.

The routing tree is the single configuration decision that shapes who is on-call for what. Treat it as a load-bearing part of the platform.

How it works

A realistic tree, annotated:

route:
  receiver: 'default-slack'
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Severity gate: only criticals page PagerDuty.
    - matchers:
        - severity = critical
      receiver: 'pagerduty-oncall'
      group_wait: 10s
      group_interval: 2m
      repeat_interval: 1h
      routes:
        # Critical + payments routes to a dedicated rotation.
        - matchers:
            - severity = critical
            - team = payments
          receiver: 'pagerduty-payments'
        # Critical + database routes to the DBA team.
        - matchers:
            - severity = critical
            - team = dba
          receiver: 'pagerduty-dba'
        # Critical + infra (network, host, kubernetes) routes broadly.
        - matchers:
            - severity = critical
            - team = infra
          receiver: 'pagerduty-infra'

    # Warnings go to team-specific Slack channels.
    - matchers:
        - severity = warning
        - team = payments
      receiver: 'slack-payments'
    - matchers:
        - severity = warning
        - team = dba
      receiver: 'slack-dba'

    # Info goes to the audit channel.
    - matchers:
        - severity = info
      receiver: 'slack-audit'

# The default-slack receiver is the catch-all.

receivers:
  - name: 'default-slack'
    slack_configs:
      - channel: '#oncall-default'
        api_url_file: '/etc/alertmanager/secrets/slack-default.url'

  - name: 'pagerduty-oncall'
    pagerduty_configs:
      - routing_key_file: '/etc/alertmanager/secrets/pd-oncall.key'
        send_resolved: true

  - name: 'pagerduty-payments'
    pagerduty_configs:
      - routing_key_file: '/etc/alertmanager/secrets/pd-payments.key'

  - name: 'pagerduty-dba'
    pagerduty_configs:
      - routing_key_file: '/etc/alertmanager/secrets/pd-dba.key'

  - name: 'pagerduty-infra'
    pagerduty_configs:
      - routing_key_file: '/etc/alertmanager/secrets/pd-infra.key'

  - name: 'slack-payments'
    slack_configs:
      - channel: '#oncall-payments'
        api_url_file: '/etc/alertmanager/secrets/slack-payments.url'

  - name: 'slack-dba'
    slack_configs:
      - channel: '#oncall-dba'
        api_url_file: '/etc/alertmanager/secrets/slack-dba.url'

  - name: 'slack-audit'
    slack_configs:
      - channel: '#alerts-audit'
        api_url_file: '/etc/alertmanager/secrets/slack-audit.url'

Two principles to read off the tree:

• Severity before team. The first split is severity = critical vs everything else. Inside the critical branch, the second split is by team. This means a critical alert always pages somebody, and the right team always gets it.
• Timer inheritance. The critical branch overrides group_wait, group_interval, and repeat_interval because a critical page should fire fast and repeat often. The group_by is inherited from the parent — critical pages still collapse by alertname, cluster.

Under the hood

When an alert arrives, AM walks the tree. The matchers in each route: block are AND-combined. A child route’s matchers are AND-combined with its parent’s matchers, not OR-combined. The key insight: a child route adds matchers, it does not replace them. A child route at depth 2 with team = dba underneath a parent with severity = critical only matches severity=critical AND team=dba.

The tree is evaluated in memory once per alert. The cost is linear in the number of routes and the size of the matchers; for realistic trees (under 100 routes) this is sub-millisecond per alert. The bottleneck is rarely the route evaluation; it is the receiver send.

Routes can also use match_re: (a regular expression on a label) and equal: (the matched label must have the same value on both the alert and some external object — useful for grouping behaviour but not for routing itself). For most production routes, exact-match matchers: is sufficient and easier to audit.

How to configure it

The configuration is the route: block. Three points to verify before saving:

1. The catch-all receiver exists. The root route: names a receiver. That receiver must be defined in receivers: or AM will refuse to load the file.
2. Every nested receiver: is defined. amtool check-config catches this; amtool config show shows you what AM actually parsed.
3. Matchers do not over-constrain. A child route under severity = critical should not repeat severity = critical — it is already inherited. Repeating it is harmless but misleading when reading the file later.

The exact-match matcher syntax is the canonical form:

- matchers:
    - alertname = DiskFillingSoon
    - team = dba
    - severity =~ "warning|critical"

=~ is a regex match. != and !~ are the negated forms. Quote any value that contains punctuation.

How to validate it

The workhorse is amtool config routes test. It accepts a label set as key=value pairs and tells you which receiver the alert would route to, what group it would join, and which timers apply.

# Severity: READ-ONLY. Does not contact the running AM.
amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  alertname=DiskFillingSoon severity=warning team=dba cluster=prod-eu-1
# Output:
# Selected receiver: slack-dba
# Grouping:
#   alertname: DiskFillingSoon
#   cluster: prod-eu-1
#   team: dba
# Group wait: 30s
# Group interval: 5m
# Repeat interval: 4h

Compare with a critical variant of the same alert:

amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  alertname=DiskFillingSoon severity=critical team=dba cluster=prod-eu-1
# Output:
# Selected receiver: pagerduty-dba
# Grouping:
#   alertname: DiskFillingSoon
#   cluster: prod-eu-1
#   team: dba
# Group wait: 10s
# Group interval: 2m
# Repeat interval: 1h

The same alert changes receiver, group timing, and downstream channel because of the severity label. That is the routing tree working as intended.

For programmatic checks, the v2 API lists the alerts AM is currently processing and their assigned receiver:

# Severity: READ-ONLY.
curl -s 'http://localhost:9093/api/v2/alerts?receiver=pagerduty-dba' \
  | jq '.[].labels.alertname'

How it can fail

The recurring mistakes, in descending order of operational cost:

1. A matcher that matches nothing in practice but the catch-all saves it. The route team = payments-new was added during a re-org. The team name in the alert rules was never updated. Every “payments-new” alert falls through to default-slack, which has nobody watching it. Detect by comparing amtool config routes test output against the alert rule labels.
2. A regex matcher without anchors. match_re: cluster=prod matches prod, prod-eu-1, prod-staging, and the experimental prod-clone-do-not-use. Anchor it: match_re: cluster=^prod(-eu-1)?$.
3. The repeat_interval is longer than the runbook SLA. A critical alert set to repeat_interval: 24h will not remind the on-call again until tomorrow. Critical alerts should repeat within the SLA window of the responding team (usually 1h).
4. Inheritance surprise. A child route omits group_by. The parent sets it. The child inherits it. The on-call channel receives groups of 40 alerts because the parent group_by was inherited by a deep branch. Override group_by explicitly at each branch where the granularity differs.
5. The root receiver was renamed. A find-and-replace renamed default-slack to slack-default. The new name does not match the old PagerDuty integration. Every catch-all alert returns a 404 from PagerDuty. Validate after every rename.
6. No default route. A typo in route: receiver: causes the whole file to fail to load. Worse, a hand-edited file may load but parse the catch-all as null, in which case AM drops unmatched alerts silently. Always end with a deliberately named default receiver.

How to troubleshoot it

When an alert reaches the wrong channel, the order matters:

1. amtool config routes test with the exact label set. Use the labels from GET /api/v2/alerts, not from the rule file — external labels and group_* labels may have been added.
2. Walk the tree by hand. Find the first matching node. Note which matchers matched and which were inherited. If you cannot see the inheritance, add the matchers explicitly as a check.
3. Inspect amtool config show. This is what AM actually parsed. If the parsed tree does not match what you wrote, the YAML is wrong.
4. Tail the AM log. The line component=route aggrGroup=... includes the receiver name. Cross-check against GET /api/v2/alerts?receiver=....

Security implications

The routing tree is not security-sensitive on its own — it contains no secrets. The receivers it names are security-sensitive, and a misnamed receiver that points at a test integration in production can leak alert content to a Slack workspace that has no production access. The control is the review process: every PR that touches route: should be reviewed by someone who understands the on-call rotations, and every PR that touches receivers: should be reviewed by someone with access to the secrets manager.

Performance implications

Route evaluation is cheap. The cost lives in two places:

• The number of child routes evaluated per alert. A flat list of 50 routes means 50 evaluations per alert. A nested tree of depth 5 with a severity gate at the top means 1 evaluation in the common case. Prefer nesting over flatness when the fleet is large.
• The number of unique receivers touched by the same alert. Each unique receiver creates a goroutine and a notification log entry. The grouping lesson addresses this directly.

Production guidance

• Severity gate at the top, team split underneath. This is the shape that ages best. Team-only trees rot during re-orgs. Severity-only trees do not route to the right team.
• Always define the catch-all explicitly. default-slack with a real channel beats null or default.
• Critical pages use shorter timers. group_wait: 10s, group_interval: 2m, repeat_interval: 1h are reasonable defaults; tune by incident class, not by preference.
• Validate after every change. amtool check-config for syntax, amtool config routes test for behaviour, SIGHUP to load, then a synthetic alert to confirm.
• Document the tree. A diagram in the team’s instrumentation guide beats reading 200 lines of YAML during an incident.

Verification

You should now be able to answer:

• Why is a nested tree preferred over a flat list for a large fleet?
• What does equal: on a route do, and why is it not used for routing?
• How do group_wait, group_interval, and repeat_interval differ, and which one is the SLA boundary for a critical alert?
• What is the difference between match_re and matchers: [= "regex"]?
• Why does the catch-all receiver matter even when every alert should match a specific route?

Quiz