Alertmanager Config as Code

A high-error-rate page fires for the payment-service team at 02:14. The receiver list says payments-pager. The Slack channel says #payments-incidents. The page goes out to the on-call rotation and to a Slack channel the team no longer monitors, because someone changed the route.matchers.team from team=payments to team=payments|operations two months ago and forgot to update the receiver. The continue: true on the parent route meant the alert was routed to both children. The on-call rotation got the page; the team Slack never read it.

This is the failure shape that alertmanager.yml as code exists to prevent. The routing tree is the operative definition of “who gets paged on what”. A change to that tree is a change to human behaviour, and it goes through review like any other production code.

What it is

alertmanager.yml is the configuration file Alertmanager reads at startup and on every SIGHUP. It declares four operational contracts:

• global — defaults that every receiver inherits (SMTP, Slack, PagerDuty, OpsGenie, VictorOps, webhook).
• templates — Go-templated notification bodies reused across receivers.
• route — the routing tree, a parent node with a list of child routes. Each route has matchers, a receiver, and routing-control flags (continue, group_by, group_wait, group_interval, repeat_interval, mute_time_intervals).
• receivers — the named destinations: Slack, PagerDuty, email, webhook, and so on.

The file is YAML. Alertmanager parses it with the same loader whether at startup, on SIGHUP, or via amtool check-config. Any schema mistake that amtool rejects will also fail Alertmanager on reload.

Why a sysadmin cares

The alertmanager config is the loudest part of the on-call experience. When it is wrong:

1. Pages go to the wrong rotation. A matcher that catches team=operations does not match team=ops (Alertmanager does not support substring matchers without a regex).
2. Pages are duplicated. continue: true on a parent route and a default child that always matches sends every alert to two receivers.
3. Pages are silenced forever. mute_time_intervals: referencing a time interval name that does not exist produces a routing error that Alertmanager logs and ignores, leaving the alert actively muted without a workable on-call path.
4. Pages never reach the destination. A slack_configs.api_url that points at a webhook revoked two quarters ago produces silent failure.

Each of these is preventable. Each is amplified when the file is not under configuration management.

How it works

The mental model is “alerts come in, the routing tree sends them out”:

  Prometheus
     |
     |  active alerts (group, label-set, since)
     |
     v
  Alertmanager
     |
     |  route: tree walk top to bottom
     |  +-- matcher check on each child
     |  +-- continue: true? keep walking siblings
     |  +-- otherwise stop at first match
     |
     v
  Receiver (slack_configs, pagerduty_configs, ...)
     |
     v
  Pager / Slack / Email / Webhook

Two details follow. First, Alertmanager’s default for continue is false. If a route matches and continue: false (the default), sibling routes are not evaluated. To fan out to multiple teams, each sibling must explicitly state continue: true or be configured separately.

Second, Alertmanager does not enforce a unique-receiver constraint. Multiple routes may point at the same receiver. The grouping, deduplication, and silencing logic deduplicate alert fingerprints, not deliveries; if the same alert reaches two sibling receivers and both fire, the on-call gets two pages.

How to configure it

A working alertmanager.yml for a small production tree:

global:
  resolve_timeout: 5m
  slack_api_url_file: /etc/alertmanager/secrets/slack-default.url

templates:
  - default.tmpl

route:
  receiver: default
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="critical"
      receiver: pager-rotation
      group_wait: 10s          # critical pages faster
      repeat_interval: 1h
      continue: true
    - matchers:
        - team="payments"
      receiver: payments-slack
    - matchers:
        - team="platform"
        - severity!="info"
      receiver: platform-pager-and-slack
      continue: true
      routes:
        - matchers:
            - severity="info"
          receiver: platform-slack-only

receivers:
  - name: default
    slack_configs:
      - channel: '#observability-default'
        send_resolved: true

  - name: pager-rotation
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pagerduty.key
        send_resolved: true
        severity: critical
    slack_configs:
      - channel: '#ops-pager'
        send_resolved: true

  - name: payments-slack
    slack_configs:
      - channel: '#payments-incidents'
        send_resolved: true

  - name: platform-pager-and-slack
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/platform-pagerduty.key
    slack_configs:
      - channel: '#platform-incidents'
        send_resolved: true

  - name: platform-slack-only
    slack_configs:
      - channel: '#platform-info'

inhibit_rules:
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal: ['alertname', 'cluster', 'service']

Notes on the choices:

• slack_api_url_file and routing_key_file are the credentials surface. The file is mode 0400 and owned by the alertmanager user; the contents are read at reload. No plaintext secrets in Git.
• continue: true on the critical route lets it continue to the team-specific child so that the team also gets a Slack notification in parallel. Default is false.
• repeat_interval: 4h on the top route with repeat_interval: 1h on the critical child escalates critical alerts faster.
• inhibit_rules suppress the warning alert when a critical alert fires for the same service and alertname. Useful for “API errors critical” silencing “API errors warning”.
• The templates directory lives at /etc/alertmanager/. The path is supplied as --config.file=/etc/alertmanager/alertmanager.yml and the templates dir is relative to that path.

The CI gate:

.PHONY: alertmanager
alertmanager:
  amtool check-config alertmanager/alertmanager.yml

How to validate it

Three validations across the CI/deploy lifecycle.

# 1. CI: file parses with the runtime loader
amtool check-config alertmanager/alertmanager.yml

# 2. Prod: Alertmanager is using the loaded config
curl -fsS http://alertmanager-prod-01:9093/-/ready
curl -fsS http://alertmanager-prod-01:9093/api/v2/status \
  | jq '.data.config.original' -r

# 3. Prod: a routing test against the live Alertmanager
amtool config routes test \
  --alertmanager.url=http://alertmanager-prod-01:9093 \
  --config.file=alertmanager/alertmanager.yml \
  <<EOF
[
  {
    "labels": {
      "alertname": "HighErrorRate",
      "severity": "critical",
      "team": "payments",
      "cluster": "prod-eu-west-1"
    }
  }
]
EOF

amtool config routes test walks the live routing tree with a synthetic alert, returning which receiver(s) would fire. The output is a JSON list of matched routes. A test that returns the empty list when it should return a team-specific receiver is a routing regression that the CI gate should catch.

How it can fail

Six concrete failure modes appear repeatedly.

1. continue: true fan-out. A new sibling with continue: true sends every alert to the new receiver as well as the existing one. The on-call gets two pages. The fix is to model “page only” paths and “page plus Slack” paths as separate receivers with explicit routing, and to lint continue flags in CI.
2. Matcher case sensitivity. Alertmanager matchers are case-sensitive. severity=critical does not match severity=Critical. The fix is a CI rule that asserts the metric-side labels are normalised at emission.
3. Receiver name typo. A route points at payments-pager but the receivers block declares payments-pagerduty. The YAML loader accepts the name; Alertmanager logs “no receivers configured for route”; the alert is dropped. The fix is amtool check-config plus a CI assertion that every route[*].receiver exists in receivers[*].name.
4. mute_time_intervals: referencing an undefined interval. A route points at a time_intervals.business-hours block that does not exist. Alertmanager logs the error and treats the interval as empty (mute never). The fix is a CI assertion that interval names exist.
5. Webhook URL changes silently. The Slack webhook is rotated; the secret file is updated; the deployment that should pick it up does not. Slack rejects the post with invalid_auth. The fix is a CI smoke test that posts a synthetic test alert and asserts the response from each configured receiver.
6. inhibit_rules over-broad. An inhibit rule with empty target_matchers silences everything. The fix is a CI rule that asserts every inhibit rule has at least one target_matchers entry.

Security implications

The alertmanager config is a credentials store in disguise. The discipline:

• No credentials in YAML. Use *_file (api_url_file, routing_key_file) for every credential, with the file mode 0400 and owned by the alertmanager user.
• Webhook URLs and PagerDuty keys are high-value secrets. They grant the ability to page internal users or post to internal channels.
• The -/ready and api/v2/status endpoints expose the loaded configuration. Anyone who can reach them can see the routing tree. Bind Alertmanager to a private listener and front it with auth.
• inhibit_rules do not move across teams. A team that adds an inhibit rule against another team’s alerts can silence them silently. The fix is a code-review rule that inhibits rules must be reviewed by both teams’ CODEOWNERS.

Performance implications

Alertmanager’s cost grows with the rate of incoming alerts and the size of the alert label set, not with the size of the routing tree. The big knobs:

• group_by — wider grouping means fewer notifications; coarser aggregation loses detail.
• group_wait — short wait means faster notifications; long wait means fewer.
• group_interval — short interval means more notifications; long interval means fewer.
• repeat_interval — short repeat means more noise; long repeat means longer to escalation.

The routing tree itself is O(routes) per alert. A tree of 100 routes per alert matches most production teams; the cost is not the bottleneck. The bottleneck is the rate of incoming active alerts during a real incident, which can be in the tens of thousands.

Production guidance

• One alertmanager.yml per cluster, in Git.
• amtool check-config in CI on every change.
• amtool config routes test for every critical alert in CI, asserted to hit a PagerDuty receiver.
• All credentials in *_file paths. No plaintext anywhere.
• Test silence rules and time-interval rules in CI.
• Run Alertmanager as a cluster (replicas) with a gossip layer for HA; the file is the source of truth, the cluster is the runtime.
• Reload Alertmanager with SIGHUP after every deploy. CI smoke test that the loaded config matches Git.

Verification

You should now be able to answer:

• What does continue: true on a parent route do, and why does the default of false require care when modelling “page and Slack” paths?
• Why is slack_api_url_file the right posture for the Slack webhook URL, and what file mode should the credentials file have?
• What is the failure shape when a route matcher is case-sensitive and the alert label is emitted in a different case?
• What does amtool config routes test do, and what should it be asserted to produce for every critical alert label set?

Quiz