Alert routing — the routing trees and the receivers

The Alertmanager routing trees are the conditional routing for the alerts. The routing is based on the matchers; the receivers are the destinations. This lesson walks the routing trees, the matchers, the continues, and the production patterns.

The routing tree

The routing tree:

flowchart LR
    A[Alert] --> B{severity?}
    B -->|critical| C[pagerduty-critical]
    B -->|warning| D[slack-warnings]
    B -->|info| E[Email]
    D --> F{team?}
    F -->|team-a| G[team-a-slack]
    F -->|team-b| H[team-b-slack]

The tree is the conditional routing.

The matchers

The matchers:

route:
  receiver: default
  routes:
  - matchers:
    - severity = critical
    receiver: pagerduty-critical
  - matchers:
    - severity = warning
    receiver: slack-warnings
  - matchers:
    - alertname =~ "High.*"
    receiver: high-alerts

The matchers are the conditions.

The match operator

The match operators:

# Exact match
matchers:
- severity = critical

# Regex match
matchers:
- alertname =~ "High.*"

# Negation
matchers:
- severity != critical

# Multiple matchers (AND)
matchers:
- severity = critical
- team = team-a

The match operators are the conditions.

The continue field

The continue field:

route:
  receiver: default
  routes:
  - matchers:
    - severity = critical
    receiver: pagerduty-critical
    continue: true  # the alert continues down the tree
  - matchers:
    - severity = critical
    receiver: slack-critical  # also route to Slack

The continue field allows the alert to be routed to multiple receivers.

The group_by

The group_by:

route:
  receiver: default
  groupBy: ['alertname', 'severity', 'cluster']
  groupWait: 30s
  groupInterval: 5m

The group_by aggregates the alerts.

The receivers

The receivers:

receivers:
- name: pagerduty-critical
  pagerdutyConfigs:
  - serviceKey: <key>
    severity: critical
    description: "{{ .CommonAnnotations.summary }}"
- name: slack-warnings
  slackConfigs:
  - apiURL: https://hooks.slack.com/services/...
    channel: "#alerts-warnings"

The receivers are the destinations.

The routing for multi-tenant

The routing for multi-tenant:

route:
  receiver: default
  routes:
  - matchers:
    - team = team-a
    receiver: pagerduty-team-a
  - matchers:
    - team = team-b
    receiver: pagerduty-team-b

The routing is per tenant.

The routing for severity and tenant

The routing for severity and tenant:

route:
  receiver: default
  routes:
  - matchers:
    - severity = critical
    routes:
    - matchers:
      - team = team-a
      receiver: pagerduty-team-a-critical
    - matchers:
      - team = team-b
      receiver: pagerduty-team-b-critical
  - matchers:
    - severity = warning
    receiver: slack-warnings

The routing is per severity and tenant.

The production patterns

The production patterns:

flowchart LR
    A[Alert] --> B{severity?}
    B -->|critical| C[On-call PagerDuty]
    B -->|warning| D[Slack]
    D --> E{team?}
    E -->|team-a| F[team-a channel]
    E -->|team-b| G[team-b channel]

The pattern is the production routing.

The amtool

The amtool is the CLI:

# Check the routing
amtool config routes --alertmanager.url=http://alertmanager:9093

# Test the routing
amtool config routes test --alertmanager.url=http://alertmanager:9093 --config.file=alertmanager.yaml

The amtool is the CLI for the Alertmanager.

The cross-course references

• The Alertmanager course covers the routing.
• The Prometheus course covers the alerting.
• The PagerDuty / Slack courses cover the receivers.

Quiz

Production discipline

• Configure the routing tree. The conditional routing.
• Use the matchers. The conditions.
• Use the continue. The multi-receiver routing.
• Use the group_by. The aggregation.
• Test the routing. amtool config routes.
• Document the routing. The trees, the receivers.

The routing trees are the cluster’s alert routing. Operating it well is the matchers, the receivers, the continues, and the production patterns.