Alertmanager — the alert routing engine

Alertmanager is the alert routing engine. It receives alerts from Prometheus, routes them to the receivers, applies inhibition and silences, and manages the notifications. The Alertmanager is HA via a pair. This lesson walks the architecture, the routing, the receivers, and the production patterns.

The Alertmanager architecture

The Alertmanager architecture:

flowchart LR
    A[Prometheus] --> B[Alertmanager]
    B --> C{Group}
    C --> D[Inhibit]
    D --> E[Silence]
    E --> F[Route]
    F --> G[Receiver]
    G --> H[Slack]
    G --> I[PagerDuty]
    G --> J[Email]

The architecture is the alert flow.

The Alertmanager HA

The HA is via a pair:

flowchart LR
    A[Prometheus] --> B[Alertmanager 1]
    A --> C[Alertmanager 2]
    B --> D[Gossip]
    C --> D
    B --> E[Receivers]
    C --> E

The HA is via the gossip protocol.

The AlertmanagerConfig

The AlertmanagerConfig:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: my-config
  labels:
    alertmanagerConfig: main
spec:
  route:
    receiver: default
    groupBy: ['alertname', 'severity']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 4h
    routes:
    - matchers:
      - severity = warning
      receiver: slack-warnings
    - matchers:
      - severity = critical
      receiver: pagerduty-critical
  receivers:
  - name: default
    slackConfigs:
    - apiURL: https://hooks.slack.com/services/...
      channel: "#alerts"
  - name: slack-warnings
    slackConfigs:
    - apiURL: https://hooks.slack.com/services/...
      channel: "#alerts-warnings"
  - name: pagerduty-critical
    pagerdutyConfigs:
    - serviceKey: <key>

The AlertmanagerConfig is the configuration.

The routing

The routing tree:

flowchart LR
    A[Alert] --> B{severity?}
    B -->|warning| C[slack-warnings]
    B -->|critical| D[pagerduty-critical]
    B -->|info| E[Email]

The routing tree is the conditional.

The receivers

The receivers:

The receivers are the destinations.

The inhibition

The inhibition:

inhibit_rules:
- source_matchers:
  - alertname = ClusterDown
  target_matchers:
  - severity = warning
  equal: [cluster]

The inhibition suppresses the warning alerts when the cluster is down.

The silences

The silences:

amtool silence add --alertmanager http://alertmanager:9093 \
  --comment "Maintenance window" \
  --duration 1h \
  --start "2026-08-16T10:00:00Z" \
  --match "severity=warning"

The silences are the temporary suppression.

The grouping

The grouping:

route:
  receiver: default
  groupBy: ['alertname', 'severity']
  groupWait: 30s
  groupInterval: 5m

The grouping is the alert aggregation.

The production patterns

The production patterns:

flowchart LR
    A[Prometheus alert] --> B[Alertmanager]
    B --> C[Group]
    C --> D[Inhibit]
    D --> E[Silence]
    E --> F[Route]
    F --> G[Slack]
    F --> H[PagerDuty]

The pattern is the production flow.

The cross-course references

• The Prometheus course covers the alerting.
• The Grafana course covers the dashboards.
• The SRE course covers the production patterns.

Quiz

Production discipline

• Deploy the Alertmanager HA. The pair.
• Configure the routes. The conditional routing.
• Configure the receivers. Slack, PagerDuty, email.
• Use the inhibition. The noise reduction.
• Use the silences. The temporary suppression.
• Document the configuration. The routes, the receivers.

The Alertmanager is the cluster’s alert routing engine. Operating it well is the HA, the routes, the receivers, and the inhibition.