Alerting rules — the Prometheus to Alertmanager flow

The Prometheus alerting rules define the alerts. The PrometheusRule CRD is the Operator-style. The Alertmanager receives the alerts and routes them. This lesson walks the alerting rules, the CRD, the Alertmanager integration, and the production patterns.

The alerting flow

flowchart LR
    A[Prometheus] --> B[Alerting rule]
    B --> C[Alert]
    C --> D[Alertmanager]
    D --> E[Route]
    E --> F[Receiver]
    F --> G[Slack]
    F --> H[PagerDuty]
    F --> I[Email]

The alerting flow is the alert path.

The PrometheusRule alerting

The PrometheusRule alerting:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-alerts
  labels:
    release: my-prometheus
spec:
  groups:
  - name: my-alerts
    rules:
    - alert: HighErrorRate
      expr: |
        sum(rate(http_requests_total{status=~"5.."}[5m]))
        /
        sum(rate(http_requests_total[5m])) > 0.05
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High error rate on {{ $labels.service }}"
        runbook_url: "https://runbook.example.com/error-rate"

The PrometheusRule is the alerting configuration.

The alert fields

The alert fields:

- alert: <alert-name>
  expr: <PromQL>
  for: <duration>
  labels:
    severity: <level>
    team: <team>
  annotations:
    summary: <summary>
    description: <description>
    runbook_url: <url>

The fields are the alert’s metadata.

The for clause

The for clause:

for: 5m

The alert fires only after the condition has been true for 5 minutes. The clause prevents flapping.

The Alertmanager integration

The Alertmanager is configured via the Prometheus:

spec:
  alertmanager:
    endpoints:
    -Port: web
      path: /alertmanager

The Prometheus sends the alerts to the Alertmanager.

The Alertmanager config

The Alertmanager config:

apiVersion: monitoring.coreos.com/v1
kind: AlertmanagerConfig
metadata:
  name: my-config
  labels:
    release: my-prometheus
spec:
  route:
    receiver: default
    groupBy: [alertname, severity]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 4h
    routes:
    - matchers:
      - severity = warning
      receiver: slack-warnings
    - matchers:
      - severity = critical
      receiver: pagerduty-critical
  receivers:
  - name: default
    slackConfigs:
    - apiURL: https://hooks.slack.com/services/...
      channel: "#alerts"
  - name: slack-warnings
    slackConfigs:
    - apiURL: https://hooks.slack.com/services/...
      channel: "#alerts-warnings"
  - name: pagerduty-critical
    pagerdutyConfigs:
    - serviceKey: <key>

The Alertmanager config is the routing.

The routes

The routes:

flowchart LR
    A[Alert] --> B{Severity?}
    B -->|warning| C[Slack warnings]
    B -->|critical| D[PagerDuty critical]
    B -->|info| E[Slack info]

The routes are the routing.

The SLO-driven alerts

The SLO-driven alerts:

- alert: HighBurnRate
  expr: |
    1 - (
      sum(rate(http_requests_total{status=~"2.."}[5m]))
      /
      sum(rate(http_requests_total[5m]))
    ) > (1 - 0.999) * 14.4
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Error budget burning at 14.4x"
    runbook_url: "https://runbook.example.com/slo"

The SLO-driven alerts are the production pattern.

The production patterns

The production patterns:

flowchart LR
    A[Recording rules] --> B[Alerting rules]
    B --> C[Alertmanager]
    C --> D{Priority?}
    D -->|Low| E[Slack]
    D -->|High| F[PagerDuty]

The SLO-driven alerts are the production pattern.

The alert testing

The alert testing:

# Verify the alert is configured
promtool query instant http://prometheus:9090 ALERTS{alertname="HighErrorRate"}

# Check the alertmanager
amtool alert list

The testing is direct.

The alert silencing

The alert silencing:

amtool silence add --alertmanager http://alertmanager:9093 \
  --comment "Maintenance window" \
  --duration 1h \
  --start "2026-08-16T10:00:00Z" \
  --match "severity=warning"

The silencing is the temporary suppression.

The cross-course references

• The Alertmanager course covers the routing.
• The SLO/SLI course covers the burn rate.
• The Grafana course covers the dashboards.

Quiz

Production discipline

• Use the PrometheusRule CRD. The Operator style.
• Configure the routes. By severity, by team.
• Configure the receivers. Slack, PagerDuty, email.
• Use the SLO-driven alerts. The burn rate.
• Test the alerts. Verify the alerts are firing.
• Document the alerting. The rules, the routes, the receivers.

The Prometheus alerting is the cluster’s alert source. Operating it well is via the PrometheusRule CRD, with the Alertmanager routes, and the SLO-driven alerts.