Alert rules — Prometheus alerting rules for production

The Prometheus alerting rules are the production inputs. The severity levels, the SLO-driven alerts, the runbook integration, and the production patterns are the discipline. This lesson walks the alerting rules, the severity, the SLO-driven alerts, and the production patterns.

The alert rule structure

The alert rule structure:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-alerts
spec:
  groups:
  - name: my-alerts
    rules:
    - alert: HighErrorRate
      expr: |
        sum(rate(http_requests_total{status=~"5.."}[5m]))
        /
        sum(rate(http_requests_total[5m])) > 0.05
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High error rate on {{ $labels.service }}"
        description: "Error rate is {{ $value | humanizePercentage }}"
        runbook_url: "https://runbook.example.com/error-rate"

The structure is the alert’s contract.

The severity levels

The severity levels:

The levels are the routing.

The SLO-driven alerts

The SLO-driven alerts:

- alert: HighBurnRate
  expr: |
    1 - (
      sum(rate(http_requests_total{status=~"2.."}[5m]))
      /
      sum(rate(http_requests_total[5m]))
    ) > (1 - 0.999) * 14.4
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Error budget burning at 14.4x"
    runbook_url: "https://runbook.example.com/slo"

The SLO-driven alerts are the production pattern.

The multi-window multi-burn-rate alerts

The multi-window multi-burn-rate alerts:

groups:
- name: slo
  rules:
  # Fast burn (1h)
  - alert: SLOHighBurnRateFast
    expr: |
      1 - (
        sum(rate(http_requests_total{status=~"2.."}[1h]))
        /
        sum(rate(http_requests_total[1h]))
      ) > (1 - 0.999) * 14.4
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Error budget burning at 14.4x (1h window)"

  # Slow burn (6h)
  - alert: SLOHighBurnRateSlow
    expr: |
      1 - (
        sum(rate(http_requests_total{status=~"2.."}[6h]))
        /
        sum(rate(http_requests_total[6h]))
      ) > (1 - 0.999) * 6
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Error budget burning at 6x (6h window)"

The multi-window alerts catch the different time scales.

The runbook integration

The runbook integration:

annotations:
  summary: "High error rate"
  description: "Error rate is {{ $value | humanizePercentage }}"
  runbook_url: "https://runbook.example.com/error-rate"

The runbook URL is the input for the operator.

The alert testing

The alert testing:

# Validate the rule
promtool check rules my-rules.yaml

# Test the alert
promtool query instant http://prometheus:9090 ALERTS{alertname="HighErrorRate"}

# Check the alert expression
promtool test rules my-test.yaml

The testing is direct.

The alert ownership

The alert ownership:

annotations:
  summary: "High error rate"
  owner: "team-a@example.com"
  oncall: "team-a-oncall"

The ownership is the input for the on-call.

The production patterns

The production patterns:

flowchart LR
    A[Metric] --> B{Threshold?}
    B -->|yes| C[Alert]
    C --> D[For: 5m]
    D --> E[Route]
    E --> F{severity?}
    F -->|critical| G[PagerDuty]
    F -->|warning| H[Slack]
    G --> I[On-call]
    H --> J[Investigate]

The pattern is the production flow.

The cross-course references

• The Prometheus course covers the alerting.
• The SRE course covers the on-call patterns.
• The Slack / PagerDuty courses cover the receivers.

Quiz

Production discipline

• Use the multi-window multi-burn-rate alerts. The SRE pattern.
• Configure the severity. Critical, warning, info.
• Add the runbook URLs. The on-call’s input.
• Test the alerts. promtool check rules.
• Document the alerts. The rules, the runbooks.
• Review the alerts. Quarterly review.

The alert rules are the production inputs. Operating it well is the multi-window alerts, the severity, the runbook URLs, and the production patterns.