Runbook links — the alert to the documentation

The runbook URLs are the link from the alert to the documentation. The runbook content is the alert name, the impact, the diagnosis, and the remediation. This lesson walks the runbook URLs, the structure, the on-call integration, and the production patterns.

The runbook URL

The runbook URL:

annotations:
  summary: "High error rate"
  description: "Error rate is {{ $value | humanizePercentage }}"
  runbook_url: "https://runbook.example.com/error-rate"

The runbook URL is the input for the operator.

The runbook structure

The runbook structure:

# Alert: HighErrorRate

## Impact
- User-facing errors are above 5%
- The SLO error budget is being consumed

## Diagnosis
- Check the error rate: promtool query instant
- Check the affected services

## Mitigation
- Roll back the recent deployment
- Scale the cluster
- Disable the affected feature

## Escalation
- Primary: team-a-oncall
- Secondary: team-a-lead

## Related
- Dashboards: https://grafana.example.com/d/...
- Logs: https://grafana.example.com/explore?...

The runbook is the structured documentation.

The alert annotations

The alert annotations:

annotations:
  summary: "High error rate"
  description: "Error rate is {{ $value | humanizePercentage }}"
  runbook_url: "https://runbook.example.com/error-rate"
  dashboard: "https://grafana.example.com/d/error-rate"
  logs: "https://grafana.example.com/explore?service=nginx"
  escalation: "team-a-oncall"

The annotations are the context.

The on-call integration

The on-call integration:

# Slack receiver
title: "{{ .CommonAnnotations.summary }}"
text: |
  *Runbook:* {{ .CommonAnnotations.runbook_url }}
  *Dashboard:* {{ .CommonAnnotations.dashboard }}
  *Logs:* {{ .CommonAnnotations.logs }}
  *Escalation:* {{ .CommonAnnotations.escalation }}

The on-call integration is the Slack message.

The runbook severity

The runbook severity:

# Critical alert
runbook_url: "https://runbook.example.com/critical/error-rate"

# Warning alert
runbook_url: "https://runbook.example.com/warning/error-rate"

The runbook is per severity.

The runbook maintenance

The runbook maintenance:

flowchart LR
    A[Alert fires] --> B[Operator clicks runbook]
    B --> C[Runbook is updated]
    C --> D[Post-mortem input]
    D --> E[Runbook improvement]

The runbook is the input for the on-call.

The runbook for the cluster

The cluster-level runbook:

# Cluster: production

## Components
- 3 control plane (HA)
- 5 workers
- Cluster autoscaler enabled

## Common alerts
- ClusterDown: see "cluster-down" runbook
- NodeNotReady: see "node-not-ready" runbook

## Emergency contacts
- Primary: cluster-sre-oncall
- Secondary: cluster-sre-lead

The cluster-level runbook is the index.

The runbook for the workload

The workload-level runbook:

# Workload: nginx

## Common alerts
- HighErrorRate: see "error-rate" runbook
- HighLatency: see "latency" runbook
- PodCrashLooping: see "pod-crash" runbook

## Recent changes
- 2026-08-16: scaled to 10 replicas
- 2026-08-10: upgraded to v1.34.1

The workload-level runbook is the application.

The runbook validation

The runbook validation:

# Verify the URL is valid
curl -I https://runbook.example.com/error-rate

# Verify the URL is in the alert
promtool query instant http://prometheus:9090 ALERTS{alertname="HighErrorRate"}

The validation is direct.

The production patterns

The production patterns:

flowchart LR
    A[Alert] --> B[Slack message]
    B --> C[Runbook URL]
    C --> D[Runbook site]
    D --> E[Operator reads]
    E --> F[Diagnosis]
    F --> G[Mitigation]
    G --> H[Resolution]

The pattern is the production flow.

The cross-course references

• The Prometheus course covers the alerting.
• The Alertmanager course covers the routing.
• The SRE course covers the on-call patterns.

Quiz

Production discipline

• Add runbook URLs to all alerts. The on-call’s input.
• Structure the runbook. Impact, diagnosis, mitigation.
• Update the runbook. The post-mortem input.
• Test the runbook URL. Verify the URL is valid.
• Document the runbook URL. The canonical documentation.
• Review the runbook. Quarterly review.

The runbook URLs are the on-call’s input. Operating it well is the URLs, the structure, and the production patterns.