Runbook Ownership

A page arrives at 03:24. The alert is “Checkout5xxRateHigh.” The runbook_url annotation points to a Confluence page last edited in 2024 by an engineer who left the company in 2025. The page says “investigate the payment-svc dependency.” The engineer on-call opens the page, scrolls past a year of comments, and finds that the suggested command was a kubectl exec against a deployment that has been renamed twice since. The command fails; the engineer searches for the new deployment name; the alert continues to fire; forty minutes elapse before the dependency is identified.

This is what runbook ownership prevents. A runbook without an owner is a stale document: a procedure that worked for the last quarter’s architecture and that no one has updated for this quarter’s. The ownership is the commitment that the runbook is current, that the commands work, that the links resolve, and that the procedure reflects the current incident-response practice.

What runbook ownership is

Runbook ownership is the assignment of a named team to every runbook. The team is responsible for:

• The runbook’s accuracy — that the commands work, the links resolve, the dashboards are reachable, and the procedure reflects the current architecture.
• The runbook’s currency — that the runbook is reviewed at least quarterly and updated when the architecture changes.
• The runbook’s accessibility — that the URL is reachable from the alert annotation, the runbook site is searchable, and the runbook is in the team’s folder.
• The runbook’s retirement — that the runbook is removed or archived when the service is retired or the alert is removed.
• The runbook’s linkage — that every alert in the team’s alert rules has a runbook entry, and every runbook entry has a firing alert.

The ownership is not a Confluence label. The ownership is the conjunction of the runbook site folder, the alert annotation, the runbook review date, and the team’s review cadence.

Why a sysadmin cares

A runbook is the on-call engineer’s first action. The engineer wakes up, reads the alert, opens the runbook, and follows the procedure. If the procedure works, the incident is resolved in minutes. If the procedure is stale, the engineer spends twenty to sixty minutes reverse-engineering what the runbook should have said.

The trade-off is real. A runbook takes four hours to write; it takes two hours to review and update. The benefit is paid in every incident that follows the procedure. The cost is paid by the team that owns the runbook. Without executive sponsorship, the review cadence is the first thing to slip when the team is busy.

How it works

The most common shape in production is three runbook types:

   +----------------------+
   |  Alert runbooks      |  Audience: on-call engineer
   |  /payments/checkout  |     Scope: one alert
   |  -5xx                |     Action: follow the steps
   +----------------------+     to mitigate
            |
            v
   +----------------------+
   |  Service runbooks    |  Audience: service owner
   |  /payments/checkout  |     Scope: one service
   |  -service            |     Action: full operational
   +----------------------+     reference
            |
            v
   +----------------------+
   |  Playbook runbooks   |  Audience: incident commander
   |  /payments/incident  |     Scope: cross-service
   |  -response           |     Action: orchestration
   +----------------------+     reference

The three types have different audiences and different lifetimes. Alert runbooks survive the alert’s lifetime; service runbooks survive the service’s lifetime; playbook runbooks survive years and are reviewed annually.

The right approach is one runbook per alert, plus one service runbook per service, plus one playbook per team for incident response. The alert runbook is the on-call’s first action; the service runbook is the owner’s reference; the playbook is the incident commander’s coordination tool.

Under the hood: where runbook ownership lives

How to configure it

Three artefacts encode runbook ownership. The first is the runbook frontmatter, the second is the alert annotation that links to the runbook, the third is the audit script that catches the gaps.

Runbook page with frontmatter

---
title: 'Checkout 5xx rate high'
slug: checkout-5xx
owner: payments
team: payments
last_reviewed: '2026-08-01'
review_cadence_days: 90
alerts:
  - Checkout5xxRateHigh
tier: '1'
severity: critical
slo: availability
---

# Checkout 5xx rate high

## What this alert means

Checkout is returning HTTP 5xx responses for more than 1% of
requests over a 10-minute window. This is a user-visible
failure; users cannot complete purchases.

## First action

Open the [checkout service dashboard][dashboard] and confirm
the 5xx rate. The dashboard shows the rate per status code
and per endpoint.

[dashboard]: https://grafana.example.com/d/team-payments-checkout-service

## Likely causes (in order of probability)

1. **Payment service dependency is failing.** Check the
   [payment-svc dashboard][payment-dash]. If payment-svc is
   returning 5xx, the issue is downstream.

   [payment-dash]: https://grafana.example.com/d/team-payments-payment-svc

2. **Database connection pool exhausted.** Check the
   Postgres metrics:

   ```bash
   # READ-ONLY. Inspect current connection count.
   kubectl exec -n checkout deploy/checkout -- \
     curl -s http://localhost:8080/metrics | \
     grep pg_pool_in_use

3. Recent deploy introduced a regression. Check the deploy log for the checkout service in the last hour. If a deploy coincides with the alert, roll back.

Mitigation

If the cause is the payment service dependency, escalate to the payments on-call. If the cause is the database, scale the checkout deployment:

# CONFIGURATION: scale checkout to absorb the load.
kubectl scale -n checkout deploy/checkout --replicas=10

If the cause is a recent deploy, roll back:

# SERVICE-IMPACT: roll back to the previous revision.
kubectl rollout undo -n checkout deploy/checkout

After the incident

1. Confirm the alert has cleared.
2. Open a post-mortem ticket within 24 hours.
3. Update this runbook if any step was wrong or missing.

The frontmatter is what the audit greps for. The
`last_reviewed` date is the proof that the runbook is current;
the `alerts` field is the proof that the runbook covers an
existing alert.

### Alert annotation linking the runbook

```yaml
# /etc/prometheus/rules/checkout.yml
# CONFIGURATION: the runbook_url references the runbook
# slug, not the team root. The slug is stable across
# runbook-site refactors.
groups:
  - name: checkout.slo
    rules:
      - alert: Checkout5xxRateHigh
        expr: |
          sum(rate(checkout_http_requests_total{
            service="checkout",status=~"5.."}[5m]))
          /
          sum(rate(checkout_http_requests_total{
            service="checkout"}[5m]))
          > 0.01
        for: 10m
        labels:
          team: payments
          service: checkout
          severity: critical
          slo: availability
          tier: '1'
          environment: production
        annotations:
          summary: 'Checkout 5xx rate above 1% for 10m'
          # The runbook URL uses the runbook site's stable
          # path. Refactoring the runbook site must preserve
          # the slug or update the alert annotation.
          runbook_url: 'https://runbooks.example.com/payments/checkout-5xx'
          dashboard_url: 'https://grafana.example.com/d/team-payments-checkout-service'

The URL uses the runbook slug. Refactoring the runbook site must either preserve the slug or update the alert annotation; the CI gate catches the gap when the annotation is updated.

Runbook audit script

# /usr/local/bin/audit-runbooks.sh
# CONFIGURATION: every-quarter audit that confirms the
# runbook site and the alert rules agree on the mapping.
set -euo pipefail

runbook_root="https://runbooks.example.com"
audit_report="/var/log/runbook-audit.csv"

echo "runbook,owner,team,last_reviewed,alert,alert_has_runbook" \
  > "$audit_report"

# Iterate over every alert rule and confirm a runbook exists
# and is reviewed within cadence.
for rule_file in /etc/prometheus/rules/*.yml; do
  alert=$(grep -oP 'alert: \K\w+' "$rule_file" | head -1)
  runbook_url=$(grep -oP "runbook_url: '\K[^']+" "$rule_file" \
                | head -1)
  if [ -z "$runbook_url" ]; then
    echo "$alert,,,,MISSING,"
    continue
  fi
  status=$(curl -s -o /dev/null -w '%{http_code}' "$runbook_url")
  if [ "$status" != "200" ]; then
    echo "$alert,,,,$runbook_url,HTTP $status"
    continue
  fi
  # Fetch the runbook frontmatter.
  frontmatter=$(curl -s "$runbook_url" | sed -n '/^---$/,/^---$/p')
  owner=$(echo "$frontmatter" | grep -oP 'owner: \K\w+' | head -1)
  last_reviewed=$(echo "$frontmatter" \
                  | grep -oP 'last_reviewed: .K[0-9-]+' \
                  | head -1)
  age_days=$(( ( $(date +%s) - $(date -d "$last_reviewed" +%s) ) \
               / 86400 ))
  echo "$runbook_url,$owner,$owner,$last_reviewed,$alert,$age_days"
done

# Report any runbook reviewed more than 90 days ago.
awk -F, '$6 != "" && $6+0 > 90 {print "STALE:", $0}' "$audit_report"

The audit produces a CSV that the platform team reviews quarterly. A row with STALE is a runbook that has not been reviewed within the cadence; the team owes an update.

How to validate it

Validation is the conjunction of the five artefacts. The audit confirms every runbook has all five.

# READ-ONLY. List every runbook URL referenced by an alert
# and confirm the URL resolves.
for url in $(grep -RH "runbook_url:" /etc/prometheus/rules/ \
              | awk -F"'" '{print $2}'); do
  status=$(curl -s -o /dev/null -w '%{http_code}' "$url")
  echo "$status $url"
done

# READ-ONLY. List every runbook page and confirm the
# frontmatter has owner, team, last_reviewed, and alerts.
for page in $(curl -s https://runbooks.example.com/api/pages \
              | jq -r '.[].slug'); do
  frontmatter=$(curl -s https://runbooks.example.com/$page \
                | sed -n '/^---$/,/^---$/p')
  for field in owner team last_reviewed alerts; do
    if ! echo "$frontmatter" | grep -q "^$field:"; then
      echo "MISSING $field: $page"
    fi
  done
done

# READ-ONLY. List every runbook reviewed more than 90 days
# ago.
find /var/log/runbook-audit.csv -mtime -1 -exec \
  awk -F, '$6+0 > 90 {print "STALE:", $1}' {} \;

Illustrative output for the runbook audit:

$ for url in $(grep -RH "runbook_url:" /etc/prometheus/rules/ \
                | awk -F"'" '{print $2}'); do
    status=$(curl -s -o /dev/null -w '%{http_code}' "$url")
    echo "$status $url"
  done
200 https://runbooks.example.com/payments/checkout-5xx
200 https://runbooks.example.com/payments/checkout-latency
404 https://runbooks.example.com/payments/checkout-cache-miss

$ find /var/log/runbook-audit.csv -mtime -1 -exec \
    awk -F, '$6+0 > 90 {print "STALE:", $1}' {} \;
STALE: https://runbooks.example.com/payments/checkout-cache-miss

The 404 is the gap: the alert’s runbook URL returns 404. The audit catches it. The STALE row is the same page, because the 404 also means the frontmatter cannot be read.

How it can fail

Six failure shapes appear repeatedly when runbooks are unowned:

1. Confluence graveyard. Runbooks live in a Confluence space that no one reviews. Symptom: the runbook URL 404s after the Confluence space is restructured; the alert annotation points to the old URL.
2. Generic team root. The alert’s runbook_url points to https://runbooks.example.com/payments instead of the specific alert’s runbook. Symptom: the on-call engineer opens the team index and spends minutes navigating to the right page.
3. Stale commands. The runbook says kubectl exec deploy/checkout-api but the deployment was renamed to deploy/checkout six months ago. Symptom: the command fails; the engineer searches for the new name; the alert continues to fire.
4. Review cadence drift. The runbook was last reviewed fourteen months ago. Symptom: the procedure is correct in spirit but wrong in detail; the engineer has to improvise.
5. Runbook with no alert. A page in the runbook site documents a service that has no alerting rule. Symptom: the runbook is read once when the service is deployed and never again.
6. Alert with no runbook. A firing alert has no runbook URL. Symptom: the on-call engineer has no documented action; the page is paid for nothing.

How to troubleshoot it

The diagnostic order for “is this runbook owned?”:

1. Confirm the URL resolves. curl -I the runbook_url annotation. A 404 means the runbook has been moved or deleted.
2. Confirm the frontmatter. Open the runbook page and confirm owner, team, last_reviewed, alerts are present.
3. Confirm the alert mapping. The runbook’s alerts field should include the alertname. A runbook that documents an alert that does not exist is one reorganisation away from being unowned.
4. Confirm the commands. Run the kubectl exec, curl, and other commands in the runbook on a staging host. The commands should work; if they do not, the runbook is stale.
5. Confirm the review cadence. The last_reviewed date should be within review_cadence_days of today. A runbook older than the cadence is stale.
6. Form the diagnosis. Missing URL, or missing frontmatter, or broken alert mapping, or broken commands, or stale review. Each is a separate fix.

Security implications

Runbook ownership intersects with security at the credential boundary. Runbook commands may include credentials (database passwords, API keys, cloud provider tokens). The discipline:

• Runbook commands use service-account credentials, not personal credentials. The credentials are short-lived and scoped to the action.
• The runbook does not include long-lived secrets in plaintext. A runbook that says “use the database master password in 1Password” is acceptable; a runbook that pastes the password is not.
• The runbook site access is gated by SSO. A runbook URL that resolves without authentication is an exposure.
• Runbook commands that include user data (a SELECT with a user ID) are reviewed by the privacy team.

Performance implications

Runbook site performance is bounded by page count and search latency. A runbook site with 1,000 pages loads the search in 200ms; one with 10,000 pages loads it in 2s. The discipline:

• Retire runbooks for retired services. The retirement is a checklist item on the service retirement ticket.
• Use static site generators. The runbook site should be a static site (Hugo, MkDocs, Astro) served from a CDN, not a dynamic CMS that requires a database query per page.
• Search the runbook site from the alert. A search box on the alert page that hits the runbook site’s search API reduces the time to find a runbook from minutes to seconds.

Production guidance

• Review runbooks quarterly. The review confirms commands work, links resolve, and the procedure reflects the current architecture.
• Use a stable runbook URL. The URL uses the runbook slug, not the team’s root.
• Test the commands. The quarterly review includes running the commands on a staging host.
• Link the runbook from the alert. The CI gate enforces that every alert has a runbook URL and that the URL resolves.
• Audit the five-artefact conjunction monthly. A runbook that fails any of the five is a candidate for retirement or update.

Verification

You should now be able to answer:

• What five artefacts prove runbook ownership?
• What is the difference between alert, service, and playbook runbooks?
• Why must every alert have a runbook URL and every runbook be reviewed quarterly?
• What is the failure shape when a runbook references a deployment that has been renamed?
• How do you audit a runbook site for ownership?

Quiz