Writing runbooks that actually get used

What a runbook is (and isn’t)

A runbook is a procedure document. Its purpose: a stressed engineer at 3am can follow it without thinking. Specifically:

• Step-by-step — exact commands, exact order
• Deterministic — same input, same output
• Bounded — clear start, clear end, clear success criteria
• Standalone — does not require context from other docs

A runbook is not:

• An architecture document (those explain why)
• A post-mortem (those explain what happened)
• A tutorial (those explain how to learn)

The anatomy of a good runbook

Every runbook should have:

Header

• Title: what the runbook does, not what it’s for (“Restart a hung corosync” beats “Corosync stuff”)
• Owners: who maintains it (with team and on-call handle)
• Last tested: date the runbook was last executed successfully in a real or test environment
• Severity: what happens if this isn’t done
• Estimated time: how long it takes

Pre-flight

• What you need — credentials, access, tools
• Verification — confirm you’re looking at the right system
• Pre-conditions — what state must be true before starting

Steps

• Numbered, in order
• One action per step
• Expected output after each step
• Decision points clearly marked

Verification

• How to confirm the runbook succeeded
• What good looks like
• What bad looks like

Rollback

• How to undo the change if it fails
• When to invoke rollback (clear trigger)

A real example: restarting a hung VM

# Restart a hung VM (vm-id 100, web01)

**Owner**: platform-team (slack: #platform-oncall)
**Last tested**: 2026-08-05
**Severity**: P3 (degraded user experience, no data loss)
**Estimated time**: 5 minutes

## Pre-flight

- You have root SSH access to all cluster nodes
- You are NOT in a maintenance window (if so, see "Maintenance restart" runbook)

## Verify

1. SSH to the node hosting vm-100:
   `ssh root@pve1`
2. Confirm the VM is hung:
   `qm status 100`
   Expected: status "running" but no console response, or status "stopped" unexpectedly.
3. Check the cluster state:
   `pvecm status`
   Expected: cluster is healthy, quorum is intact.

## Steps

1. Try graceful shutdown:
   `qm shutdown 100 --timeout 30`
   Expected: status changes to "stopped" within 30 seconds.

2. If shutdown hung, force stop:
   `qm stop 100 --forceStop 1 --timeout 30`
   Expected: status "stopped".

3. Wait 5 seconds, then start:
   `qm start 100`
   Expected: status "running".

4. Watch the console:
   `qm monitor 100`
   Exit with Ctrl+].

## Verification

- [ ] `qm status 100` shows "running"
- [ ] Console responds within 60 seconds of boot
- [ ] Smoke test the application (URL, key service check)
- [ ] Backup job runs on next schedule

## Rollback

If the VM doesn't start cleanly, restore from last backup:
1. Open Proxmox UI → VM 100 → Backups
2. Pick the most recent backup
3. Restore to a NEW VM ID (e.g., 199)
4. Boot 199 and verify
5. If 199 works, swap DNS or IP, then power off 100

Notice: no prose explanation. Just steps.

Runbook vs. procedure vs. checklist

Don’t make every doc a runbook. Use the right format for the job.

Common runbook mistakes

“It’s obvious”

It’s not. What you do without thinking is invisible to you. Write it down.

“I tested it once”

When did you test it? What was the system state? When did the runbook last run in production? If “Last tested” is more than 6 months old, the runbook is stale.

“It’s just commands”

Where do you run the commands? As what user? With what credentials? What if the command isn’t available? Runbooks should be runnable end-to-end.

“Step 5: debug and fix”

That’s not a step. Break it down into specific actions or escalate.

“It assumes you know”

Write for the on-call at 3am, who has never seen this system before. Be explicit. Be tedious. Tedium saves incidents.

Testing runbooks

A runbook that hasn’t been tested is fiction. Test it:

1. In staging — exactly execute the runbook in a non-prod environment
2. With a junior — if a junior can follow it, anyone can
3. Under time pressure — give someone 10 minutes to execute it cold; time the result
4. Periodically — quarterly at minimum; update after any failure

Tools for runbooks

• Markdown files in a Git repo — versioned, reviewable, diffable
• Confluence / Notion — easier for non-engineers to edit
• Runbook automation — Rundeck, StackStorm, n8n — for runbooks that need many steps across many systems
• Static site generators (this site, ironically) — searchable, linkable

Pick the simplest one that fits. Markdown in Git is often the right answer.

Runbook lifecycle

1. Identify the need — after an incident, recurring task, or new system
2. Draft — write the steps while the work is fresh in your mind
3. Review — peer review by someone who didn’t write it
4. Test — execute in staging, with a junior, under time pressure
5. Publish — link from the team’s runbook index, training material
6. Maintain — test quarterly; update when the system changes
7. Retire — if the system goes away, retire the runbook (don’t leave dead docs)

Key takeaways

• A runbook is a procedure, not a tutorial
• Every step has an expected output
• Test runbooks with a junior before relying on them
• “Last tested” date is essential metadata
• Runbooks go stale; test them quarterly

Knowledge check