Runbooks that work at 3am

The previous two lessons covered what to document. This one is about writing it for the person who will actually read it, and that person is very specific.

They were asleep twenty minutes ago. They are not the owner of this system; the owner is on annual leave. They have one terminal, a page that says something is down, and no appetite for a document that assumes context they do not have. Everyone they could ask is also asleep, and waking someone costs ten minutes they do not have.

Every rule below follows from that reader.

The reader cannot ask a question

This is the whole design constraint. A runbook step that prompts a question the reader cannot answer stops the procedure dead, and what happens next is improvisation on a production system by someone with partial context - which is the outcome the runbook existed to prevent.

Three phrases guarantee it, and all three are common:

Expected output is the highest-value line in a step

A step without an expected result tells the reader what to type. A step with one tells them whether they are still on the path - which is the only information they actually need to make the next decision.

BEFORE
  3. Check that the cluster is healthy.

AFTER
  3. Check that the cluster is healthy:

       pcs status --full

     EXPECT: every node listed as Online, and no resource in
     a Stopped or FAILED state.

     IF any node is OFFLINE  -> go to step 11 (node recovery)
     IF any resource FAILED  -> go to step 14 (resource cleanup)

The “after” version is longer, and the length is the point. The reader can execute it, compare, and branch, without knowing what a healthy pcs status normally looks like on this cluster. The “before” version silently requires that knowledge and gives no signal that it is missing.

Commands must be runnable exactly as written

A runbook is copied and pasted at 3am. Anything that needs to be mentally substituted will be substituted wrongly or not at all.

The specific trap on Linux is the angle-bracket placeholder, because in a shell < is an input redirection operator rather than a hint:

# Broken: the shell tries to read from a file called "vip"
# curl http://<vip>/health

# Correct: bind the variable once at the top of the runbook
VIP=192.0.2.10
curl -sS -o /dev/null -w '%{http_code}\n' "http://$VIP/health"

Put every such value in a “set these first” block at the top of the runbook, so substitution happens once, deliberately, before anything is executed:

VIP=192.0.2.10
NODE=app01.example.com
UNIT=myapp.service

Then every later step is copy-pasteable verbatim. This also makes the runbook reviewable: a reviewer can see at a glance what the procedure is parameterised on.

Bound every wait

“Wait for the service to come up” has no end. Under pressure it becomes either thirty seconds - too short, so the responder escalates a recovery that was working - or twenty minutes of watching a spinner while the outage continues.

  7. Start the service and watch it:

       systemctl start "$UNIT"
       systemctl status "$UNIT" --no-pager

     EXPECT: active (running) within 60 seconds.

     IF still "activating" after 2 minutes, do not restart it
     again. Go to step 12 and collect the diagnostics before
     escalating.

Three things are bounded there: how long to wait, what to do when the bound is reached, and - explicitly - what not to do. The instruction not to restart again matters, because repeated restarts are the natural reflex and they destroy the evidence the escalation will need.

The title is a search term

A runbook is found by someone typing what they are seeing into a search box. So the title should contain the symptom, not the subsystem:

• Runbook: Investigate a systemd service that will not start is found by someone whose service will not start.
• MyApp Operational Guide is found by someone who already knows the answer is in the MyApp documentation.

The same reasoning gives the “when to use this runbook” section its job: it lets a reader confirm in ten seconds that they are in the right document, and - just as importantly - a “when not to use this” list that redirects them. A responder who works the wrong runbook to completion loses more time than one who never found it.

The shape that holds up

The runbooks in this course carry their structure in frontmatter so it cannot be omitted by accident:

$ grep -E '^(pre_checks|procedure_steps|verification|rollback|escalation|risk|impact):' src/content/courses/linux/runbooks/linux-runbook-reboot-production-server.mdx

risk: high
impact: service-affecting
pre_checks:
procedure_steps:
verification:
rollback:
escalation:

Each of those answers a question the reader has:

• risk / impact - what am I about to do to production? Read before starting, not after.
• pre_checks - what must be true before I begin? This is where “confirm nobody else is already working this incident” lives, and it prevents the second-worst incident outcome: two people fixing the same thing in opposite directions.
• procedure_steps - the ordered work.
• verification - how do I know it worked? Distinct from the steps, and stated in terms of service behaviour rather than command exit status. “The service answers its health check” beats “systemctl returned 0”.
• rollback - how do I undo this? Written before the change is attempted, because writing it afterwards means writing it during the failure.
• escalation - who, and when. Named rotas, with the condition that triggers each.

If your documentation system has no such schema, put the same headings in the template. The value is that a missing section is visible.

Testing a runbook

A runbook that has never been executed is a hypothesis. The test is cheap and almost nobody runs it:

Hand it to a colleague who has not worked on this system. Ask them to execute it in a non-production environment while you watch. Do not help. Do not answer questions - write each one down instead.

Every question they ask is a defect in the document. Every pause is a step that needs an expected result. Every moment they scroll back up is a value that should have been in the “set these first” block.

Half an hour of that finds more defects than any review, because a reviewer reads with the author’s context and an executor does not. Schedule it when the runbook is written and again after any significant change to the system, and record the date it was last executed in the document - a runbook last rehearsed two years ago should be read with suspicion, and the reader deserves to know that.

Knowledge check