Writing a DR plan people can actually execute

Why this matters in production

Most DR plans are written to satisfy a requirement and are unusable by the person holding a pager at 03:00.

Structure by scenario

For each scenario in the model:
  how to recognise it
  what it means for data
  what to do, as ordered steps
  how to verify each step worked
  what to tell whom
  when to escalate and to whom

A plan organised by scenario is navigable. A plan organised by
component is a reference document, and nobody reads a reference document
during an incident.

Steps that work under pressure

# a step written usefully
# Step 4: confirm monitor quorum
#   Run:      ceph quorum_status --format json | python3 -c #               'import sys,json; d=json.load(sys.stdin); print(d["quorum_names"])'
#   Expect:   at least 2 of 3 monitor names listed
#   If fewer: STOP. Go to scenario "Lost monitor quorum".

"Verify the cluster is healthy" is not a step. "Run ceph -s; expect
HEALTH_OK; if HEALTH_WARN, record the warning and continue; if
HEALTH_ERR, stop and escalate" is a step.

Non-technical elements

Frequently missing and frequently needed:
  who has authority to declare a disaster
  who can authorise a destructive action
  where the credentials are, and how to get them
  who to contact at the hardware vendor and the facility
  where the backups physically are
  what to tell users, and who says it
  how to work if the usual chat and ticketing systems are down

# credentials access is the one that stops recoveries
# document: which secret manager, which break-glass procedure,
#           who holds the second factor

A plan that assumes the wiki, the chat system, and the ticket system are
available has assumed away the scenarios where they are not.

Keeping it current

# the plan should reference commands that still exist
grep -oE 'ceph [a-z-]+ [a-z-]+' /docs/dr-plan.md | sort -u | \
while read -r c; do
  $c --help >/dev/null 2>&1 || echo "check: $c"
done

Quiz

Production discipline

Organise the plan by scenario and write literal commands with expected output — a reader woken by a page has a symptom, not a diagnosis. Keep an offline copy; the scenarios covered include ones where the wiki runs on the infrastructure being recovered.

Cross-course references

• Kubernetes: runbooks stored in the cluster they recover are unavailable when needed
• Linux: a plan is usable to the extent it starts from a symptom