Runbook maintenance: keep procedures accurate and used

A runbook is only useful if it reflects the current system. A runbook written two years ago for PVE 7 is misleading on PVE 9. A runbook never tested in practice is a guess. This lesson covers how to keep runbooks accurate, current, and actually used.

The format question

Runbooks can live in many places:

• Wiki (Confluence, Notion, MediaWiki): collaborative editing, comments, search. Good for narrative content.
• Repository (Markdown in git, /docs folder): version controlled, reviewable, but harder to search. Good for command sequences.
• Static site (this course’s /runbooks/ page): version controlled, renders nicely, easily linkable.
• Paper / PDF: nothing breaks it. Hard to update.

For a PVE cluster, the best fit is usually:

• Markdown in git for the canonical version (version controlled, reviewable, diff-able)
• Static site or wiki for human consumption (rendered, indexed)

Both copies are kept in sync via CI: push to git, CI builds the site.

Anatomy of a good runbook

A useful runbook has these sections:

1. Trigger conditions

When should someone use this runbook? Be specific:

## When to use this runbook

Use this runbook when:
- A VM is reported as unreachable AND pvecm status shows the node
  is "offline" in the cluster dashboard
- A VM failed to migrate AND `pvecm nodes` shows the target node
  missing
- A backup job has been failing for >1 hour

Do NOT use this runbook for:
- A single VM is slow (different runbook: VM performance triage)
- A network outage (different runbook: network incident)

Without trigger conditions, an on-call engineer has to guess which runbook applies.

2. Pre-checks (read-only)

Read-only commands to verify the runbook applies:

## Pre-checks

Before any state-changing step, confirm:

1. `pvecm status` reports the expected quorum
2. The failing node is actually offline: `ping "$NODE"` returns
   "Destination host unreachable"
3. The remaining nodes have capacity for the affected VMs:
   `pvesh get /cluster/resources --type node`
4. No active maintenance window is set

If any of these don\'t match, STOP and consult a different runbook.

Pre-checks catch the cases where the runbook doesn’t apply.

3. Procedure with verification

Each step in the procedure should have a verification:

## Procedure

### Step 1: Remove the failed node from corosync

NODE_NAME=pve-04 pvecm delnode “$NODE_NAME”

**Verify**: `pvecm status` now shows the remaining nodes with the
expected vote count.

### Step 2: Verify quorum

pvecm status

**Verify**: Quorum: Yes, expected_votes matches the new total.

### Step 3: Re-add the replacement node (see add-node-to-cluster runbook)

A step without verification means the engineer doesn’t know if the step worked.

4. Rollback

Every state-changing procedure has a rollback:

## Rollback

If verification fails at any step:

1. Re-add the failed node: `pvecm add "$PEER_IP"` from that node
2. Revert corosync.conf to the previous config_version
3. Restart corosync on every node
4. Verify quorum restored

5. Escalation

When to call for help:

## Escalation

Contact the cluster admin if:
- The failed node cannot be brought back after the procedure
- Quorum cannot be restored
- The replacement node fails to join the cluster
- Any "I don\'t know what to do next" moment

Escalation contact: cluster-admin@company.com / +1-555-CLUSTER

Testing runbooks

A runbook that hasn’t been tested is a guess. Test quarterly:

# Pick a non-critical scenario
# Schedule during low-traffic window
# Walk through the runbook step-by-step on the actual environment

# Verify each step\'s verification command works
# Time each step
# Note deviations from the runbook
# Update the runbook with corrections

The test produces a small report:

Runbook: replace-failed-disk.mdx
Date: 2024-01-15
Tester: alice
Outcome: PASS (with deviations)

Deviations:
- Step 4 "verify cluster status" — pvecm status didn\'t update
  because of corosync caching. Added "sleep 5; pvecm status".
- Pre-check "verify spare disk present" — assumed SAS, but our
  spares are SATA. Updated step 1 to mention both.
- Time: estimated 30 min, actual 45 min (mostly due to the
  resilver). Updated estimate.

Actions: 2 runbook corrections, 0 escalations.

Capturing lessons learned

After every incident:

1. What happened?
2. What did we do?
3. What worked? What didn't?
4. What runbook was used? Did it help or hinder?
5. What would we do differently next time?
6. What runbook changes does this suggest?

The post-incident review captures these answers. The runbook gets updated with the lessons.

Maintenance cadence

A runbook that hasn’t been reviewed in a year is suspect. A runbook that hasn’t been tested in a year is fiction.

Production considerations

• Runbook ownership. Each runbook has an owner (the team member most familiar with it). When they leave, the runbook gets reassigned.
• Accessibility. Runbooks must be reachable during an incident. If your wiki is down during the incident, the runbook doesn’t help. Mirror critical runbooks to multiple locations.
• Search. When the on-call is panicking, they search by symptom, not by runbook name. Tag runbooks with searchable keywords.

Common mistakes

• Out-of-date commands. A runbook that says qm start but the current version is qm start --state running is worse than no runbook.
• No pre-checks. A runbook that jumps straight to the procedure will be followed in the wrong situation.
• Untested procedures. A runbook that no one has ever run is a guess. Test in staging.
• No rollback. A runbook that doesn’t tell you how to undo itself forces improvisation under pressure.

Key takeaways

• Runbooks need trigger conditions, pre-checks, procedure, verification, and rollback.
• Test every runbook quarterly.
• Update after every incident with lessons learned.
• A runbook that hasn’t been reviewed in a year is stale.

Knowledge check