Operational runbooks and incident management · Proxmox VE

Why this matters in production

Incidents will happen. How you respond and what you learn determines whether the platform improves over time.

Runbooks

A runbook documents a procedure. Good runbooks:

Short — readable in 5 minutes.
Specific — exact commands, not “restart the service.”
Decision points — what to do if X happens vs Y.
Reversible — how to undo if the fix makes things worse.
Off-site accessible — printed or in a system that survives the incident.

Runbook template

# Title

Risk: low / medium / high / critical
Impact: informational / service-affecting / cluster-affecting / data-loss-risk

## Prerequisites

- [ ] Tool X is installed
- [ ] User has admin role
- [ ] Backup of Y exists

## Pre-checks

- Run command A; verify result is X.
- Run command B; verify result is Y.

## Procedure

1. Step one: command or action.
2. Verify: expected result.
3. Step two: command or action.
4. Verify: expected result.

## Verification

- [ ] Symptom resolved.
- [ ] No new symptoms.
- [ ] Monitoring back to normal.

## Rollback

If something goes wrong:

1. Action to undo step N.
2. Verify.

## Escalation

If X condition is met, escalate to Y:
- IC: incident commander
- Slack: #incident
- On-call rotation

A runbook that lives on the cluster is not available during the incident it is for

This is the property that separates a runbook from a wiki page, and it is tested exactly once.

The quorum-recovery runbook, the cold-start procedure, the network-lockout recovery — every one of them addresses a situation in which the cluster may be inaccessible. If they live in a Confluence instance running as a guest on that cluster, or in a Git repository behind an SSO provider whose directory server is a VM on it, they are unavailable at precisely the moment they exist for.

The dependency chain is longer than people expect. A runbook is unavailable if the cluster is down, if DNS is down, if the identity provider is down, if the VPN concentrator is a guest, or if the only copy is on the laptop of somebody who is on a flight.

The test is one question: can the person on call read this procedure with the cluster switched off and no corporate network?

Practical answers, in rough order of how often they save someone:

A copy on the on-call engineer’s laptop, synchronised automatically, in a format readable offline.
A printed folder in the rack room for the handful of genuinely cluster-down procedures: cold start, quorum recovery, network lockout, restoring /etc/pve.
Hosting the documentation somewhere with no dependency on the estate it documents — a different provider, a different account.

The set that needs this treatment is small. Most runbooks describe things you do to a working cluster and can live wherever your documentation lives. Sort them once, mark the ones that must survive, and treat those differently.

What makes a runbook still correct in a year

Runbooks rot, and they rot in a specific way: the commands stay valid and the context stops being true. The storage was renamed. The node count changed. The GUI path moved. The person who is escalated to has left.

Three properties make a runbook resistant to that, and they are all about what you leave out.

Name things by role, not by identity. “Migrate the guests to another node” survives a cluster growing from three nodes to five. “Migrate the guests to pve-02” does not, and it fails silently — an operator will do it, onto a node that is now the wrong choice.

Verify by observation, not by assumption. A step that says run this command has no way to detect that the world changed. A step that says run this command and confirm the output contains X fails loudly the first time it is wrong, during a drill rather than during an incident. Every step in a runbook should have a stated expected result for exactly this reason.

Link, do not copy. A runbook that reproduces the contents of the storage configuration, the node list or the escalation rota has created a second copy that will diverge. Reference the authoritative source — the inventory, the rota — and the runbook stays true when that source changes.

The mechanism that actually keeps them honest, though, is not a writing technique. It is that the runbook is the thing you follow during the drill, and any deviation is a defect in the runbook rather than in the operator. A team that improvises during a drill and then does not update the document has learned nothing durable. A team that treats every improvisation as a bug report ends up with procedures that work.

Incident management framework

When an incident is declared:

Detect: alert or user report.
Triage: severity, scope, initial diagnosis.
Mitigate: stop the bleeding; restore service.
Resolve: fix the root cause.
Document: timeline, decisions, customer impact.
Review: post-incident review (PIR).

Separate the roles, even in a team of three

One person doing all of it is the default and it is where most incidents go wrong. The failure is not competence; it is that the jobs interrupt each other. Somebody typing a recovery command and simultaneously answering “any update?” every four minutes will do both badly, and the mistakes land in the one that matters.

Role	Owns	Does not do
Incident commander	Decisions, priorities, who does what, when to escalate	Type commands
Operations lead	Actually performing the technical work	Talk to stakeholders
Communications	Stakeholder updates on a fixed cadence, the timeline	Make technical decisions

On a small team these collapse, and the collapse that works is commander plus operations. The commander takes communications as well; what must not happen is the person at the keyboard also being the person answering the phone.

The single most valuable thing a commander does is not technical. It is saying “stop, what do we actually know?” twenty minutes in, when three people are pursuing three theories and nobody has written down which observations are facts.

Mitigate before you diagnose, with two Proxmox exceptions

The instinct of a good engineer is to understand the problem before acting. During an outage that instinct is expensive: users are affected while you learn, and root cause is usually available afterwards from logs that persist.

So the default order is restore service, then investigate. Fail over, restart, roll back, restore from backup — whatever ends the user impact — and keep the evidence.

Two Proxmox situations invert this, and both are worth knowing before you are in them:

Quorum loss. Acting before diagnosing is exactly the failure mode: the fast mitigation is forcing quorum, and forcing quorum on a partition whose peers are alive is how a frozen cluster becomes a corrupted one. Here you must establish whether the missing nodes are dead before touching anything, and the guests are still running while you do.
Suspected data corruption or a security event. Mitigating first can destroy the evidence you need, and in a ransomware case restoring from a compromised backup reinfects. Capture state before changing it.

Everything else — a failed node, a full datastore, a hung guest, a broken storage path — is mitigate-first.

Write the timeline as it happens, not afterwards

The timeline is the single most useful artefact an incident produces, and it cannot be reconstructed accurately from memory. People misremember order, compress duration, and forget the thing they tried that did not work — which is frequently the most informative entry.

The mechanism that works is a channel where every action is stated as it is taken, with a timestamp:

02:41 - pvecm status on pve-01: Quorate No, 2 of 5 votes 02:44 - iDRAC on pve-03/04/05 reachable, all three powered off 02:47 - PDU B shows no output; escalated to facilities 03:12 - facilities restored PDU B 03:19 - all five nodes present, quorate, guests starting

Five lines, written as they happened, and the post-incident review has everything it needs. Without them somebody spends a morning reconstructing approximately this from three people’s recollections and a partial log.

It also does the communications job for free: a stakeholder reading that channel does not need to ask for an update.

Post-incident review

After resolution:

Schedule the PIR within 7 days.
Attendees: responders, leadership, representatives of affected teams.
Agenda: timeline, contributing factors, what worked, what to change.
Output: action items with owners.

Production considerations

What happens when post-incident action items are never completed?

The review keeps happening and stops mattering, and the team’s confidence in its own improvement process quietly inverts into cynicism.

The mechanism is ordinary. A review produces eight action items. Three are done in the following fortnight because they are small. Four require somebody’s time against competing priorities. One requires money. Nobody tracks them, the next incident arrives, and its review produces eight more.

Six months later there are forty open action items, several of which would have prevented incidents that have since recurred, and the team has learned that writing action items is a ritual rather than a mechanism. Attendance drops. The reviews get shorter. The document gets written because it is expected.

Three practices prevent this, and the third is the one that actually works:

Every action item has an owner and a date, and neither is “the team” or “next quarter”.
The first agenda item of every review is the previous review’s action items. Reading out four items from three months ago that nobody has started is uncomfortable exactly once, which is the point.
Cap the list. A review that produces more than three or four items has not prioritised; it has transcribed. Ask which single change would most reduce the chance or the cost of a recurrence, do that one properly, and let the rest be a backlog rather than a commitment.

The related discipline is honesty about what an action item is. “Improve monitoring” is not one. “Add a check that alerts when Ceph maintenance flags have been set for more than four hours, owned by Sam, by the 30th” is.

Common mistakes

A runbook stored where the incident makes it unreachable.
Runbooks naming specific nodes rather than roles, so they silently become wrong as the cluster changes.
Steps with no expected result, which cannot detect that the world changed.
One person commanding, fixing and communicating simultaneously.
Diagnosing before mitigating for ordinary failures — and mitigating before diagnosing during quorum loss or a suspected security event.
Reconstructing the timeline afterwards, which loses the attempts that did not work.
Blame-focused reviews.
“We’ll do the review later”, which never happens.
Action items with no owner, no date and no follow-up.

Key takeaways

Runbooks: short, specific, verified by observation, and available with the cluster switched off.
Name roles, not nodes; state an expected result for every step; link rather than copy.
The drill follows the runbook, and every deviation is a defect in the document.
Separate command from operations; on a small team the commander also takes communications, but never the person at the keyboard.
Mitigate before diagnosing, except during quorum loss and suspected corruption or compromise.
Write the timeline live; it is the most useful artefact an incident produces and cannot be reconstructed.
Reviews are blameless, produce three or four owned items, and begin by reading out the previous set.

Knowledge check

Why this matters

Incidents are the moments when everything else in this part is tested at once: the inventory that says who owns the guest, the monitoring that decided whether anybody was woken, the change record that says what happened yesterday, the runbook that says what to do.

What this lesson adds is the observation that most of the avoidable cost in an incident is organisational rather than technical. The recovery command was usually known. What took the extra ninety minutes was that the runbook lived somewhere unreachable, or one person was trying to fix and explain at the same time, or nobody wrote down what had already been ruled out, or the action item that would have prevented this exact recurrence was written four months ago and never assigned.

None of that requires deeper Proxmox knowledge. It requires a document that survives the outage, two people with distinct jobs, a channel with timestamps in it, and a review short enough that its outcomes actually happen.