DR runbook exercise: restoring a fictional business · Proxmox VE

Why this matters in production

A runbook you have never executed will fail when you need it. This lesson walks through building and rehearsing a runbook for a realistic business.

The fictional business

“ExampleCorp Inc” — 150 employees, B2B SaaS, headquartered in Datacenter East.

Critical systems:

System	RPO	RTO	Notes
Customer-facing API	5 min	30 min	Fronted by load balancer
PostgreSQL primary	1 min	15 min	Streaming replication to secondary
Identity (AD + Keycloak)	15 min	30 min	Two AD DCs
File server (Windows)	1 h	4 h	NFS for some VMs
Internal wiki	4 h	8 h
Monitoring (Prometheus + Grafana)	1 h	1 h	Must come up early

Architecture

Primary site:

5-node Proxmox cluster.
Ceph RBD for VM storage.
PBS primary, syncing hourly to off-site PBS.

DR site:

3-node Proxmox cluster (cold standby).
PBS DR receiving hourly sync.
Public IPs allocated but not in DNS.

Runbook: declare disaster

00:00 - On-call engineer is paged by monitoring (or a customer).
00:05 - Engineer confirms the failure is real (not a false alarm).
00:15 - Engineer declares disaster. Calls incident commander.
00:30 - IC assembles response team. Decision: activate DR?
00:45 - Decision: yes. Activate DR.

The 45 minutes before any technology happens

Look at that timeline again. Nothing technical has occurred. Three quarters of an hour of the RTO has been spent on detection, confirmation, escalation and authorisation, and those minutes are as real as the transfer time - they just never appear in the architecture diagram.

They are also the part most estates have never designed, so in a real incident they take longer than this. The three questions that must have written answers before the incident:

Who can declare a disaster? Name roles, not people, and name at least three so a holiday does not block it. If the answer is “we’d escalate to management”, you have not answered it - you have deferred it to a phone tree at 03:00.

What is the threshold? “The primary site is unreachable and has been for more than N minutes” is a threshold. “It’s really bad” is not. A written threshold lets the on-call engineer act rather than seek permission, which is where most of the 45 minutes goes.

What is the cost of declaring wrongly? DR activation is expensive and usually somewhat destructive - DNS changes, restores over things, a split-brain risk if the primary returns. If nobody has priced a false positive, everybody hesitates, and hesitation is the single largest term in most real RTOs.

The technical runbook is the easy half. An organisation with excellent automation and no written declaration authority still takes an hour to start.

Runbook: activate DR

01:00 - Power on DR cluster nodes.
01:15 - Verify network connectivity to off-site PBS.
01:30 - Attach off-site PBS as PVE storage.
01:45 - Restore critical VMs in dependency order:
       - DNS (external resolver)
       - Identity (AD, Keycloak)
       - Database (PostgreSQL primary)
       - Application servers
       - Load balancers
       - Monitoring
02:30 - Update DNS records (TTL was 10 min).
02:40 - Verify end-to-end: customer can log in.
03:00 - All critical systems restored. RTO 3 h.

The activation steps, as commands rather than prose

A runbook step that says “attach the off-site PBS as PVE storage” is a paragraph the reader has to turn into commands at 01:30. Write the commands.

Configuration changeDR activation: attach the offsite datastore and confirm it is usable— Adds the DR PBS server as a storage on the DR cluster. The fingerprint must come from your printed documentation, not from the server you are connecting to.

set -euo pipefail
PBS_HOST=pbs-dr.example.com
FPRINT='REPLACE_WITH_PRINTED_FINGERPRINT'

# --password with no value makes pvesm prompt, so the secret is not in
# shell history or in this runbook.
pvesm add pbs pbs-dr \
--server "$PBS_HOST" \
--datastore dr-main \
--username 'restore@pbs' \
--password \
--fingerprint "$FPRINT"

# Prove it before anyone waits on a restore that cannot start.
pvesm status --storage pbs-dr
pvesm list pbs-dr --content backup | head -20

Data-loss riskDR activation: restore in dependency order— Restores guests into the DR cluster. Restore identity and databases before applications, and verify each tier before starting the next - a stalled application tier is much harder to diagnose than a stalled database.

set -euo pipefail
STORE=local-zfs

# Tier 1: identity and DNS. Nothing else works until these do.
for SNAP in \
'pbs-dr:backup/vm/101/2026-08-12T02:00:00Z' \
'pbs-dr:backup/vm/102/2026-08-12T02:05:00Z'
do
VMID=$(printf '%s' "$SNAP" | sed 's#.*/vm/\([0-9]*\)/.*#\1#')
qmrestore "$SNAP" "$VMID" --storage "$STORE"
qm start "$VMID"
done

# Verify tier 1 before moving on. This is the step that gets skipped.
getent hosts idp.example.com || echo 'TIER 1 NOT READY - STOP'

# Tier 2: databases. Then applications. Same shape, same verification.

The --fingerprint line is the one that earns the printed page. The fingerprint is verified when the storage is added, and taking it from the server you are connecting to defeats the check - so it has to come from documentation that survived the site, which means it must have been printed before the site was lost.

Runbook: communicate

Throughout: status updates to stakeholders every 30 min.
At declare: notify leadership, customer support, security team.
At decision: notify customers (status page).
At restore: notify customers (status page) of restored services.
At complete: post-incident review scheduled within 7 days.

Tabletop exercise

A tabletop exercise walks through the runbook without actually performing actions. The team discusses each step, identifies gaps, and refines.

flowchart LR
  A[Read scenario] --> B[Walk through steps]
  B --> C[Realistic?]
  C -->|yes| D[Refine]
  C -->|no| E[Identify gap]
  E --> D
  D --> F[Document]

Common gaps found in tabletops:

Credentials are not at the DR site.
DNS TTL is too long.
The DR cluster nodes cannot reach the off-site PBS without primary-site routing.
The team does not know who the incident commander is.

What happens if the primary site comes back mid-recovery?

This is the scenario tabletops almost never cover and it is the one most likely to turn a recovery into a data-loss event.

Two hours into the DR activation, the primary site’s power is restored. Its nodes boot. Its guests start. They still hold the DNS names, the same IP addresses on their own network, and - critically - they still have their disks, with data as of the moment of the outage.

You now have two live copies of every service. Both are writing. Whichever one receives a given request depends on DNS caching and routing, so writes are being distributed between two divergent copies of the truth, and no automated mechanism will reconcile them afterwards.

What the runbook must contain, decided in advance:

Someone owns “the primary must not come back on its own”. Whether that is a documented instruction to facilities, a network isolation step, or disabled auto-start on the guests, the decision is made before the incident, not shouted into a phone during one.
An isolation step in the activation procedure. If the primary is reachable at all when you activate, isolate it at the network before you start restoring. Cheap, reversible, and it removes the entire scenario.
A written failback procedure that goes DR to primary, never both ways at once. Failback is a second planned outage, with its own window and its own verification. Treating it as “just switch DNS back” is how the split happens in the other direction.
A rule about which copy wins. After a split, one copy is authoritative and the other is evidence. Deciding which during the incident, with two teams each attached to their own copy, does not go well.

The uncomfortable version of this question is worth asking in the tabletop directly: if the primary came back right now, what stops it serving traffic? If the answer is “nothing automatic”, that is the finding.

Post-incident review

After every real incident or drill:

What worked?
What didn’t?
What surprised us?
What should change?

Document the review and update the runbook.

Production considerations

Common mistakes

Long DNS TTLs.
DR site without credentials.
“We’ll figure it out when it happens.”
Tabletop exercises that don’t include the people who will execute.

Key takeaways

Per-system RPO/RTO.
DR runbook must be short and rehearsed.
Tabletop quarterly, live drill annually.

Knowledge check

Why this matters

A DR runbook is a document with an unusual specification: it must be executable by a tired person, under time pressure, with several of their normal tools missing, possibly at a keyboard they do not usually sit at.

Almost every runbook failure traces to writing for the wrong reader. Prose instead of commands, because the author knew what “attach the off-site PBS” meant. A reference to a wiki page that was hosted in the destroyed site. An implicit assumption that whoever executes this has the authority to start.

Two habits fix most of it. Write the steps as commands with the placeholders already named, so the reader substitutes values rather than composing procedures. And walk the whole thing with the question “and how do you reach that?” at every step - which is what a tabletop is for, and why a tabletop finds more than a technical review does.

The other half is not technical at all. The declaration authority, the threshold, and the agreed cost of a false positive are the difference between a recovery that starts in five minutes and one that starts in forty-five, and they cost one page and one meeting.