DR test, exercise, and continuous improvement · Proxmox VE

Why this matters in production

A DR plan that has not been exercised is a hope, not a plan. This lesson teaches the exercise cadence and the continuous improvement loop.

Exercise types

Type	Cost	Risk	Value
Tabletop	Low	None	Identifies procedural gaps
Simulation (partial)	Medium	Low	Validates specific procedures
Live drill (full)	High	High (if real systems)	True confidence

Tabletop

The team sits around a table. The facilitator presents a scenario. The team walks through the runbook step by step. Identifies gaps, missing information, ambiguities.

Cost: 1-2 hours of team time. No production impact.

Simulation

Some steps are actually performed. Examples:

Restore a VM from PBS to the DR site.
Update DNS records (then revert).
Failover the database (then revert).

Cost: hours to a day. Limited production impact.

Live drill

Full activation of DR. Production traffic redirected to DR for the duration.

Cost: significant. Some risk of disruption. Reserved for the most critical exercises.

What each type can and cannot find

The cadence advice is easy to follow and easy to follow uselessly. Four quarterly tabletops that never touch a system will not find a single one of the failures in the second column.

Failure	Tabletop	Simulation	Live drill
Nobody knows who declares a disaster	✔	✔	✔
The runbook references a wiki hosted at the primary site	✔	✔	✔
Recovery order has a missing dependency	✔ often	✔	✔
Restore credentials have expired	✘	✔	✔
The PBS fingerprint in the document is wrong	✘	✔	✔
Real restore throughput is half the assumed figure	✘	✔	✔
The DR site cannot resolve names without the primary	✘	✔	✔
A bridge is missing on the DR nodes	✘	✔	✔
Encryption keys are not actually escrowed	✘	✔	✔
The application does not work when its dependencies are at DR	✘	partly	✔
DNS cutover takes longer than the TTL suggests	✘	✘	✔
Capacity at DR is insufficient under real load	✘	✘	✔

The column boundary is the point: a tabletop tests the plan; a simulation tests the systems; a live drill tests the assumptions. An organisation doing only tabletops has verified that its document is coherent, which is worth having and is not the same as being able to recover.

The cheapest useful simulation is one restore. Pick a guest, restore it from the offsite datastore to a scratch VMID at the DR site using only DR-side credentials and the escrowed key, time it, and destroy it. It takes an hour and it exercises seven of the rows above.

Service impact possiblethe one-hour simulation that finds most of the real gaps— Restores one guest at the DR site into a scratch VMID on an isolated bridge. Uses only DR-side credentials and the escrowed key, deliberately - using production credentials tests nothing.

set -euo pipefail
DRILL=9500
SNAP='pbs-dr:backup/vm/101/2026-08-12T02:00:00Z'
START=$(date +%s)

# Deliberately using DR-side credentials only.
pvesm status --storage pbs-dr

qmrestore "$SNAP" "$DRILL" --storage local-zfs --unique 1
qm set "$DRILL" --net0 "virtio,bridge=vmbr-isolated"
qm start "$DRILL"

# Wait for the guest agent, which proves the OS is actually up.
for _ in $(seq 1 60); do
qm agent "$DRILL" ping >/dev/null 2>&1 && break
sleep 5
done

END=$(date +%s)
printf 'restore-to-responsive: %d seconds\n' "$(( END - START ))"

qm stop "$DRILL"
qm destroy "$DRILL" --purge 1

Write that number down. It is the only honest input to the RTO arithmetic in RPO, RTO and dependency modelling, and it is the number the person who wrote your SLA does not have.

Exercise cadence

Exercise	Cadence	Audience
Tabletop	Quarterly	All responders
Simulation	Monthly or quarterly	Specific teams
Live drill	Annually	Full team + leadership

Measuring DR readiness

Track these metrics:

RTO actual: how long did recovery actually take?
RPO actual: how much data was actually lost?
Tabletop completion rate: % of planned tabletops that happened.
Runbook freshness: when was the runbook last updated?
Exercise outcomes: pass/fail counts and gap inventory.

Why 'we passed the drill' is a weaker statement than it sounds

An exercise measures the plan against a scenario, and the scenario was chosen by the people who wrote the plan. That is not dishonesty; it is an unavoidable property, and it means a passed drill has a specific and limited meaning.

Three ways a drill passes without telling you much:

The scenario was the one you designed for. “The primary site is gone” is the case the plan was written against, so of course it survives it. The cases that break plans are the awkward ones: a partial failure where half the services are still up, a failure during a maintenance window when things are already moved, a failure that also takes out the network path to the DR site.

The people were the ones who wrote it. The plan’s authors carry a large amount of undocumented context, and they supply it without noticing. The same plan handed to the actual on-call rotation reveals which steps only worked because the author was in the room.

The systems were healthy going in. A drill on a well-patched, fully operational estate is not a drill on the estate as it is at 03:00 on a Sunday during a partial storage degradation.

None of that makes exercises less valuable - it makes the design of the exercise the thing to invest in. Three cheap ways to make a drill sharper:

Rotate the scenario, and include partial failures. They are more common than total ones and they are much harder, because the first job is deciding whether to declare at all.
Let someone who did not write the plan execute it, with the authors present but silent. Every question they have to ask out loud is a gap in the document.
Remove one thing deliberately. The person who normally does the restores is unavailable. The wiki is unreachable. The primary site’s DNS is gone. Each removal tests a dependency the plan does not admit to having.

Continuous improvement

After every exercise or incident:

Document what happened.
Identify gaps.
Update the runbook.
Track remediation (who, when, what).
Verify in the next exercise.

flowchart LR
  A[Exercise] --> B[Document]
  B --> C[Identify gaps]
  C --> D[Update runbook]
  D --> E[Track remediation]
  E --> F[Next exercise]
  F --> A

Production considerations

Common mistakes

Treating a single successful drill as proof of readiness.
Not following up on identified gaps.
Letting the runbook go stale between exercises.

Key takeaways

Tabletops quarterly; live drills annually.
Measure RTO/RPO actuals vs targets.
Improve continuously.

Knowledge check

Why this matters

Exercising is the only part of disaster recovery that produces evidence. Everything else - the architecture, the runbook, the replication schedule - is a hypothesis about what will happen, and hypotheses in this field have an unusually poor track record.

What makes exercises hard to sustain is not cost. A quarterly tabletop is two hours and a quarterly single-guest restore is one. It is that they produce findings, findings become work, and the work competes with things that have customers waiting. So the exercise gets deferred, and the plan quietly reverts to being a document rather than a capability.

The countermeasure is to make the readiness visible in the same place as everything else that gets funded: a short table with a row per failure, the mechanism that covers it, and the date it was last tested. Empty cells in that last column are the entire output, and they are far more persuasive to whoever holds the budget than any description of the architecture. A row that has never been tested is not a plan - it is an intention with infrastructure attached.