Running DR drills that find something

Advanced⏱ ~18 mincephrbd

What you'll learn

Design a drill with a real objective
Run it without causing an incident
Capture the findings honestly
Escalate drill scope over time

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A drill that confirms everything works has usually been designed to confirm everything works.

Designing with a real objective

A useful drill has a question it might answer "no" to.

Weak objective	Strong objective
“Verify backups exist”	“Restore image X and start the application against it”
“Confirm the plan is current”	“Have someone who did not write it follow it”
“Test failover”	“Fail over with the primary genuinely powered off”
“Check monitoring works”	“Silence the alerting and see how long detection takes”

The distinguishing property: the drill can fail, and failing is the
useful outcome.

# a restore drill with a real objective
# Substitute your own pool before running:
POOL=rbd-vms

rbd import /backups/app-db.raw "$POOL/drill-restore"
rbd map "$POOL/drill-restore"
mount "/dev/rbd/$POOL/drill-restore" /mnt/drill
# start the application against /mnt/drill and confirm it works

Running without causing an incident

Technique	Effect
Restore to a new name, never over production	no risk to live data
Use a non-production cluster for destructive drills	full realism, no exposure
Announce the drill window	avoids a real incident response
Have an abort condition and someone empowered to call it	bounded blast radius
Drill one scenario at a time	findings are attributable

# safe: a restore drill on production storage, into a scratch name
# Substitute your own pool before running:
POOL=rbd-vms
DRILL_IMAGE="drill-$(date +%Y%m%d)"

rbd import /backups/app-db.raw "$POOL/$DRILL_IMAGE"
# ... verify ...
rbd rm "$POOL/$DRILL_IMAGE"

Drills that touch production should be additive. Drills that must be
destructive belong on a cluster that can be rebuilt.

# announce, so a real response is not triggered
echo "DRILL: restore exercise 14:00-16:00, no production impact expected"

Capturing findings honestly

Record for each drill:
  the objective
  what actually happened, including the parts that went badly
  every step that was wrong, missing, or slower than expected
  the measured times
  what will change as a result

Finding type	Value
A step that did not work	the highest value; fix the plan
A step nobody could perform	a knowledge or access gap
A duration far above the target	the commitment is wrong
A missing credential or contact	trivial to fix, blocks everything
“Everything went fine”	check whether the drill was hard enough

A drill with no findings is a signal about the drill, not about
readiness.

Escalating scope

Progression across drills:
  1. restore one image; verify it mounts
  2. restore and start the application against it
  3. have someone unfamiliar with the system follow the plan
  4. run it without the usual tooling available
  5. run it unannounced, during business hours
  6. fail over a real workload and run on the secondary

Each step is uncomfortable and each finds things the previous did not.
Stopping at step 1 and repeating it annually finds nothing after the
first time.

Quiz

Knowledge check · 4 questions

Q1. Why is having an unfamiliar person follow the plan the highest-yield drill?

They work more carefully than the author does, because everything is unfamiliar and no step can be skipped from memory The author supplies missing context automatically; an unfamiliar person stops exactly where context was assumed, which is where the plan fails at 03:00 They are more likely to make mistakes, and each mistake marks a step where taking the wrong action was easy to do It tests the training programme as well as the plan, so a single exercise produces findings about both the documentation and the people who will have to use it
Q2. A drill where everything worked is evidence of readiness.

True False
Q3. Design a DR drill programme.

A team runs an annual drill consisting of confirming backup files exist on the backup host. It has passed for three years.
Q4. What keeps a drill from causing a real incident?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Design each drill around a question it could answer “no” to, and escalate scope every cycle — a repeated passing check stops finding anything after the first run. Have someone who did not write the plan follow it; the gaps are in the plan more often than in the cluster.

Cross-course references

Kubernetes: chaos experiments are valuable exactly when they can fail
Linux: an author cannot test their own runbook for missing context