The disaster recovery drill — the quarterly rehearsal

A DR plan that has never been executed is a hypothesis. The drill is the proof that the runbook works. The drill converts a document into a tested capability.

flowchart LR
    A["quarterly DR drill"] --> B["tabletop exercise"]
    A --> C["live regional failover"]
    B --> D["runbook reviewed"]
    B --> E["gaps identified"]
    C --> F["timeline recorded"]
    C --> G["RTO measured"]
    C --> H["RPO measured"]
    D --> I["updated runbook"]
    E --> I
    F --> I
    G --> I
    H --> I

The drill has two shapes: the tabletop exercise, which is a discussion of the runbook, and the live regional failover, which is an actual recovery in a non-production region. The two shapes are complementary: the tabletop catches gaps in the runbook text; the live failover catches gaps in execution.

The tabletop exercise

The tabletop is a meeting. The on-call engineer, the incident commander, and a representative from each team that owns part of the CI/CD path sit in a room (or a video call) and walk through the runbook against a scripted scenario.

The scenario names a disaster (regional outage, credential compromise, registry corruption) and the team walks through the runbook step by step. For each step, the team states who acts and what they act on.

The live regional failover

The live drill is a real recovery in a non-production region:

1. Provision a non-production region that mirrors the production region.
2. Simulate the disaster by deleting the non-production CI/CD infrastructure.
3. Execute the recovery following the runbook.
4. Measure the timeline — every step starts and ends at a known time.
5. Measure the RPO — the gap between the last replicated state and the disaster time.

The live drill is destructive of the non-production environment; the team rebuilds the environment after the drill. A drill that is not destructive has not tested the recovery.

RTO and RPO measurements

The two numbers are the Recovery Time Objective (how long the recovery took) and the Recovery Point Objective (how much data was lost).

The RTO measurement:

date -u +%s > /tmp/recovery_start.txt
# ... recovery steps ...
date -u +%s > /tmp/recovery_end.txt

The RPO measurement is the gap between the last replicated state and the simulated disaster time. A registry replicated every 15 minutes has an RPO of up to 15 minutes; a database replicated synchronously has an RPO of seconds.

The four artefacts

The drill must produce four artefacts:

1. The timeline. Every step with its start time, end time, and the engineer who executed it.
2. The RTO and RPO measurements. Compared against the design constraints. A drill that exceeds the designed RTO has surfaced a gap.
3. The gap list. Every step that failed or was unclear. Drives the next iteration of the runbook.
4. The updated runbook. The runbook after the gap list has been addressed.

Cadence and scope

The drill runs quarterly. The cadence is the minimum that keeps the runbook fresh; a drill that runs once a year is one the team has forgotten.

The scope is every layer: control plane, runners, secret store, artifact registry. A drill that covers one layer has not tested the dependency order.

Production discipline

1. Quarterly cadence. The drill runs every quarter; the runbook stays fresh.
2. Live failover in a non-production region. The runbook is tested against real infrastructure.
3. RTO and RPO are measured, not assumed. The measurement is the artefact that catches the gap.
4. The gap list drives the next runbook. The drill starts an improvement cycle, not a one-off event.

Cross-course references

• Git, CI/CD & GitOps — Parts XCVII-02 through XCVII-05 cover the four layers the drill exercises.
• Terraform for Production Sysadmins — Part XXI (State Backup) covers the state backup.
• Linux for Production Sysadmins — Part XII (RepoSecurity) covers the backup retention.

Quiz