Periodic DR Testing

A disaster recovery runbook that has not been exercised is a hypothesis. The test exists to convert it from a hypothesis into evidence. For a Terraform estate, the test has two shapes - a tabletop review of the runbook with the on-call rotation, and an actual restore exercise against a sandbox account - and the cadence has to catch the runbook before the disaster does.

This lesson is the second-to-last in the chapter. It requires a runbook to test; the lesson before this one (the restore-the-estate lesson) is where the runbook comes from.

Why testing is a discipline, not an event

Three reasons the test is not optional:

• Procedures rot. The state backend bucket name changes. The lock table is migrated. The provider authentication pivots from access keys to OIDC. The runbook that was correct in February is wrong in August.
• People rotate. The on-call engineer who last read the runbook is on holiday or has left the team. The next disaster is run by someone whose training is “read the doc”. Reading the doc under stress is not the same as having run it before.
• Controls silently break. Versioning on the state bucket gets disabled during a bucket migration. CRR replication lag spikes. Locks fail. Without a test, the team finds out the control is broken at exactly the wrong time.

The right cadence is quarterly, with two more-frequent shapes: a tabletop after every schema change (new provider, new backend), and a sandbox restore after every backend migration.

The two shapes of test

Tabletop

The on-call rotation walks through the runbook. No real resources are touched. The objective is to confirm that the runbook is readable, complete, and understood by more than one person.

   Run a tabletop with the on-call rotation
        |
        v
   +----+----+----+----+
   |    |    |    |    |
   v    v    v    v    v
   1    2    3    4    5

   1. Read the runbook aloud
   2. Pick a disaster scenario (region down)
   3. Walk through the steps with realistic input
   4. Time the exercise; record deviations
   5. Hand over findings to the runbook owner

A one-hour tabletop reveals about 80% of the runbook rot: missing commands, missing links, missing credentials, unmentioned dependencies.

Actual restore in a sandbox

The runbook is exercised against a clone of the production estate. The production estate is untouched. The sandbox estate is built once and re-used for every quarterly exercise.

   Sandbox = production-like clone in a separate account
        |
        v
   1. Disable the sandbox state backend (simulate disaster)
        |
        v
   2. Run the runbook against the sandbox
        |
        v
   3. Time the actual recovery; record RTO achieved
        |
        v
   4. Compare actual vs target RPO/RTO
        |
        v
   5. Sign off; update the runbook if it failed

The sandbox exercise takes about half a day the first time, then roughly two hours per quarter. The cost of the sandbox estate is small relative to the cost of an unrecovered disaster.

Sandbox design

A good sandbox is not the production estate. It is a production-like estate that has the same backend topology (the same versioning, the same CRR or equivalent), the same number of state files, the same lock table, and the same CI integration. The complexity must be representative. Five resources in a single state file is not a useful sandbox.

The minimum sandbox:

• A separate account/subscription/project. The sandbox is firewalled off from production. A disaster recovery exercise that accidentally touches production is a bigger disaster than the one it is rehearsing.
• The same backend topology. S3 with versioning and CRR; or GCS with object versioning; or Azure Blob with immutable blob policy. The backend is set up in the sandbox just as it is in production.
• At least one lock table. DynamoDB, a PostgreSQL table, or the cloud equivalent. Lock-table behaviour is tested by the exercise.
• At least three workspaces. Single-workspace sandboxes miss the multi-workspace recovery shape.
• A representative state size. A 50 KiB state file recovers differently from a 50 MiB state file. The sandbox should match the median.

Building the sandbox is itself an exercise. The first quarterly test is the one that builds the sandbox; the subsequent tests re-use it.

The exercise procedure

   Set calendar hold:    4 hours, on a Tuesday afternoon
   Invite:               on-call rotation, runbook owner,
                         independent observer (sign-off)
   Tooling:              runbook, sandbox account,
                         communication channel for the
                         team to simulate "production is down"
        |
        v
   1. Brief: "production region us-east-1 has lost the
      state backend. You have 30 minutes to start the
      recovery; the rest goes into timing."
        |
        v
   2. Responder reads the runbook, follows it.
        |
        v
   3. Observer records: time to each step, deviations,
      errors, missing steps, confusing commands.
        |
        v
   4. End: ask "is the estate recovered?" If the
      refresh-only plan is clean, yes.
        |
        v
   5. Debrief, write findings, schedule fixes

The observer’s notes are the primary artefact of the exercise. They capture the small things - a link that 404’d, an IAM role that was renamed, a sandbox-only command that does not work in production - that the written runbook missed.

What the exercise catches

Real findings, abbreviated from actual exercises:

• The state bucket ARN was correct in the runbook; the bucket name changed during a refactor six months ago. The runbook referenced a bucket that no longer existed.
• The Terraform binary in the runner container was 1.5 but the state format expected 1.7. The apply failed with an unhelpful “state format mismatch”.
• The lock table was in the failover region but the lock read IAM was scoped to the primary region. The recovery-applied lock acquisition failed silently.
• The OIDC trust policy referenced a repo that had been renamed. The runner could not authenticate. The runbook had a note “use access key from vault” - but the access key was no longer issued.
• The cross-region replication target had drifted; the replicated state version was 47 minutes old. The recorded state RPO of 60 seconds was not met.

Every finding is a fix. A test that finds nothing is suspicious; the runbook probably has not changed in months.

Cadence

The right cadence is a function of how fast the runbook goes stale:

• Quarterly tabletop. Always. The cost is one afternoon per quarter; the benefit is catching procedural rot before the disaster.
• Quarterly sandbox restore. The first quarter builds the sandbox; subsequent quarters reuse it. Two hours per quarter.
• After every backend migration. A versioned bucket move, a lock-table migration, a backend type change. These are the events that invalidate the runbook.
• After every schema change. A new provider, a new Terraform/OpenTofu major version. Run the sandbox exercise before the change ships; the change ships knowing the recovery still works.

The cadence is not “annual”. Annual is too slow. The disaster will not wait for the calendar.

Sign-off

The exercise is signed off by an owner who is not the runbook author. The sign-off says:

• The exercise was run.
• The sandbox was production-like.
• The recovery met (or did not meet) the target RPO and RTO.
• The findings are recorded with owners and dates.
• The next exercise is on the calendar.

A sign-off by the runbook author is not a sign-off. The separation of authorship from review is what makes the exercise credible to an auditor and to the team.

Production guidance

• The exercise calendar is owned. A quarterly slot is on the team calendar before the year starts. An exercise that needs a calendar slot next month will not happen.
• The findings have owners and dates. Findings from the exercise become tickets with owners and due dates. A finding without an owner does not get fixed.
• The sandbox is rebuilt annually at minimum. A sandbox that drifts from production is a sandbox that produces misleading evidence. Rebuild against the current production topology at least yearly.
• The runbook is in source control. The exercise produces patches to the runbook; those patches go through review. The runbook that lives on the wiki is the runbook that no one updates.
• The sign-off is filed. The sign-off artefact lives in the same source repository as the runbook, dated and signed. The auditor reads it.

Verification

# 1. Confirm a sandbox exists and is separated.
aws organizations list-accounts \
  --query 'Accounts[?Tags[?Key==`Purpose` && Value==`tf-dr-sandbox`]]'

# 2. Confirm the sandbox has a backend with versioning.
aws s3api get-bucket-versioning --bucket acme-tfstate-sandbox

# 3. Confirm the last exercise calendar slot.
grep -E 'dr-test|sandbox-restore' .calendar/quarterly.yaml

# 4. Confirm the last exercise findings are tracked as
#    tickets.
ls tickets/dr-findings/ 2>/dev/null | tail

# 5. Confirm the runbook is in source control (not the wiki).
test -f runbooks/terraform-dr-estate-recovery.md && \
  echo "runbook in source" || echo "runbook missing"

# 6. Confirm the sign-off artefact exists and is recent.
ls -la .dr/sandbox-restore-signoff-*.md 2>/dev/null | tail -1
# Expected: a file dated within the last quarter.

To confirm the lesson:

• You can describe the two shapes of test (tabletop and sandbox restore).
• You can list the four cadence triggers.
• You can refuse the runbook-by-itself posture; the test is part of the runbook.