Testing the Recovery Procedure

A recovery procedure that has not been tested is a procedure on paper. The first time it is exercised is during an incident, when the team is under pressure, the clock is running, and the documentation is wrong in three places. The DR drill moves that first exercise out of the incident and into a planned window where the gaps can be found and fixed without consequence.

Two cadences of testing

Two tests, two cadences:

Monthly: routine restore. Restore state from the daily pull into a staging environment. Verify with a plan. Document the time taken. This is the cheap, fast test that confirms the backups are usable.

Quarterly: full DR. Simulate a failure scenario end to end: regional outage, account compromise, or full state corruption. Exercise every backup layer; verify the runbook; measure the RTO. This is the expensive, slow test that confirms the architecture meets the targets.

The monthly test is automated; it runs without manual intervention. The quarterly test is a planned event with the team lead, the security auditor, and the on-call engineer.

The monthly routine restore

A scheduled job in a staging environment:

# GitHub Actions in the staging account
name: monthly-state-restore-test
on:
  schedule:
    - cron: '0 4 1 * *'   # 04:00 UTC on the first of the month
jobs:
  test-restore:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    steps:
      - name: Check out configuration
        uses: actions/checkout@v4

      - name: Assume role in the backup account
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::BACKUP-ACCOUNT:role/restore-tester
          aws-region: us-east-1

      - name: Download the most recent nightly pull
        run: |
          LATEST=$(aws s3 ls s3://state-backups-prod/nightly/ | sort | tail -1 | awk '{print $4}')
          aws s3 cp s3://state-backups-prod/nightly/$LATEST /tmp/state-test.json

      - name: Validate JSON
        run: |
          jq -e '.version, .serial, .lineage' /tmp/state-test.json
          jq -e '.resources | length' /tmp/state-test.json
          # Confirm the state is a valid Terraform state JSON.

      - name: Configure test backend
        run: |
          mkdir -p /tmp/restore-test
          cd /tmp/restore-test
          cat > backend_override.tf <<EOF
          terraform {
            backend "local" {
              path = "/tmp/state-test.json"
            }
          }
          EOF

      - name: terraform init
        run: |
          cd /tmp/restore-test
          terraform init -input=false

      - name: terraform plan (read-only)
        run: |
          cd /tmp/restore-test
          terraform plan -input=false -detailed-exitcode
          # Exit 0 = no changes, exit 1 = error, exit 2 = changes
          # Either is acceptable; the test passes if init and plan
          # both succeed and the JSON is valid.

      - name: Report results
        if: always()
        run: |
          echo "Test passed: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
          echo "Backup: $LATEST"
          # Send to the team channel or to a CI artefact

The job:

1. Downloads the most recent nightly pull.
2. Validates the JSON (version, serial, lineage, resource count).
3. Configures a local backend pointing at the downloaded state.
4. Runs terraform init and terraform plan.
5. Reports the result to the team channel.

A failure (the JSON is invalid, init errors, plan errors) is a P2 incident — the backup is broken.

The quarterly full DR

A planned event. The team picks a scenario:

Scenario A: Accidental state overwrite
   Simulate: a wrong state mv; restore from versioning
   Time:     < 30 minutes expected
   Verify:   plan is empty after restore

Scenario B: Regional outage
   Simulate: disable the primary bucket (or use a chaos engineering
             tool to simulate); restore from cross-region replica
   Time:     < 1 hour expected
   Verify:   applies succeed against the replica

Scenario C: Account compromise
   Simulate: revoke the source account; restore from daily pull
             in a new account
   Time:     < 4 hours expected
   Verify:   applies succeed; gap is reconciled against saved plans

Each scenario is a runbook. The drill executes the runbook:

1. Notify the team (DR drill in progress; do not be alarmed).
2. Execute the scenario (simulate the failure).
3. Run the runbook (restore from the appropriate layer).
4. Measure the time from scenario start to plan-empty.
5. Document the gaps (steps that took longer than expected, steps that did not work as written, IAM roles that did not exist).
6. Produce control changes (runbook updates, new IAM roles, automation).

The drill is reviewed in a post-drill document. The control changes are tracked; the next drill verifies the changes worked.

The drill report

A documented drill report:

# State Recovery Drill — 2026-08-13

## Scenario
   Account compromise simulated; production account isolated;
   recovery from daily pull to a new staging account.

## Timeline
   14:00 UTC — drill started
   14:05 — backup identified (most recent nightly pull)
   14:12 — backup downloaded to staging
   14:18 — staging backend configured
   14:22 — terraform init succeeded
   14:25 — terraform plan produced diff (3 resources)
   14:35 — diff reconciled against saved plans
   14:42 — apply completed; plan now empty
   14:45 — drill complete; total time 45 minutes

## RTO
   Actual:   45 minutes
   Target:   30 minutes
   Status:   MISSED (15 minutes over)

## Gaps found
   1. The IAM role for restore-tester did not exist in the new
      staging account; had to be created mid-drill.
   2. The KMS key for decrypting the backup was in the backup
      account, not the staging account; had to be granted
      cross-account access.
   3. The saved plan archive for the last 24 hours was in the
      production account; not accessible from the staging account.
      Reconciled manually from the change log.

## Control changes
   1. Pre-create the restore-tester IAM role in every staging
      account; cross-account trust from the backup account.
   2. Pre-grant the staging account access to the backup KMS key.
   3. Replicate the saved plan archive to the backup account.

## Sign-off
   Team lead:        ____________________
   Security auditor: ____________________
   On-call engineer: ____________________
   Date:             2026-08-13

The report is filed in the team’s runbook repository. The next drill (quarterly) verifies the control changes.

Automating the test

The monthly routine restore is fully automated. The CI job runs on the schedule; the result is posted to the team channel; a failure pages the on-call.

The quarterly DR is semi-automated. The scenario execution is manual (a human simulates the failure); the runbook execution is scripted where possible. The drill is led by the team lead or a delegate; the report is filed within 24 hours.

For teams that want full automation, chaos engineering tools (Chaos Toolkit, Gremlin, AWS Fault Injection Service) can simulate the failure scenario automatically. The drill becomes a continuous test.

Validation

READ-ONLY

# Confirm the monthly test has run recently
gh workflow list --all | grep monthly-state-restore-test

# Confirm the latest drill report exists
ls /path/to/runbook-reports/state-recovery/

# Confirm the control changes from the last drill are tracked
gh issue list --label state-recovery --state open

Production failure modes

Symptom: the monthly test fails because the backup JSON is invalid. Cause: the backup job produced a corrupt file (a partial write, a permissions issue). Recovery: investigate the backup job; rerun; verify the next monthly test passes.

Symptom: the quarterly drill exceeds the RTO target by a wide margin. Cause: the runbook has manual steps that take longer than expected, or the architecture has drifted. Recovery: identify the slow steps; automate; re-test.

Symptom: the drill surfaces a control change that is not implemented. Cause: the change was identified but not tracked. Recovery: open a ticket for the change; assign an owner; verify the change in the next drill.

Symptom: the team cannot run the drill because the production environment is too critical to simulate a failure. Cause: the team is unwilling to take the risk. Recovery: run the drill in a non-production environment that mirrors production; verify the runbook against the production IAM roles in a separate test.

Recovery

A failed DR test is itself a recovery event. The procedure:

1. Identify what failed (the backup, the restore procedure, the RTO).
2. Fix the immediate issue.
3. Open a ticket for the underlying control gap.
4. Re-run the test in the next cycle to confirm the fix.
5. Document the failure and the fix in the runbook.

What comes next

The next part of the course covers the broader operational discipline: change management, runbook maintenance, and the team structures that keep Terraform production-grade.

Verification

• You can describe the two cadences of state recovery testing (monthly routine, quarterly full DR).
• You can run a routine restore test against the daily pull and verify the result with terraform plan.
• You can write a DR drill report with timeline, RTO measurement, gaps, and control changes.
• You can automate the monthly test so it runs without manual coordination.