DR testing and game days — the validation cadence

DR testing is the validation cadence that turns a backup program into a recovery program. This lesson walks the testing cadence (weekly, monthly, quarterly, annually), the test scenarios, game day exercises, the post-test report, and the operational discipline.

The testing cadence

flowchart LR
    A[Weekly] --> B[Monthly]
    B --> C[Quarterly]
    C --> D[Annually]
    A --> A1[Single PVC restore]
    B --> B1[Namespace restore]
    C --> C1[Full cluster restore]
    D --> D1[Cross-region restore]

Each cadence catches different failure modes. Skipping any one leaves a gap.

The weekly test: single PVC restore

# 1. Create a sandbox namespace
kubectl create ns restore-test-weekly

# 2. Restore a single PVC into the sandbox
velero restore create weekly-test \
  --from-backup daily-full \
  --include-resources persistentvolumeclaims \
  --namespace-mappings prod-data:restore-test-weekly

# 3. Verify the PVC is bound
kubectl get pvc -n restore-test-weekly

# 4. Mount the PVC in a debug Pod and verify data
kubectl run debug --rm -it --image=postgres:16 \
  --overrides='{...}' -- bash
# Inside: psql -c "SELECT count(*) FROM orders;"

# 5. Cleanup
velero restore delete weekly-test
kubectl delete ns restore-test-weekly

The weekly test takes minutes. It catches CSI driver issues, Restic/Kopia daemon issues, and snapshot readability issues. It does not catch namespace-wide ordering issues.

The monthly test: namespace restore

# Restore a full namespace into sandbox
velero restore create monthly-test \
  --from-backup daily-full \
  --namespace-mappings prod-app:restore-test-monthly-app,prod-data:restore-test-monthly-data

# Wait for Completed
velero restore get monthly-test

# Verify all workloads are Ready
kubectl get pods -n restore-test-monthly-app
kubectl get pods -n restore-test-monthly-data

# Verify Services route
kubectl get svc -n restore-test-monthly-app

# Verify Ingress
curl https://restore-test-monthly-app.example.com/health

# Cleanup
velero restore delete monthly-test
kubectl delete ns restore-test-monthly-app,restore-test-monthly-data

The monthly test exercises dependency ordering (CRDs, StorageClasses, ConfigMaps) and catches issues the weekly test misses.

The quarterly test: full cluster restore

# 1. Provision a sandbox cluster (kubeadm or cloud-managed)
# 2. Install CNI, ingress, cert-manager, Velero
# 3. Restore from backup
velero restore create quarterly-test \
  --from-backup daily-full \
  --include-cluster-resources=true \
  --namespace-mappings prod-app:prod-app-restored,prod-data:prod-data-restored,...

# 4. Verify every namespace, every workload, every PVC
for ns in $(kubectl get ns -o name); do
  kubectl get all -n $ns
  kubectl get pvc -n $ns
done

# 5. Verify Ingress, DNS, TLS
curl https://prod-app-restored.example.com/health

# 6. Document gaps and follow up

The quarterly test takes hours. It is the only test that exercises the complete recovery chain.

The annual test: cross-region restore

# 1. Provision a fresh cluster in another region
# 2. Configure Velero with the source bucket's CRR
# 3. Restore from a recent backup
velero restore create annual-test \
  --from-backup daily-full \
  --storage-location dr-region-bsl \
  --include-cluster-resources=true

# 4. Validate
# 5. Document gaps

The annual test exercises cross-region CRR, network latency to the source bucket, and the team’s ability to operate under regional failure conditions.

Game days

A game day is a simulated disaster with the team:

sequenceDiagram
    participant Lead as DR Lead
    participant Team as On-call team
    participant Eng as Engineering
    participant Bus as Business

    Lead->>Team: 'Cluster is lost. Recover.'
    Team->>Eng: Follow the runbook
    Eng->>Lead: Report progress
    Lead->>Bus: Update stakeholders
    Note over Lead,Bus: Real-time exercise
    Team->>Lead: Recovery complete
    Lead->>Bus: Service restored, post-mortem scheduled

The game day:

1. The DR lead announces the disaster.
2. The on-call team executes the runbook under time pressure.
3. Engineering reports progress.
4. The DR lead updates stakeholders.
5. After recovery, a post-mortem is scheduled.

Game days catch the gaps the cadence tests miss:

• Communication breakdowns.
• Runbook gaps under pressure.
• Decision-making delays.
• Tool failures the runbook assumed worked.

The post-test report

Every test produces a report:

Test: Quarterly full cluster restore
Date: 2026-08-16
Cluster: prod-us-east-1
Backup: daily-full-20260816030000

Outcome: SUCCESS (with 3 gaps)

Time to recovery: 3h 47m
RTO target: 4h
Status: MEETS RTO

Gaps:
1. CRDs not in Git - had to manually apply 4 CRDs
2. Sealed-secrets controller not installed - secrets not restored
3. DNS update required manual runbook step (not automated)

Follow-up:
1. Add CRDs to Git repository
2. Add sealed-secrets to bootstrap manifests
3. Automate DNS update in DR runbook

Next test: 2026-11-16

The report is part of the runbook. The gaps are tracked to closure. The next test verifies the gaps were closed.

The operational failure modes

DR testing fails for predictable reasons:

• No sandbox cluster. The team has no cluster to restore into. Tests are skipped.
• Test scope too small. Weekly PVC restore is necessary but not sufficient. Skipping quarterly leaves gaps.
• No follow-up on gaps. The post-test report lists gaps; nobody closes them. The next test finds the same gaps.
• Game days announced. A “surprise” game day that the team knows about in advance does not exercise the communication and decision-making paths.
• Cost of testing. Annual cross-region tests are expensive. Operators skip to save money; the disaster exposes the gap.

Quiz

Production discipline

DR testing in production rests on five non-negotiable elements:

• Test every cadence. Weekly, monthly, quarterly, annual. Skipping any one leaves a gap.
• Track gaps to closure. The post-test report lists gaps; somebody owns each gap; the next test verifies closure.
• Run surprise game days. A game day the team knows about in advance does not exercise communication and decision-making.
• Use real sandbox clusters. A namespace restore is not a full cluster restore. The quarterly test must use a real cluster.
• Document the runbook from the test. The runbook is updated based on what the test exposed. A runbook that has never been updated is stale.

DR testing is the discipline that turns a backup program into a recovery program. A program without testing is hope. A program with testing is confidence.