DR exercises — levels, cadence and the findings they produce

Three statements, each stronger than the last, and teams routinely treat them as equivalent:

1. The backup job exited zero. An archive exists.
2. The archive restores. The bytes are readable and produce a working dataset.
3. We can recover. People, credentials, documentation and infrastructure combine to put the service back within the committed RTO.

The first is monitoring. The second is a restore test. Only the third is disaster recovery, and it is the only one whose failure modes are mostly not technical.

Four levels of exercise

Most organisations do level 2 and believe they have done level 3. The gap between them is where every DR failure lives.

Level 1 — tabletop

Sit down with the runbook and the person who would be on call. Read it aloud, step by step, and stop at every point where someone has to supply knowledge that is not written down.

It costs an hour and it reliably finds four or five gaps: a step that assumes access nobody has, a decision with no named owner, an instruction that has been wrong since a migration two years ago.

Nothing is touched, so there is no reason not to do it quarterly.

Level 2 — automated component restore

The backup restore test, run by a machine, every night. Take the most recent archive, restore it to a scratch volume, start the engine against it, and assert something about the contents.

#!/bin/sh
set -eu
ARCHIVE=$(ls -1t /mnt/backups/app-postgres-data-*.tar.gz | head -1)
VOL="restore-test-$(date +%s)"
OUT=/var/lib/node_exporter/textfile_collector/restore_test.prom

cleanup() {
docker rm -f "$VOL" 2>/dev/null || true
docker volume rm "$VOL" 2>/dev/null || true
}
trap cleanup EXIT

docker volume create "$VOL"
docker run --rm -v "$VOL":/target -v /mnt/backups:/b:ro alpine:3.20 \
tar xzf "/b/$(basename "$ARCHIVE")" -C /target

docker run -d --name "$VOL" -v "$VOL":/var/lib/postgresql/data \
-e POSTGRES_PASSWORD=REPLACE_ME postgres:16
sleep 20
ROWS=$(docker exec "$VOL" psql -U postgres -At -c 'SELECT count(*) FROM orders;')

printf 'restore_test_rows %s\nrestore_test_timestamp_seconds %s\n' \
"$ROWS" "$(date +%s)" > "$OUT.tmp"
mv "$OUT.tmp" "$OUT"

Then alert on the metric going stale, exactly as in docker-monitoring-blind-spots:

time() - restore_test_timestamp_seconds > 172800

That rule fires when the verification has not succeeded in two days — whether the restore failed, the archive was missing, or the timer was disabled during maintenance and never re-enabled. The staleness alert catches all three; a failure alert catches only the first.

Writing to a temporary file and moving it into place matters: node_exporter reads the directory on every scrape, and a half-written file produces a parse error rather than a metric.

Level 3 — full rebuild on a clean host

This is the exercise. Provision a scratch host, and rebuild the service from backups alone, following docker-rebuilding-a-host-from-images-and-volumes.

The findings from a first honest level-3 drill are remarkably consistent across organisations:

• Registry credentials were only on the lost host.
• The runbook lives in a wiki hosted on the affected infrastructure.
• The pinned engine version no longer exists in the repository.
• A volume nobody knew about — anonymous, and holding data.
• The DNS change requires an account only one person can access, and they are on leave.
• The restore took four times longer than anyone estimated, because nobody had measured the transfer rate from cold storage.

None of these is found by a restore test. All of them are found in one afternoon.

Level 4 — failover with traffic

Reserved for architectures with a standby, and worth it only when levels 1 to 3 are clean. It tests cutover mechanics: DNS TTLs that are longer than you thought, clients that cache addresses, a load balancer health check that passes before the application is really ready, and whether rollback works.

Plan it like a change, announce it, and have an abort criterion written down before you start.

Cadence

Calendar-driven, with change-driven triggers on top:

The change-driven rows matter more than the calendar rows. A DR plan degrades when the system changes, not when time passes, and a quarterly cadence alone will let a six-month-old architecture change sit untested.

Recording the outcome

A drill that produces no artefact did not happen. Three things go in the record:

1. The timing log — phase boundaries and the total, compared against the committed RTO.
2. The newest restored record’s timestamp — the measured RPO, compared against the committed RPO.
3. The findings — each one a ticket, with an owner and a due date, closed before the next drill.

1. Schedule the tabletop quarterly and hold it even when nothing changed.
2. Automate the component restore nightly, and alert on its timestamp going stale rather than on its failure.
3. Run the full rebuild twice a year, from a clean host, with no access to production and a driver who did not write the runbook.
4. Add change-driven triggers for engine upgrades, new stateful services and backup tooling changes.
5. Record timings, measured RPO, and findings. Each finding becomes a ticket with an owner.
6. Close the findings before the next drill, and measure success by the list shrinking.

Sanity check