Restore validation — the part most teams skip

A backup that has never been restored is a hypothesis. It is a plausible belief about a file, and every property you care about — that it is complete, that it is consistent, that the tooling to read it still exists, that the passphrase is retrievable, that the schema matches the application — is untested.

The awkward part is that most restore tests do not test it either. They restore, start the stack, see green health checks, and record a pass. A container with an empty database also starts and also reports healthy. The drill and the disaster produce the same output, which is why the drill keeps passing.

What a check that can fail looks like

Compare these two.

docker compose -f /srv/restore/compose.yaml up -d
docker compose -f /srv/restore/compose.yaml ps
curl -fsS http://localhost:8080/healthz

#!/usr/bin/env bash
set -euo pipefail

DB=restore-db
MIN_ORDERS=1450000
EXPECTED_TABLES=61

# 1. The schema is complete
tables=$(docker exec "$DB" psql -U app -tAc "SELECT count(*) FROM information_schema.tables WHERE table_schema='public';")
[ "$tables" -eq "$EXPECTED_TABLES" ] || {
echo "FAIL: $tables tables, expected $EXPECTED_TABLES" >&2; exit 1; }

# 2. The largest table has a plausible row count
orders=$(docker exec "$DB" psql -U app -tAc 'SELECT count(*) FROM orders;')
[ "$orders" -ge "$MIN_ORDERS" ] || {
echo "FAIL: $orders orders, expected at least $MIN_ORDERS" >&2; exit 1; }

# 3. The most recent row is close to the backup time, not months old
age=$(docker exec "$DB" psql -U app -tAc "SELECT EXTRACT(epoch FROM now() - max(created_at))::int FROM orders;")
[ "$age" -lt 172800 ] || {
echo "FAIL: newest order is ${age}s old; backup is stale" >&2; exit 1; }

# 4. Referential integrity actually holds
orphans=$(docker exec "$DB" psql -U app -tAc 'SELECT count(*) FROM order_items oi
   LEFT JOIN orders o ON o.id = oi.order_id WHERE o.id IS NULL;')
[ "$orphans" -eq 0 ] || {
echo "FAIL: $orphans orphaned order_items - restore is not consistent" >&2; exit 1; }

echo 'PASS: restored data passes all four content assertions'

Every line has a number it must beat. Run this against a container that started with an empty data directory and it fails on assertion 1 with an exact statement of what was wrong.

Assertion 4 is the one that earns its place against the smeared-copy failure from the earlier lessons. A crash-consistent or smeared restore can produce a database that opens, has all its tables, and has broadly right row counts — while the child rows written in the last minutes reference parents that were captured before they existed. A foreign-key integrity query is the cheapest direct test of the consistency property those lessons were about.

The isolation problem

A restore drill has to run somewhere. Getting this wrong turns a safety exercise into an incident, and the mechanism is always the same: the restored stack reaches production.

PROD_DB=db.prod.example.com
PROD_REDIS=redis.prod.example.com

for target in "$PROD_DB:5432" "$PROD_REDIS:6379"; do
host=${target%:*}
port=${target#*:}
if timeout 5 bash -c "cat < /dev/null > /dev/tcp/$host/$port" 2>/dev/null; then
  echo "FAIL: restore host can reach $host:$port" >&2
  exit 1
fi
echo "OK: no route to $host:$port"
done

The drill

1. Pick the backup deliberately, not the newest. Use one from the far end of your retention — 30 days if that is your window. The newest backup is the one most likely to work and the least likely to be what you need.
2. Start the clock. A wall-clock timestamp, recorded, not estimated afterwards.
3. Prove isolation. Run the connectivity check above and confirm every production endpoint is unreachable.
4. Retrieve the backup using only the documentation. Not from memory, and ideally not by the person who wrote the backup job. This is where credential and passphrase escrow failures surface.
5. Restore the data. Into fresh volumes, never over anything existing.
6. Bring up the stack with workers scaled to zero, using the restore override file.
7. Run the content assertions. The four-check script, with thresholds from the backup manifest.
8. Exercise one real end-to-end path. Log in as a known user; fetch a known record; render a page. Something a person would do.
9. Stop the clock and record the number. This is your measured RTO for this scope. Write it next to the committed RTO.
10. Write down everything that was wrong, including every step where the documentation was inadequate and you had to ask somebody.
11. Destroy the drill environment. A half-live restored copy of production sitting on a forgotten host is a data-protection incident waiting to be found by an auditor.

Step 4 is the one that finds the most defects and is the most often skipped. Handing the runbook to someone who has never run it, and staying quiet, tests the documentation and the escrow rather than the archive — and those fail more often than the archive does.

$ ./restore-drill.sh --backup 2026-07-13

[00:00:00] drill start, backup 2026-07-13 (30 days old)
[00:00:04] isolation check: OK no route to db.prod.example.com:5432
[00:00:04] isolation check: OK no route to redis.prod.example.com:6379
[00:02:11] restic restore latest --host app-01: 18.4 GiB restored
[00:41:52] pg_restore: 61 tables, 204 indexes
[00:41:53] assertion 1 schema:     PASS 61 tables
[00:41:55] assertion 2 row count:  PASS 1,462,118 orders (manifest: 1,462,118)
[00:41:56] assertion 3 freshness:  PASS newest order 6h before backup
[00:42:31] assertion 4 integrity:  FAIL 2,341 orphaned order_items
[00:42:31] DRILL FAILED after 42m31s

Findings:
1. FK integrity broken: backup is a live tar, not a dump (see BK-114)
2. Measured 42m31s vs committed RTO 30m - pg_restore --jobs not used
3. Runbook step 6 references /etc/restic/password; file is now
   /run/secrets/restic-password. Operator had to ask.

That is what a drill is worth. Finding 1 is the smeared-backup failure caught before it mattered; finding 2 is an RTO commitment that was fiction; finding 3 is a documentation defect that would have cost twenty minutes at 03:00.

The four false passes

A drill can return PASS and still have told you nothing. These are the ways.

The third is the most uncomfortable and the most valuable. A runbook is only correct if a person who does not already know the answer can follow it.

Knowledge check