Backup strategy and restore testing

A backup strategy is not a list of commands. It is four decisions, made once per data object and written down:

1. What — the complete inventory, derived from the engine rather than from memory.
2. How often — derived from the RPO for that object’s tier.
3. How consistency is obtained — stopped, dumped, or atomically snapshotted.
4. How you find out it stopped — the decision that is almost never made.

The fourth one is the subject of the most expensive failure in this lesson, so it gets the most space.

Step 1: classify what you hold

Not everything on a Docker host deserves the same treatment, and treating it uniformly means either overpaying for build caches or underprotecting user data.

The classification does real work. Tier C is the bulk of the disk on most hosts and needs no backup at all — which is what makes an aggressive tier A schedule affordable. Tier D changes rarely and should be versioned rather than backed up, which is a different mechanism with different properties.

The mistake worth naming: certificates are tier A, not tier D. They feel like configuration because they live next to it. A private key that was issued once and exists nowhere else is irreplaceable data.

# Volumes, with size and reference count
docker system df -v | sed -n '/Local Volumes/,$p'

# Every bind mount, from the engine rather than from compose.yaml
docker ps -a --format '{{.Names}}' | while read -r c; do
docker inspect --format   '{{range .Mounts}}{{if eq .Type "bind"}}{{.Source}} -> {{.Destination}}
{{end}}{{end}}' "$c"
done | sort -u

# Volumes that no container references - orphan candidates, or forgotten data
docker volume ls --filter dangling=true --format '{{.Name}}'

Run all three, put the output in a spreadsheet, and put a tier letter against every line. This takes twenty minutes and is the single highest-value hour in this part of the course, because it is the step that finds the volume nobody knew about.

Step 2 and 3: the matrix

One row per object. This is the artefact — not the script, the table. The script is derived from it.

Two rows carry the interesting decisions.

app_searchindex is deliberately not backed up. It is derivable from the database. But that decision creates an obligation: the rebuild has to be timed, because if reindexing takes six hours it is an RTO component whether or not it is a backup one. A “we can rebuild it” that nobody has timed is an estimate wearing a decision’s clothes.

The WAL archive is a separate row from the dump. This is what converts an hourly RPO into a five-minute one at almost no cost, and it is the single highest-leverage line in a typical matrix — continuous archiving of the write-ahead log means recovery to any point between base backups rather than to the last one.

Step 4: monitoring the job, not the backup

Here is the failure this lesson exists for, and it is more common than a corrupt archive.

A backup that runs and produces a bad archive is a hard problem. A backup that stopped running is an easy problem that goes undetected for months, because the absence of a file generates no alert. Nothing fires. The dashboard has no red. The last successful run was in April and it is now July.

Causes, all mundane: the credentials expired, the destination filled, a container name changed and docker exec now fails, the cron job was on a host that was rebuilt, somebody commented it out during an incident and never uncommented it.

#!/usr/bin/env bash
set -euo pipefail

export RESTIC_REPOSITORY='rest:http://backup-01.example.com:8000/app-01'
export RESTIC_PASSWORD_FILE=/etc/restic/password
MAX_AGE_HOURS=26

newest=$(restic snapshots --host app-01 --latest 1 --json | jq -r '.[0].time')
if [ -z "$newest" ] || [ "$newest" = 'null' ]; then
echo 'CRITICAL: no snapshots at all for host app-01' >&2
exit 2
fi

age_h=$(( ( $(date +%s) - $(date -d "$newest" +%s) ) / 3600 ))
echo "newest snapshot: $newest (${age_h}h ago)"

if [ "$age_h" -gt "$MAX_AGE_HOURS" ]; then
echo "CRITICAL: newest backup is ${age_h}h old, limit ${MAX_AGE_HOURS}h" >&2
exit 2
fi
echo 'OK'

$ /usr/local/bin/check-backup-freshness.sh

newest snapshot: 2026-06-11T02:14:07Z (1466h ago)
CRITICAL: newest backup is 1466h old, limit 26h

Add a size floor alongside the age check. A backup that runs, succeeds and writes 4 KB — because the dump command failed and the redirect captured an error message — passes an age check and fails a size check.

DUMP=/backup/app-$(date +%F).dump
MIN_BYTES=50000000

size=$(stat -c %s "$DUMP" 2>/dev/null || echo 0)
if [ "$size" -lt "$MIN_BYTES" ]; then
echo "CRITICAL: $DUMP is $size bytes, expected at least $MIN_BYTES" >&2
exit 2
fi

# And that it is structurally readable, not merely large
docker run --rm -v /backup:/backup:ro postgres:16 pg_restore --list "/backup/$(basename "$DUMP")" > /dev/null || { echo "CRITICAL: $DUMP is not a readable archive" >&2; exit 2; }
echo 'OK'

Three checks, three distinct failures caught: the job stopped, the job produced nothing, the job produced garbage. None of them requires a restore, all of them run nightly, and together they cover everything except “the contents are wrong”, which is what the drill is for.

The drill calendar

Drills that always rehearse the same scenario stop finding things after the second one. Rotate the scope so that over a year you have exercised every part of the plan.

1. Q1 — the database. Restore the newest dump to an isolated host, run the content assertions, measure the time. The easy case, which establishes the baseline.
2. Q2 — the old backup. Restore from the far end of the retention window, 30 or 90 days back. This is where schema drift, missing roles and format changes surface.
3. Q3 — the full host rebuild. Everything from the DR inventory: engine, daemon.json, images by digest, all volumes, all bind mounts, certificates. Time the whole thing against the committed RTO.
4. Q4 — the unfamiliar operator. Somebody who did not build the backup runs Q1 from the documentation alone, unaided, while the author stays silent. This tests the runbook and the credential escrow, which decay faster than the archives.

Q4 is the one that gets skipped and the one that finds the most. A runbook is only correct if a person who does not already know the answer can follow it, and that is not a property you can assess by reading your own document.

Knowledge check