Capstone stage 5 — the DR drill and the go-live sign-off · Docker & Containers

The stack runs, it is hardened, and it is observed. The remaining question is the only one the capstone is really about: what happens when it breaks, and can you prove your answer?

Stage 5 rehearses four failures, measures the recovery of each, and assembles the evidence pack. Nothing here is optional — a capstone that has never been broken on purpose is a capstone whose recovery properties are unknown.

What must survive, and what may not

Decide this explicitly before writing a backup script, because it determines the whole design:

Data	Verdict	Reason
`db-data` (Postgres)	Must survive	The system of record
`caddy-data`	Should survive	ACME account key and certificates; rate limits punish re-issuance
`redis-data`	May be lost	A cache; rebuilt from the database
`prom-data`	May be lost	Operational history; painful, not fatal
`loki-data`	Should survive to retention	Often the only forensic record of an incident
Secrets	Must survive, separately	Restoring them from the same medium as the data defeats the separation

The backup

#!/usr/bin/env bash
# capstone-backup.sh — run from a systemd timer, nightly.
set -euo pipefail

STAMP=$(date -u +%Y%m%dT%H%M%SZ)
DEST=/srv/backups
mkdir -p "$DEST"

# 1. Postgres, logical, consistent, compressed. Custom format so pg_restore
#    can do a selective restore.
docker compose exec -T db \
  pg_dump -U app -d app --format=custom --compress=9 \
  > "$DEST/db-$STAMP.dump"

# 2. Caddy's certificate and account state, from the volume, without
#    stopping the container.
docker run --rm \
  -v capstone_caddy-data:/data:ro \
  -v "$DEST":/backup \
  --user 0:0 \
  alpine:3.20 tar -czf "/backup/caddy-$STAMP.tar.gz" -C /data .

# 3. The configuration that reproduces the stack.
tar -czf "$DEST/config-$STAMP.tar.gz" \
  compose.yml Caddyfile prometheus.yml rules.yml digests.env

# 4. Integrity manifest, so a corrupted transfer is detectable.
( cd "$DEST" && sha256sum ./*-"$STAMP".* > "manifest-$STAMP.sha256" )

# 5. Off-host, to storage this host cannot delete from.
rclone copy "$DEST" "backup-remote:capstone/$STAMP" --immutable

find "$DEST" -name '*.dump' -mtime +14 -delete

Five properties make that a backup rather than a copy:

--format=custom allows pg_restore to restore a single table, which is what most real recoveries need.
The config tarball includes digests.env, so the rebuild uses the same image digests that were running rather than whatever the tags point at on the day of the incident.
The manifest turns a silently truncated transfer into a failed verification.
--immutable on the remote, plus credentials that cannot delete, is what makes the backup survive a compromise of this host.
Local retention is short and remote retention is long, because local copies exist for speed and remote copies exist for safety.

Read-only / Safeprove the dump is readable before you need it— pg_restore --list parses the archive without restoring. A truncated or corrupt dump fails here in seconds.

LATEST=$(ls -1t /srv/backups/db-*.dump | head -1)
sha256sum -c --ignore-missing /srv/backups/manifest-*.sha256 2>/dev/null | tail -3
pg_restore --list "$LATEST" | wc -l

/srv/backups/db-20260811T020000Z.dump: OK
412

Illustrative output

Drill 1 — daemon restart

Tests the live-restore decision from stage 1.

# Start a request loop from a second machine, then:
sudo systemctl restart docker

Expected: the request loop shows no failures. docker ps is briefly unavailable, health check states freeze and then resume, and no container restarts.

Record: the number of failed requests (target: 0) and the daemon downtime.

Drill 2 — container loss

docker compose kill -s SIGKILL api

Expected: restart: unless-stopped brings it back within seconds; Caddy’s health checks route around it in the meantime; the burn-rate alert from stage 4 does not fire for a single short outage.

Record: time from kill to healthy, and whether any request failed.

That last point is the interesting one. If requests failed, the edge was routing to an instance the health check had not yet marked down — the gap between interval x retries and reality. It is fixable with a shorter interval or with connection draining, and it is worth knowing the number.

Drill 3 — database restore

The one people skip and the one that matters.

# 1. Note the current state so you can prove the restore worked.
docker compose exec -T db psql -U app -d app -c \
  "SELECT count(*) FROM orders;"

# 2. Restore the latest dump into a scratch database on the same server.
LATEST=$(ls -1t /srv/backups/db-*.dump | head -1)
docker compose exec -T db createdb -U app restore_test
docker compose exec -T db pg_restore -U app -d restore_test --no-owner \
  < "$LATEST"

# 3. Compare.
docker compose exec -T db psql -U app -d restore_test -c \
  "SELECT count(*) FROM orders;"

# 4. Clean up and record the result.
docker compose exec -T db dropdb -U app restore_test
date -u +%FT%TZ | sudo tee /var/log/restore-tests/last-success

Restoring into a scratch database rather than over the live one is what makes this drill runnable on a schedule. A restore procedure that requires an outage gets run once, at go-live, and never again.

Record: restore duration (this is your RTO contribution) and the row delta between live and restored (this is your RPO, in rows rather than in minutes, which is the more useful unit).

Drill 4 — total host loss

The full rebuild, on a fresh machine, timed from nothing.

Provision a host to the stage 1 specification: filesystems, daemon configuration, time sync, firewall.
Retrieve the config tarball and the latest dump from the off-host copy, and verify the manifest before using either.
Retrieve the secrets from the credential store — a separate system, with its own access path.
**Restore caddy-data** into the volume before the first start, so certificates are not re-issued and rate limits are not consumed.
Bring up the data plane, restore the database dump into it, and confirm the row counts.
Bring up the application and edge using the digests from digests.env, not the tags.
Repoint DNS and record the time at which the first request succeeded through the public name.
Run the stage 3 gate in full, positive and negative, on the rebuilt host.

Record the wall-clock time for each step. The total is your real RTO, and it is invariably longer than the estimate — the DNS TTL alone is often the largest single term, and it is the one nobody includes.

Read-only / Safethe drill log— Four lines. Dated, with measured numbers, kept in the repository next to the runbook.

cat /srv/runbooks/drill-log.tsv

date		drill			target		measured	pass
2026-08-11	daemon-restart		0 failed req	0 failed req	yes
2026-08-11	container-kill		RTO 30s		RTO 11s		yes
2026-08-11	db-restore		RTO 15m		RTO 6m20s	yes
2026-08-11	db-restore		RPO 24h		RPO 15h	 	yes
2026-08-11	host-rebuild		RTO 4h		RTO 3h05m	yes

Illustrative output

The evidence pack

The capstone overview defined the Definition of Done. Stage 5 collects the evidence for it. Two notes on collecting it correctly:

Read-only / Safeevidence: security and resource controls— One line per service, from the fields that actually exist. This is the artefact for the sign-off.

docker compose ps -q | while read -r c; do
  docker inspect --format \
    '{{.Name}} priv={{.HostConfig.Privileged}} user={{.Config.User}} ro={{.HostConfig.ReadonlyRootfs}} mem={{.HostConfig.Memory}} cpu={{.HostConfig.NanoCpus}} caps={{json .HostConfig.CapDrop}}' "$c"
done

/capstone-caddy-1 priv=false user= ro=false mem=134217728 cpu=250000000 caps=["ALL"]
/capstone-api-1 priv=false user=65532:65532 ro=true mem=536870912 cpu=1000000000 caps=["ALL"]
/capstone-db-1 priv=false user=999:999 ro=false mem=2147483648 cpu=2000000000 caps=["ALL"]

Illustrative output

Read-only / Safeevidence: no unbudgeted container— Any service with a zero memory or CPU limit is unbudgeted and fails the capacity part of the sign-off.

docker compose ps -q | while read -r c; do
  docker inspect --format '{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}} {{.Name}}' "$c"
done | awk '$1 == 0 || $2 == 0 {print "UNBUDGETED:", $3}'
echo 'budget audit complete'

budget audit complete

Illustrative output

The full pack, one file per section, dated and committed:

Section	Evidence
Functional	Stage 3 positive gate transcript; both endpoints 200 with `ssl_verify_result=0`
Segmentation	Stage 3 negative gate transcript, all four tests failing as required
Supply chain	`digests.env`; `cosign verify` output per image; scan report per digest
Hardening	Container audit script output with zero FAIL lines, plus the accepted-risk register
Capacity	Sum of limits against host RAM and cores; the growth slope and threshold date
Observability	All targets up; correlated trace ID across three pillars; recorded alert fire and clear
Recovery	The drill log above, dated within 90 days
Ownership	Named owner, on-call rotation, and the runbook location

Why the accepted-risk register is part of the pack, not a failure to fix

Every real environment has exceptions. The capstone has at least two: the log shipper mounts the Docker socket, and Caddy binds privileged ports with NET_BIND_SERVICE.

The difference between a mature environment and a sloppy one is not the absence of exceptions — it is whether each one is written down with four fields: what the exception is, why it is necessary, what compensating control exists, and when it will be reviewed.

RISK-001  Log shipper mounts /var/run/docker.sock
  Why:       container discovery via the Docker API
  Mitigation: socket proxy allow-listing GET /containers/json only
  Review:    2026-11-11, or on any change to the shipper

That register is what turns the hardening audit’s FAIL lines into signal. Without it, a known exception produces a finding every run, everyone learns to ignore the output, and the one new finding arrives in a list nobody reads.

What happens if a drill fails halfway through?

That is the drill working. The purpose is to find the gap while the stakes are a scheduled window rather than a customer outage.

Three failures are so common they are almost expected:

The restore needs a credential nobody has. The database password lives in the credential store, and the person running the drill has no access. Discovered at 15:00 on a Tuesday it is a ticket; discovered at 03:00 during an outage it adds an hour.
The config tarball is missing a file. Usually digests.env, or a Caddyfile edited on the host and never committed. The rebuild works with the wrong version and the difference surfaces days later.
The off-host copy is unreadable. Wrong credentials, expired token, or a bucket lifecycle rule that expired the objects. Nothing reports this; the copy job succeeds and the objects quietly age out.

Stop, record what failed, fix it, and re-run the drill from the beginning. A partially completed drill proves nothing — the value is in the unbroken sequence.

What must survive, and what may not

The backup

Drill 1 — daemon restart

Drill 2 — container loss

Drill 3 — database restore

Drill 4 — total host loss

The evidence pack

Sanity check