Rebuilding a host from images plus volume backups

The host is gone. You have a registry, a git repository and a directory of volume archives. This lesson is the sequence that turns those three things back into a running service, and the one ordering mistake that makes it look like it worked when it did not.

Phase 1 — the substrate

flowchart LR
  A[Provision host] --> B[Install pinned Docker]
  B --> C[Restore daemon.json]
  C --> D[Restore firewall rules]
  D --> E[Registry credentials]

Pin the engine version

Restoring onto a newer engine than you lost is a change you are making during an outage, with no rollback and no test. Pin to the version that was running.

apt-cache madison docker-ce | head
VER='5:28.0.1-1~ubuntu.24.04~noble'
sudo apt-get install -y \
"docker-ce=$VER" "docker-ce-cli=$VER" containerd.io docker-compose-plugin
sudo apt-mark hold docker-ce docker-ce-cli containerd.io
docker version --format 'server={{.Server.Version}}'

The version string is distribution-specific and long. Record the exact string in the runbook rather than reconstructing it under pressure — dpkg -l | grep docker-ce on the healthy host, today.

Restore configuration before starting anything

/etc/docker/daemon.json, any systemd drop-in under /etc/systemd/system/docker.service.d/, and the firewall rules from docker-user-chain-and-rule-persistence. All of these should come from configuration management; if they come from a backup tarball, restore them now, before the first container runs.

sudo dockerd --validate --config-file=/etc/docker/daemon.json
sudo systemctl restart docker

--validate parses the file and exits without starting the daemon, which turns a typo into a message instead of a daemon that will not come up.

Phase 2 — state before services

This is the phase where rebuilds go wrong, and the failure looks like a success.

Restore each volume with a helper container, exactly mirroring the backup:

VOL=app-postgres-data
ARCHIVE=/mnt/restore/app-postgres-data-2026-08-10.tar.gz

docker volume create "$VOL"
docker run --rm \
-v "$VOL":/target \
-v /mnt/restore:/restore:ro \
alpine:3.20 \
tar xzf "/restore/$(basename "$ARCHIVE")" -C /target
docker run --rm -v "$VOL":/target:ro alpine:3.20 ls -la /target | head

Make Compose refuse to invent state

Declare data volumes as external. The compose specification is explicit: with external: true Compose “will not attempt to create the volume” and instead returns an error if it does not exist.

services:
  db:
    image: postgres:16
    volumes:
      - app-postgres-data:/var/lib/postgresql/data

volumes:
  app-postgres-data:
    external: true

This converts the silent failure into a loud one. A rebuild that forgot to restore a volume now fails at compose up with a missing volume error, instead of starting an empty database and looking healthy.

It is the single highest-value line in a production compose file for DR purposes, and it costs one word.

Phase 3 — start, then verify data rather than liveness

$ docker compose pull
docker compose up -d
docker compose ps --format 'table {{.Name}}\t{{.State}}\t{{.Status}}'

NAME     STATE     STATUS
db       running   Up 41 seconds (healthy)
app      running   Up 38 seconds (healthy)
proxy    running   Up 38 seconds

Healthy means “the process answered its own healthcheck”. A freshly initialised empty database is healthy. Verify the data:

docker compose exec -T db psql -U app -t -c \
"SELECT count(*) FROM orders;"
docker compose exec -T db psql -U app -t -c \
"SELECT max(created_at) FROM orders;"
curl -fsS http://127.0.0.1:8080/healthz
curl -fsS http://127.0.0.1:8080/api/orders/latest

The timestamp of the newest record is your measured RPO for this recovery. Compare it against the RPO you committed to in docker-rpo-rto. If the newest order is nineteen hours old and the commitment was four, the backup schedule is the finding — not the restore procedure.

Timing the phases

Record the clock at each phase boundary. The total is your measured RTO, and the per-phase split tells you where to spend effort.

{
  echo "start           $(date --iso-8601=seconds)"
  echo "host ready      $(date --iso-8601=seconds)"
  echo "volumes restored $(date --iso-8601=seconds)"
  echo "stack up        $(date --iso-8601=seconds)"
  echo "verified        $(date --iso-8601=seconds)"
} | tee /var/log/dr-drill-timing.txt

A typical first drill splits roughly: 20% provisioning, 60% volume restore, 20% verification. That distribution says the lever is transfer speed and archive size — not a faster runbook.

1. Provision and pin. Same OS, same engine version, held packages.
2. Restore host configuration first. daemon.json, systemd drop-ins, firewall rules. Validate before starting the daemon.
3. Authenticate to the registry using break-glass credentials that do not live on the lost host.
4. Create and populate every volume, before any application container exists.
5. Check UID ownership against the image you are about to run.
6. Bring the stack up with data volumes declared external, so a missing volume is an error.
7. Verify data, not liveness. Row counts, newest record, an end-to-end read.
8. Record the timings and the newest-record timestamp. Those are your measured RTO and RPO.
9. Cut DNS or the load balancer over last, only after verification passes.

Sanity check