Lab 21: Recover when the Docker host is gone

Objective

A production host is permanently lost. Recover from documented artefacts to a running stack on a clean host. Measure the time against your RTO.

Tasks

Task 1: Set up the source stack

On the “production” host (this is your starting point):

mkdir -p ~/dr-lab && cd ~/dr-lab

cat > compose.yml <<'EOF'
services:
  db:
    image: postgres:16
    environment:
      POSTGRES_DB: app
      POSTGRES_USER: app
      POSTGRES_PASSWORD_FILE: /run/secrets/db_pw
    secrets: [db_pw]
    volumes: [pgdata:/var/lib/postgresql/data]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app"]
      interval: 5s

  api:
    image: myorg/api:1.0.0
    environment:
      DATABASE_URL: postgres://app:$(cat /run/secrets/db_pw)@db:5432/app
    secrets: [db_pw]
    depends_on:
      db: { condition: service_healthy }
    ports: ["8080:80"]

secrets:
  db_pw:
    file: ./db_pw.txt

volumes:
  pgdata:
EOF

echo 'supersecret' > db_pw.txt
chmod 0400 db_pw.txt
docker compose up -d
sleep 10

For the lab, use nginx as the api image:

sed -i 's/myorg\/api:1.0.0/nginx:1.27/' compose.yml
sed -i 's/DATABASE_URL:.*//' compose.yml
docker compose up -d

Task 2: Inject data and snapshot the artefacts

docker compose exec db psql -U app -d app -c \
  "CREATE TABLE t(id int); INSERT INTO t SELECT g FROM generate_series(1, 100) g;"
docker compose exec db pg_dump -U app -d app -Fc -f /tmp/dump.custom
docker cp dr-lab-db-1:/tmp/dump.custom ./dump.custom

Now snapshot every artefact you’d need for recovery:

# Configuration
cp compose.yml /tmp/recovery/
cp -r /etc/docker /tmp/recovery/etc-docker  # daemon.json

# Secrets (encrypted)
cp db_pw.txt /tmp/recovery/
gpg --symmetric --batch --passphrase test123 /tmp/recovery/db_pw.txt

# Volumes (backup the named volume)
docker run --rm -v dr-lab_pgdata:/source:ro -v $PWD:/backup alpine \
  tar czf /backup/pgdata-backup.tar.gz -C /source .

# Images (pull into a local registry or keep digests)
docker compose pull
docker images --digests

Task 3: Simulate host loss

docker compose down --volumes
docker system prune -a --volumes -f

The “host” is now empty.

Task 4: Recover

On the same host (or a new one) — time every step:

echo "$(date) - starting recovery"

# 1. Install Docker if needed
# 2. Restore /etc/docker/daemon.json
# 3. Restore secrets
gpg --decrypt --batch --passphrase test123 /tmp/recovery/db_pw.txt.gpg > db_pw.txt

# 4. Pull images
docker compose pull

# 5. Create the volume
docker volume create dr-lab_pgdata
docker run --rm -v dr-lab_pgdata:/target -v $PWD:/backup alpine \
  tar xzf /backup/pgdata-backup.tar.gz -C /target

# 6. Start the stack
docker compose up -d
sleep 15

# 7. Verify
docker compose ps
docker compose exec db psql -U app -d app -c "SELECT count(*) FROM t;"
# 100

echo "$(date) - recovery complete"

Task 5: Compare to RTO

The wall-clock time from “host empty” to “stack healthy” is your actual RTO. Compare to your committed RTO.

Cleanup

docker compose down --volumes
rm -rf /tmp/recovery

Objectives

Prerequisites