Disaster recovery — the production host is gone

The host is unrecoverable. The disk is gone, or the provider lost the instance, or the filesystem is corrupt beyond fsck. You have your volume backups and a new machine.

That is the point at which people discover that volume backups were maybe a quarter of the problem. This lesson is the other three quarters: the inventory of things a Docker host holds that nobody classified as data, and therefore nobody backed up.

The inventory

Nine categories. Go through them and mark each one honestly: is it captured somewhere off this host, and have you verified that?

Categories 1, 2, 8 and 9 are the ones that turn a two-hour rebuild into a one-day one. Each is examined below, with the command that captures it now, while the host still exists.

1 and 2: the engine and its configuration

Docker’s behaviour is not only in your Compose file. daemon.json changes things your application depends on and nothing in the stack records.

DEST=/backup/host-config
mkdir -p "$DEST"

# The configuration file itself
sudo cp /etc/docker/daemon.json "$DEST/daemon.json" 2>/dev/null || echo 'no daemon.json - defaults in use' > "$DEST/daemon.json.absent"

# systemd drop-ins, where proxy settings and custom flags hide
sudo cp -r /etc/systemd/system/docker.service.d "$DEST/" 2>/dev/null || true

# The daemon's own view, which includes defaults you never set
docker info --format '{{json .}}' > "$DEST/docker-info.json"

# The exact versions
docker version --format '{{.Server.Version}}' > "$DEST/engine-version"
docker info --format '{{.Driver}} {{.CgroupDriver}} {{.LoggingDriver}}' >> "$DEST/engine-version"

3: images, which are less safe than they feel

“The images are in the registry” is true right up until one of these is true:

• The image was built on that host from a Dockerfile in a repo that has moved on, and the tag myapp:latest in the registry is now a different build.
• The image is mysql:8.0 from Docker Hub, and 8.0 has been rebuilt since — so you get a different binary than the one your data was written by.
• The registry is self-hosted on the host that died.
• The base image was deleted or the account rate-limits you at the worst moment.

STACK=/srv/app
DEST=/backup/host-config

# What is actually running right now, by digest
docker ps --format '{{.Image}}' | sort -u | while read -r img; do
docker image inspect --format '{{.RepoDigests}}' "$img"
done > "$DEST/running-image-digests.txt"

# The Compose model with every tag resolved to a digest
docker compose -f "$STACK/compose.yaml" config --resolve-image-digests > "$DEST/compose-pinned.yaml"

The pinned Compose file is the artefact that makes an image recovery deterministic. Restoring against image: myapp:latest gives you whatever latest means on the day of the disaster. Restoring against image: myapp@sha256:9f2c... gives you the build that your volume data was written by, which is the only version guaranteed to understand it.

For anything you genuinely cannot rebuild or re-pull — a vendor image, a base that has been withdrawn — docker save writes the image and all its parent layers to a tar you can keep. It is not a data backup, but it is the correct tool for “this specific image must still exist in two years”.

IMAGE=vendor/appliance:4.2
DEST=/backup/images

mkdir -p "$DEST"
docker save -o "$DEST/appliance-4.2.tar" "$IMAGE"

# Verify it loads, on a host that does not already have the image
# docker load -i "$DEST/appliance-4.2.tar"

8: TLS material, which is usually nobody’s job

Certificates are the category most consistently missed, because they arrive by automation and therefore feel automatic.

Three cases, with different answers:

Certificates from a public ACME CA (Let’s Encrypt), issued by Caddy, Traefik or certbot. Technically re-issuable, so people skip them — and then hit rate limits during a rebuild, which is exactly when you are creating many certificates quickly. The account key is the more important artefact: losing it means a new ACME account and losing any rate-limit standing associated with the old one. Back up the whole data directory (/data for Caddy, acme.json for Traefik, /etc/letsencrypt for certbot), and note that acme.json must be mode 0600 or Traefik refuses to start.

Certificates from an internal or commercial CA. Not re-issuable in an outage. Re-issuance is a ticket to another team, on their timescale, and it is a hard stop in your RTO. These must be backed up with their private keys, which means they must be backed up encrypted, which means the encryption key is now also a recovery dependency. Work that chain through before the incident.

Client certificates and the Docker daemon’s own TLS material — the CA, cert and key under /etc/docker used for a TLS-protected daemon socket. If any tooling authenticates to this daemon by certificate, and the daemon’s identity changes, that tooling stops working after the rebuild.

# Host paths
sudo find /etc /srv /opt -maxdepth 4 ( -name '*.pem' -o -name '*.crt' -o -name '*.key' -o -name 'acme.json' ) -printf '%p %m\n' 2>/dev/null

# Inside volumes - the ones that are easy to forget
docker volume ls --format '{{.Name}}' | while read -r v; do
docker run --rm -v "$v":/v:ro alpine:3.20   sh -c 'find /v -maxdepth 3 ( -name "*.pem" -o -name "*.key" -o -name "acme.json" ) 2>/dev/null'   | sed "s|^|$v: |"
done

9: network identity

The host had an address, a DNS record, firewall rules that referenced it, and a place in a load balancer pool. None of that is on the host, and none of it comes back with the data.

• DNS — the record and, critically, its TTL. A 3,600-second TTL is an hour of outage after everything else is working. Lower it permanently, now.
• The IP address — if anything downstream allow-lists it (a payment provider, a partner API, a database firewall), a new address means a change request to a third party, on their timescale. This is a genuine hard stop and it belongs in the DR plan explicitly, with the contact details.
• Firewall and load balancer configuration — rules referencing the old host by address, health check paths, TLS termination settings.
• Outbound identity — SPF records, IP reputation for outbound mail. A rebuilt host with a new address sends mail that goes to spam for a fortnight.

The rebuild, in verifiable order

1. Decide, and record the time. Declare the host unrecoverable against a written trigger. This step is free and is frequently the largest term in the real RTO.
2. Provision the host. Same OS and kernel major version as the captured docker-info.json reports. Verify with uname -r before continuing.
3. Install the engine at the captured version. Pin it. Do not take the current release because it is what the repository offers.
4. **Restore daemon.json and the systemd drop-ins, then restart the daemon.** Verify with docker info and diff against the captured docker-info.json. This must happen before any volume is created, because data-root decides where volumes land.
5. **Restore the Compose files and the .env.** From git for the yaml; from the secret store for the .env, which is not in git and must not be.
6. Restore secrets and TLS material. Verify the certificate has not expired since the backup: openssl x509 -enddate -noout -in cert.pem. Fix acme.json to 0600.
7. Pull or load images by digest from the pinned Compose file. Verify with docker image ls --digests against the captured digest list.
8. Create volumes and restore data into them. Into fresh volumes. Verify contents by size and file count before starting anything.
9. **Start the stack with docker compose up -d --wait.** Non-zero exit means a service never reached healthy; do not proceed past a failure here.
10. Run the content assertions from the restore-validation lesson. Row counts against the manifest, and a referential-integrity query. This is the step that distinguishes a restore from a fresh install.
11. Verify locally, bypassing DNS. curl --resolve app.example.com:443:127.0.0.1 https://app.example.com/healthz exercises the real virtual host and TLS on the new host with no DNS involved.
12. Only then repoint DNS or the load balancer. Watch one full health check interval before declaring the incident over.

Step 11 is the one that saves you from making the outage worse. Repointing DNS and then discovering the stack is broken means production traffic is now arriving at a host that cannot serve it, and the TTL you just committed to works against you on the way back.

NEWHOST=203.0.113.42
DOMAIN=app.example.com

# Hit the new host directly, with the right Host header and SNI
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --resolve "$DOMAIN:443:$NEWHOST" "https://$DOMAIN/healthz"

# Confirm the certificate served is the right one and not expired
openssl s_client -connect "$NEWHOST:443" -servername "$DOMAIN" </dev/null 2>/dev/null | openssl x509 -noout -subject -enddate

$ curl --resolve app.example.com:443:203.0.113.42 https://app.example.com/healthz

503 0.084s

subject=CN=app.example.com
notAfter=Jul 29 11:04:22 2026 GMT

Two findings in four lines: the application is returning 503, and the restored certificate expired two weeks ago. Both are recoverable in minutes at this point and are an extended outage if discovered after the DNS change.

Knowledge check