critical riskservice affecting~180 min

Runbook: Rebuild a Docker host from scratch

1 · Prerequisites

Confirm every item is in place before any state change.

Replacement hardware or VM is provisioned, reachable over SSH, and has the same or greater disk capacity
The base OS and OS version that the lost host ran are known and available
The engine version the lost host ran is known, from configuration management or an inventory record
Volume backups exist off the lost host and their most recent successful timestamp is known
Compose files and daemon configuration are in version control, and you can clone the repository
Registry credentials can be reissued, and you have the access to reissue them
Secrets can be re-sourced from the secret manager, not from the lost host
The stated RPO and RTO for the affected services are known before you start
DNS records pointing at the lost host can be edited

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the host is genuinely unrecoverable, not merely unreachable: check console, power state and network before rebuilding
· ip addr show and ip route show on the replacement host confirm the expected address and default gateway
· lsblk and df -h confirm the disk layout, and that the intended Docker data root has capacity
· BACKUP_HOST=backup.example.com; ssh "$BACKUP_HOST" ls -lh /srv/backups/docker/ lists the archives and their timestamps
· The newest archive timestamp minus now is inside the stated RPO; if it is not, record the gap before proceeding
· git -C /srv/infra log -1 --format=%cd shows the compose and daemon config repository is current
· Confirm the registry is reachable from the replacement host: curl -fsS -o /dev/null -w "%{http_code}\n" https://registry.example.com/v2/
· Confirm the secret manager is reachable and that your credentials for it still work

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Declare the incident and record the start time; RTO is measured from the outage, not from when you started typing
2Install the OS and apply the same OS release and kernel series the lost host ran
3Set the hostname and confirm forward and reverse DNS resolve as the services expect
4Install the Docker apt repository and GPG key exactly as the install docs specify
5Pin the engine version: VERSION_STRING=5:29.7.2-1~ubuntu.24.04~noble then apt install docker-ce="$VERSION_STRING" docker-ce-cli="$VERSION_STRING" containerd.io docker-buildx-plugin docker-compose-plugin
6Hold the packages so an unattended upgrade cannot move the version under you: apt-mark hold docker-ce docker-ce-cli containerd.io
7Restore /etc/docker/daemon.json from version control, then validate it before starting the daemon: dockerd --validate --config-file /etc/docker/daemon.json exits 0
8Restore the data-root setting and any storage or logging driver settings that the services depend on
9Start and enable the daemon: systemctl enable --now docker, then docker info reports the expected storage driver and logging driver
10Restore daemon TLS material if the host exposes a remote API: ca.pem, server-cert.pem and server-key.pem, each mode 0400 for keys, owned by root
11Reissue rather than copy registry credentials: docker login registry.example.com --username "$REG_USER" --password-stdin, reading the new token from the secret manager
12Restore the compose files and any .env templates from version control into their original paths
13Re-source every secret from the secret manager into its expected location; never reconstruct a secret from memory or from an old backup
14Pull the images the compose files reference, by digest where the compose file pins one: docker compose pull
15Create the named volumes and restore each one from backup before the first start, following the volume restore runbook
16Restore the host firewall rules and confirm the ruleset is loaded and persistent across reboot
17Start the stacks in dependency order: data stores first, then internal services, then anything public facing
18Only after the services are verified healthy, move DNS to the new host address and confirm the change propagated
19Re-register the host with monitoring, log shipping and the backup job, and confirm the first backup run succeeds
20Record the actual recovery time against the stated RTO, and record every inventory item that was missing

4 · Verification

Confirm the procedure actually fixed the problem.

✓docker version prints the pinned engine version on both client and server
✓dockerd --validate --config-file /etc/docker/daemon.json exits 0 and docker info reports the storage driver and data root from daemon.json
✓systemctl is-enabled docker prints enabled, so the host survives a reboot
✓docker compose ps shows every expected service in running state with no restart loop
✓Every service with a healthcheck reports healthy: docker inspect -f "{{.Name}} {{.State.Health.Status}}" on each container prints healthy, not starting
✓A fresh authenticated pull succeeds: docker pull registry.example.com/team/app:1.4.2 completes without prompting for credentials
✓Application-level read succeeds against restored data, not merely a file listing: a record known to exist before the loss is retrievable through the application API
✓The public endpoint answers from outside the host: curl -fsS -o /dev/null -w "%{http_code}\n" https://app.example.com/health prints 200 from a machine that is not the new host
✓The firewall ruleset is loaded and denies what it denied before: a port that was closed is still refused from an external host
✓Monitoring shows the host reporting, and the first scheduled backup run completed successfully
✓Reboot the host once and confirm every service returns without manual intervention

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶This procedure has no rollback in the usual sense: the original host is already gone. What it has is a decision point at DNS cutover.
↶Before DNS is moved, the rebuild is entirely reversible in effect — the new host serves nobody. Stop, and no user is affected.
↶If verification fails after DNS is moved, revert the DNS record to the previous target if any surviving host can serve, and let the TTL expire
↶If a volume restore produced wrong or stale data, stop the consuming service before it accepts writes and restore from an earlier archive
↶If the pinned engine version turns out to be wrong, apt-mark unhold then reinstall the correct pinned version and restart the daemon; container data on the data root is unaffected
↶If the rebuild cannot meet RTO, escalate for a degraded-service decision rather than continuing silently past the deadline

6 · Escalation

When the runbook isn't enough, contact:

· The newest volume backup is outside the stated RPO: escalate to the service owner immediately; the data gap is a business decision, not yours
· No off-host backup exists for a volume: escalate as data loss and raise an incident before spending time on the rebuild
· Registry credentials cannot be reissued because the account owner is unavailable: escalate to the platform team on-call
· A secret cannot be re-sourced because the only copy was on the lost host: escalate to the security team; that secret must be rotated everywhere, not recovered
· The engine version the lost host ran cannot be determined: escalate to whoever owns configuration management before guessing, since a version mismatch changes container behaviour
· RTO will be missed: notify the incident commander with a revised estimate as soon as you know, not when the deadline passes
· The host loss cause is suspected compromise rather than hardware failure: stop, and follow the compromised-host response before restoring anything onto a new host

The value of this runbook is not that any single step is difficult. Every step here is something you have done before. The value is that the inventory is complete, because the item you forget is always discovered at the worst possible moment — usually after DNS has been moved and users are already on the new host.

Work through it in order. Do not skip ahead to starting containers.

Before anything: RPO and RTO

The inventory

Nine things must be rebuilt. Missing any one of them produces a host that looks correct and behaves incorrectly.

#	Item	Where it comes from	Failure if missed
1	Engine and version pin	Install docs, inventory record	Behaviour differs from the lost host in ways nobody predicted
2	`daemon.json`	Version control	Wrong data root, wrong log driver, unbounded log growth
3	Daemon TLS material	Certificate authority, secret manager	Remote API refuses clients, or worse, accepts them without verification
4	Registry credentials	Reissued from the registry	Pulls fail at the first private image
5	Images	Registry, pinned by digest	Silently newer image than the one that was running
6	Volumes	Off-host backups	Application starts with empty state
7	Compose files and `.env`	Version control	Wrong ports, wrong networks, missing depends_on ordering
8	Secrets	Secret manager	Service starts and fails auth against every dependency
9	Firewall and DNS	Version control, DNS provider	Host is either unreachable or fully exposed

Print this table. Tick items off physically.

Step 1: Confirm the host is actually lost

Read-only / Safereality check— Reads network and backup state. No changes.

BACKUP_HOST=backup.example.com

# Is it lost, or just unreachable?
ping -c 3 192.0.2.40
ssh -o ConnectTimeout=5 192.0.2.40 true; echo "ssh exit: $?"

# What is the newest usable backup, and how old is it?
ssh "$BACKUP_HOST" ls -lh --time-style=long-iso /srv/backups/docker/

# Is the configuration repository current?
git -C /srv/infra log -1 --format='%cd %h %s'

A host that fails to answer SSH is not necessarily gone. Check the console and the power state before you rebuild — a rebuild you did not need still costs you the RTO.

Step 2: Engine, pinned

Configuration changerepository and pinned install— Installs the Docker apt repository and a specific engine version.

sudo apt update
sudo apt install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
-o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository as the install docs specify for your release,
# then list what is available:
apt list --all-versions docker-ce

# Pin to the version the lost host ran
VERSION_STRING=5:29.7.2-1~ubuntu.24.04~noble
sudo apt install -y \
docker-ce="$VERSION_STRING" \
docker-ce-cli="$VERSION_STRING" \
containerd.io docker-buildx-plugin docker-compose-plugin

sudo apt-mark hold docker-ce docker-ce-cli containerd.io

Step 3: daemon.json, validated before the daemon starts

Configuration changedaemon config— Restores and validates the daemon configuration. Validation does not start the daemon.

sudo install -m 0644 -o root -g root \
/srv/infra/docker/daemon.json /etc/docker/daemon.json

# Validate WITHOUT starting the daemon; non-zero exit means invalid
sudo dockerd --validate --config-file /etc/docker/daemon.json
echo "validate exit: $?"

sudo systemctl enable --now docker

docker info
docker info -f '{{.DockerRootDir}}'
docker info -f '{{.LoggingDriver}}'

dockerd --validate is the flag that turns “the daemon will not start and I do not know why” into a one-line answer. Use it every time, before systemctl start.

If daemon.json set a non-default data-root, that directory must exist on the right filesystem with the right capacity before the daemon starts, or Docker quietly uses the default and you restore several hundred gigabytes onto the wrong disk.

Step 4: TLS material, if the host exposes a remote API

Configuration changedaemon TLS— Installs daemon TLS material. Key files must not be world readable.

sudo install -d -m 0700 /etc/docker/certs
sudo install -m 0444 ca.pem          /etc/docker/certs/ca.pem
sudo install -m 0444 server-cert.pem /etc/docker/certs/server-cert.pem
sudo install -m 0400 server-key.pem  /etc/docker/certs/server-key.pem

# The daemon must run with verification on:
#   --tlsverify
#   --tlscacert=/etc/docker/certs/ca.pem
#   --tlscert=/etc/docker/certs/server-cert.pem
#   --tlskey=/etc/docker/certs/server-key.pem
#   -H=tcp://0.0.0.0:2376

openssl x509 -in /etc/docker/certs/server-cert.pem -noout -dates -subject

Note that the new certificate must carry the new host address in its subjectAltName. A certificate issued for the old host address will fail verification on the replacement, and the error the client shows is not obviously about the address.

Step 5: Registry credentials — reissue, do not copy

Configuration changeregistry login— Authenticates to the registry using a freshly issued token read from stdin.

REG=registry.example.com
REG_USER=svc-deploy

# Read the NEW token from the secret manager into stdin.
# Never paste a token onto the command line: it lands in shell history.
get-secret registry/svc-deploy-token \
| docker login "$REG" --username "$REG_USER" --password-stdin

# Where did it get stored? credsStore/credHelpers mean a helper,
# otherwise the auths entry is base64 - which is encoding, not encryption.
grep -o '"credsStore"\|"credHelpers"\|"auths"' ~/.docker/config.json

The credentials the lost host held should be treated as lost with it, particularly if the loss cause is not fully understood. Reissue them. See the registry credential rotation runbook for finding every other consumer of the old credential.

Step 6: Images, by digest

Read-only / Safepull— Pulls images referenced by the compose files. Writes to the local image store only.

cd /srv/infra/stacks/app

docker compose config --images
docker compose pull

# What did you actually get?
docker image ls --digests
docker inspect -f '{{index .RepoDigests 0}}' registry.example.com/team/app:1.4.2

A tag is a moving pointer. If your compose files pin only tags, the rebuild pulls whatever that tag means today, which may not be what the lost host was running. docker compose config --images shows you what will be pulled before you pull it, and RepoDigests records what you got, so the difference is at least visible.

Step 7: Volumes

Follow the volume restore runbook for each volume. Restore before the first start of the consuming service, not after — a database that initialises an empty data directory on first start will refuse to start against restored data afterwards, and the error it gives is confusing.

Read-only / Safevolume inventory— Lists expected volumes and available archives. No changes.

BACKUP_HOST=backup.example.com

# What does the stack expect?
docker compose config --volumes

# What archives exist, and how old are they?
ssh "$BACKUP_HOST" ls -lh --time-style=long-iso /srv/backups/docker/

Step 8: Secrets

Every secret comes from the secret manager. None comes from the lost host, from a backup of the lost host, or from anybody’s memory.

If a secret existed only on the lost host, it is not recoverable and it is also now of unknown disposition. Treat it as leaked: rotate it at the issuer rather than trying to recover it.

Step 9: Firewall, then start, then DNS

Service impact possiblefirewall and start— Loads the firewall ruleset and starts the stacks. Services become live on the new host.

# Firewall FIRST - starting containers before the ruleset is loaded
# publishes ports to the world for the duration of the gap.
sudo nft -f /srv/infra/firewall/docker-host.nft
sudo nft list ruleset | head -40
sudo systemctl is-enabled nftables

# Data stores first, then services, then anything public facing
docker compose up -d db
docker compose up -d app
docker compose ps

Verification, in the order it should be done

Read-only / Safepost-rebuild checks— Reads engine, container and endpoint state. No changes.

docker version
docker info -f '{{.DockerRootDir}} {{.LoggingDriver}} {{.ServerVersion}}'
systemctl is-enabled docker

docker compose ps
for c in $(docker compose ps -q); do
docker inspect -f '{{.Name}} {{.State.Status}} {{.State.Health.Status}}' "$c"
done

# Authenticated pull works
docker pull registry.example.com/team/app:1.4.2

# From a machine that is NOT this host:
#   curl -fsS -o /dev/null -w '%{http_code}\n' https://app.example.com/health

The reboot test

The rebuild is not finished until the host has been rebooted once and every service has come back without you touching it. A host that works only because you started things by hand has not been rebuilt; it has been improvised, and the next unplanned reboot repeats this incident.