Runbook: Host reboot — services do not come back

1 · Prerequisites

Confirm every item is in place before any state change.

• You have console or SSH access to the host and can use sudo
• You know which compose projects should be running here and where their compose files live
• The compose files are in version control and you can check out the revision that was deployed
• You know which named volumes hold state, and when each was last backed up
• You know whether this host is in a load-balanced pool, and whether it should stay out of rotation until verification passes
• Set the variables reused below: PROJECT=/srv/app and CNAME=web

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · uptime -s prints the boot time. Confirm the reboot is the event you are actually investigating
• · sudo journalctl --list-boots shows the boot history and whether the previous shutdown was clean
• · systemctl is-system-running prints running, degraded or starting. A degraded host has a failed unit that may matter more than Docker
• · systemctl --failed lists the failed units. A failed mount unit is a storage fault and must be fixed before Docker, not after
• · systemctl is-enabled docker prints enabled. If it prints disabled, Docker will never start itself after a reboot, and that alone is the fault
• · systemctl is-active docker prints active, or the daemon is down
• · sudo journalctl -u docker.service -b --no-pager shows why the daemon failed to start on this boot
• · docker ps --format "{{.Names}} {{.Status}}" lists what actually came back
• · docker ps -a --filter status=exited --format "{{.Names}} {{.Status}}" lists what did not, with its exit code
• · docker compose -f "$PROJECT"/compose.yaml ps -a --format "{{.Service}} {{.Status}}" compares expected against actual for the project
• · docker inspect -f "{{.Name}} {{.HostConfig.RestartPolicy.Name}}" $(docker ps -aq) prints the restart policy of every container. A policy of no is why it did not come back
• · findmnt --verify --verbose reports any fstab entry that failed to mount, which presents to the application as missing data
• · df -h /var/lib/docker and free -h confirm there is space and memory to start anything at all

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Wait for boot to finish before diagnosing. If systemctl is-system-running prints starting, units are still coming up and half your evidence is not yet true
2. 2If it prints degraded, work systemctl --failed first. A failed filesystem mount must be resolved before you start any container that writes to it
3. 3Start the daemon if it is down: sudo systemctl start docker, then confirm systemctl is-active docker prints active. If it fails, read sudo journalctl -u docker.service -b and fix the named cause rather than retrying the start
4. 4Make the daemon survive the next reboot: sudo systemctl enable docker. Confirm systemctl is-enabled docker prints enabled
5. 5Verify storage is present before starting anything that writes: findmnt --verify reports no errors, and docker volume ls lists every named volume the project expects
6. 6Bring the stack up from the compose file rather than starting containers individually: docker compose -f "$PROJECT"/compose.yaml up -d --wait. --wait exits non-zero if a service does not reach running or healthy, which turns a silent partial start into a visible failure
7. 7If a service fails, read its logs before retrying: docker compose -f "$PROJECT"/compose.yaml logs --tail 100 "$CNAME". Repeating up -d without reading the logs destroys the evidence you need
8. 8If the failure is a missing image, pull it: docker compose -f "$PROJECT"/compose.yaml pull. If the registry is unreachable, switch to the registry-unavailable runbook
9. 9If the failure is a missing network, docker compose up recreates project networks by itself. Only a network declared external needs docker network create, and the name must match the compose file exactly
10. 10If a named volume exists but is empty, do not start the service on top of it. An application that initialises an empty volume overwrites what a restore would have recovered. Restore from backup first, then start
11. 11Fix the reason services did not auto-start: set restart: unless-stopped on every long-running service in the compose file, commit the change, and apply it with docker compose up -d --force-recreate
12. 12Confirm the policy actually took effect: docker inspect -f "{{.HostConfig.RestartPolicy.Name}}" "$CNAME" prints unless-stopped
13. 13Return the host to rotation only after verification passes, then prove the fix by rebooting an equivalent staging host and confirming the stack returns unattended

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓systemctl is-enabled docker prints enabled and systemctl is-active docker prints active
• ✓docker compose -f "$PROJECT"/compose.yaml ps --format "{{.Service}} {{.Status}}" lists every service in the compose file as running, with none missing from the output
• ✓docker ps --filter health=unhealthy -q returns no output
• ✓docker ps --format "{{.Names}} {{.HealthStatus}}" prints healthy for every service that defines a healthcheck, and no service is stuck at starting
• ✓docker inspect -f "{{.Name}} {{.HostConfig.RestartPolicy.Name}}" $(docker ps -q) prints always or unless-stopped for every running container and never no
• ✓docker inspect -f "{{.RestartCount}}" "$CNAME" prints 0. A climbing count is a crash loop, not a recovery
• ✓systemctl --failed reports 0 loaded units listed
• ✓findmnt --verify exits 0, proving every fstab entry mounted
• ✓The user-facing smoke test passes from outside the host, not from localhost on the host itself
• ✓A subsequent controlled reboot of this host brings the whole stack back with no manual intervention

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Nothing in this runbook destroys data, so the rollback is limited to the configuration changes you made
• ↶To undo restart: unless-stopped, revert the compose file from version control and run docker compose up -d --force-recreate
• ↶To undo sudo systemctl enable docker, run sudo systemctl disable docker. This is only correct if the host is deliberately managed by something other than systemd
• ↶If docker compose up started a service on an empty volume and the application initialised it, that overwrite is not reversible. Stop the service, restore the volume from backup, and start it again
• ↶A volume restore replaces whatever is currently there. Copy the current contents aside before restoring, even when you believe the volume is empty
• ↶If you created a network manually with the wrong subnet, disconnect its containers, docker network rm it, and let docker compose up recreate it from the compose file

6 · Escalation

When the runbook isn't enough, contact:

• · The Docker daemon will not start and journalctl reports a storage-driver or corrupt-metadata error: escalate to the platform team. Do not delete anything under /var/lib/docker to make it start
• · A named volume is gone rather than empty and no backup exists: escalate to the data owner and the incident commander immediately, and state plainly that the data is unrecoverable
• · journalctl --list-boots shows an unclean shutdown or the host rebooted unexpectedly: escalate to the hardware or hypervisor team in parallel. A second reboot mid-recovery makes everything worse
• · A filesystem failed its check at boot or mounted read-only: escalate to the storage team before writing anything to it
• · Services start and then immediately crash-loop: this is not a reboot fault. Switch to the container-exits-immediately runbook
• · The host is one of a cluster or load-balanced pool: escalate to the incident commander to keep it out of rotation until every verification item passes
• · Hand over: boot time, the journalctl -u docker.service output, the expected versus actual service list, the restart policy of every container, the mount and volume state, and the backup age for each volume

References

1. Start containers automatically: restart policies
2. docker compose up: --wait and --force-recreate
3. Compose file reference: restart and healthcheck
4. docker ps: --filter health and the --format placeholders
5. Keep containers alive during daemon downtime (live-restore)