high riskservice affecting~60 min

Runbook: Upgrade Docker Engine on a production host

1 · Prerequisites

Confirm every item is in place before any state change.

A change window is agreed, and someone other than you can put the host back into the load balancer
The exact target Engine version is chosen and confirmed present in the configured Docker repository
Out-of-band or console access to the host, not only SSH that depends on Docker being healthy
Current versions recorded from docker version and docker info before anything is touched
You know whether this host is a Swarm node, because live restore does not apply to Swarm services
Named volumes are backed up and at least one restore has been tested
Root or sudo on the host, and network reach to the Docker package repository

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· docker version --format "{{.Server.Version}}" - record the running server version
· docker info - read the Server Version, Storage Driver, Docker Root Dir, Swarm and Live Restore Enabled lines
· docker ps --format "{{.Names}} {{.Image}} {{.Status}}" > /var/tmp/docker-pre-upgrade.txt - the list you will diff against afterwards
· sudo dockerd --validate - exits non-zero if /etc/docker/daemon.json would be rejected; it does not start the daemon
· apt list --all-versions docker-ce (Debian/Ubuntu) or dnf list docker-ce --showduplicates (RHEL) - confirm the target version exists
· df -h /var/lib/docker - confirm free space for the new packages and any image pulls afterwards
· systemctl is-active docker containerd - both must be active before you start
· journalctl -u docker --since "24 hours ago" | grep -i level=error - a daemon already logging errors is not a daemon to upgrade

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Save the pre-upgrade state. Run docker version, docker info and docker ps -a and redirect all three to /var/tmp/docker-pre-upgrade.txt. Expect a file you can diff against later.
2Remove the host from the load balancer and confirm no new connections arrive. Expect the load balancer to report the host down or draining before you continue.
3If this is a Swarm node, drain it from a manager with docker node update --availability drain "$NODE". Expect docker node ls to show AVAILABILITY Drain and the tasks to reappear on other nodes.
4Decide the live-restore posture. Read the Live Restore Enabled line from docker info. If it says false, or this is a Swarm node, plan for the containers to stop during the upgrade and tell the change ticket so.
5Pin the target version into a variable. On Debian/Ubuntu take the version from apt list --all-versions docker-ce and set VERSION_STRING to it. On RHEL take the second column of dnf list docker-ce --showduplicates.
6Refresh package metadata only, with sudo apt-get update or sudo dnf makecache. Expect the Docker repository to be reachable and nothing yet installed.
7Install the pinned version: sudo apt-get install docker-ce="$VERSION_STRING" docker-ce-cli="$VERSION_STRING" containerd.io docker-buildx-plugin docker-compose-plugin. Read the plan apt prints and expect no package removals you did not intend.
8Wait for the package manager to restart the daemon, then check systemctl is-active docker. Expect active within a few seconds of the install finishing.
9Confirm the version moved: docker version --format "{{.Server.Version}}" prints the pinned version, not the previous one.
10Compare the running containers against /var/tmp/docker-pre-upgrade.txt. Expect the same container names. Expect the Up times to be continuous only if live restore held.
11Bring back anything that stopped or is not healthy with docker compose -f "$COMPOSE_FILE" up -d. Expect every service to reach a running and healthy state.
12Pin the packages so an unattended upgrade cannot move them: sudo apt-mark hold docker-ce docker-ce-cli containerd.io, or sudo dnf versionlock add docker-ce docker-ce-cli containerd.io where the versionlock plugin is installed.
13Return the host to service. Re-add it to the load balancer, or run docker node update --availability active "$NODE" on a Swarm manager. Expect request logs to show traffic within one health-check interval.
14Record the new versions in the change ticket and keep /var/tmp/docker-pre-upgrade.txt until the next upgrade.

4 · Verification

Confirm the procedure actually fixed the problem.

✓docker version --format "{{.Server.Version}}" prints the intended target version
✓systemctl is-active docker containerd prints active twice
✓Every container name in /var/tmp/docker-pre-upgrade.txt appears in docker ps --format "{{.Names}}" - none is missing
✓docker ps --filter health=unhealthy --format "{{.Names}}" prints nothing
✓sudo dockerd --validate exits 0, confirming the new daemon still accepts the existing /etc/docker/daemon.json
✓docker info reports the same Storage Driver and Docker Root Dir as it did in the pre-check
✓journalctl -u docker --since "30 minutes ago" contains no level=error lines
✓apt-mark showhold lists docker-ce and docker-ce-cli, or dnf versionlock list does
✓The load balancer marks the host healthy and application logs show real requests being served

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Take the host out of the load balancer again before attempting any downgrade. A half-downgraded host serving traffic is worse than a drained one.
↶Set OLD_VERSION to the version string recorded in the pre-check, not to a guess.
↶Release the hold first: sudo apt-mark unhold docker-ce docker-ce-cli containerd.io, or sudo dnf versionlock delete docker-ce docker-ce-cli.
↶Downgrade with sudo apt-get install --allow-downgrades docker-ce="$OLD_VERSION" docker-ce-cli="$OLD_VERSION", or sudo dnf downgrade docker-ce-$OLD_VERSION docker-ce-cli-$OLD_VERSION.
↶Expect containers to stop during a downgrade. Live restore is documented only for patch-level moves, and a downgrade is not one; plan the outage rather than hoping.
↶If the older daemon refuses to start, read journalctl -u docker before doing anything else. Do not delete /var/lib/docker to make it start - that destroys every volume on the host.
↶Bring the stack back with docker compose -f "$COMPOSE_FILE" up -d and confirm docker ps matches /var/tmp/docker-pre-upgrade.txt before returning traffic.
↶If the downgrade cannot be made to work, treat the host as lost: rebuild it from configuration management and restore the volumes from backup.

6 · Escalation

When the runbook isn't enough, contact:

· The new daemon does not start and dockerd --validate exits 0: escalate to the platform team with the full journalctl -u docker output attached. Do not clear /var/lib/docker.
· The daemon starts but docker image ls is empty and docker info shows a different Storage Driver: stop. Escalate before pulling or rebuilding anything; the previous store may still be recoverable.
· A Swarm manager fails to rejoin quorum after the upgrade: escalate to the Swarm owner and upgrade no further managers until it is resolved.
· A volume appears empty or wrong after the upgrade: escalate to the data owner immediately and run no prune command of any kind.
· The downgrade also fails: escalate to the platform team and switch to the host-rebuild path with the backup owner on the call.

Upgrading Docker Engine is usually uneventful and occasionally catastrophic, and the difference is decided before you type the install command. This runbook assumes the containers on the host matter, so it drains first, pins the version, and treats live restore as a bonus rather than a plan.

Step 1: Know exactly what live restore protects

live-restore keeps running containers alive while the daemon itself is unavailable. It is a real feature and it is worth enabling. It is also narrower than almost everyone assumes.

Read-only / Safelive restore state— Reads the daemon's current configuration. No side effects.

# The documented output line, read directly
docker info | grep -E 'Server Version|Storage Driver|Docker Root Dir|Swarm|Live Restore Enabled'

# What the daemon would accept as configuration, without starting it
sudo dockerd --validate
echo "validate exit: $?"

# The file itself
sudo cat /etc/docker/daemon.json

Live restore does not cover the three cases you most want it to

It is documented only for patch releases. Docker’s own documentation states that live restore is supported when installing patch releases (YY.MM.x), not for major (YY.MM) daemon upgrades. A minor-version jump is outside the guarantee, and a downgrade certainly is.

It does not apply to Swarm. The documentation is explicit: live restore pertains to standalone containers and not to Swarm services. On a Swarm node the tasks are managed by the cluster, and restarting the daemon is a cluster event, not a local one.

It does not promise anything about a containerd or shim move. Live restore is about the daemon going away while the container processes keep running. The container processes are supervised by containerd and its shims. When the upgrade replaces containerd.io as well - which the documented install command does - the shim layer under your containers is part of the change, and daemon-level configuration changes are documented as a case where “live restore may not work and you may need to manually stop the containers”.

Plan for containers to stop. Be pleased when they do not.

There is one more failure mode worth naming, because it is slow rather than loud: while the daemon is down, containers keep writing to the FIFO the daemon normally drains. If the daemon is down long enough, that log fills, and a full log blocks containers from logging more data - which for many applications means blocking the application itself. A long daemon outage under live restore is not a free outage.

Step 2: Drain before you touch the packages

Service impact possibledrain— Removes the host from service. Traffic moves elsewhere.

# Standalone host: remove from the load balancer first, by whatever
# mechanism you use. Then confirm nothing new is arriving.
ss -tn state established '( sport = :443 or sport = :80 )' | wc -l

# Swarm node: drain from a MANAGER, not from the node itself
NODE=node-03
docker node update --availability drain "$NODE"
docker node ls --filter "name=$NODE"

# On the node, watch the tasks leave
docker ps --format '{{.Names}}	{{.Status}}'

Draining is not optional politeness. It converts every subsequent failure in this runbook from an outage into a delay, which is the whole reason the runbook is long.

Step 3: Pin the version

Never upgrade to “latest” on a production host. Pin, so the change ticket, the staging host and the production host all name the same bytes.

Read-only / Safefind the version— Lists available package versions. Installs nothing.

# Debian / Ubuntu
apt list --all-versions docker-ce

# RHEL / Rocky / Alma
dnf list docker-ce --showduplicates | sort -r

Service impact possibleapt install, pinned— Installs the named version and restarts the daemon.

# Take this string verbatim from the apt list output above
VERSION_STRING='5:29.7.2-1~ubuntu.24.04~noble'

sudo apt-get update
sudo apt-get install \
docker-ce="$VERSION_STRING" \
docker-ce-cli="$VERSION_STRING" \
containerd.io \
docker-buildx-plugin \
docker-compose-plugin

Service impact possiblednf install, pinned— Installs the named version and restarts the daemon.

# On RHEL the version string is joined to the package name with a hyphen
VERSION_STRING='3:29.7.2-1.el9'

sudo dnf makecache
sudo dnf install \
"docker-ce-$VERSION_STRING" \
"docker-ce-cli-$VERSION_STRING" \
containerd.io \
docker-buildx-plugin \
docker-compose-plugin

Step 4: Verify the containers actually survived

Read-only / Safepost-upgrade check— Reads the new daemon state and compares it with the recorded baseline.

docker version --format '{{.Server.Version}}'
systemctl is-active docker containerd

# The set comparison that matters: is anything missing?
docker ps --format '{{.Names}}' | sort > /var/tmp/docker-post-upgrade.txt
diff <(awk '{print $1}' /var/tmp/docker-pre-upgrade.txt | sort) \
   /var/tmp/docker-post-upgrade.txt

# Health, not liveness
docker ps --filter health=unhealthy --format '{{.Names}}'
docker inspect --format '{{.Name}} {{.State.Health.Status}}' $(docker ps -q)

# The daemon's own opinion of the last half hour
journalctl -u docker --since "30 minutes ago" --no-pager | grep -i level=error

“docker ps shows things” is not verification. The test is that the set of names matches the baseline and that nothing is unhealthy. A container that restarted is fine if you expected it to; a container that vanished is an incident.

Step 5: Rollback, and why the downgrade is harder than it looks

Service impact possibledowngrade— Reinstalls the previously recorded version. Containers will stop.

OLD_VERSION='5:29.7.1-1~ubuntu.24.04~noble'

sudo apt-mark unhold docker-ce docker-ce-cli containerd.io
sudo apt-get install --allow-downgrades \
docker-ce="$OLD_VERSION" \
docker-ce-cli="$OLD_VERSION"

sudo systemctl restart docker
docker version --format '{{.Server.Version}}'

Four things make a Docker Engine downgrade a poorer safety net than the word “rollback” suggests:

Live restore will not carry you back. It is documented for patch upgrades. A downgrade stops the containers, so the rollback has an outage that the upgrade may not have had.
The data root has already been touched by the newer daemon. Engine occasionally migrates on-disk metadata forward. Forward migrations are supported; backward ones are not a documented path, and the older daemon can refuse a store it does not recognise.
containerd.io is not pinned by the documented install command. The install line names containerd.io without a version, so a downgrade of docker-ce alone leaves the newer containerd in place. That combination is not one anybody tested.
The apt or dnf repository may no longer carry the old version. Docker’s repositories do prune. If you did not record the exact version string during the pre-check, and the package is gone, there is nothing to downgrade to.

The honest conclusion: the reliable rollback for a Docker Engine upgrade is a rebuilt host plus a volume restore, and the downgrade is the fast path you try first. Size the change window for the slow one.

Common patterns

Symptom	Likely cause	Resolution
Containers stopped despite `live-restore: true`	The upgrade crossed a minor version, or `containerd.io` moved	Expected; bring the stack back with `docker compose up -d`
Swarm tasks all restarted	Live restore does not apply to Swarm services	Expected; drain the node first next time
`docker image ls` empty, disk still full	Storage driver changed	Restore the previous `storage-driver` in `daemon.json`; do not re-pull
Daemon will not start, `dockerd --validate` exits non-zero	`daemon.json` uses a key the new version rejects	Fix the key, revalidate, then start
Daemon will not start, `--validate` exits 0	Not a config problem	Read `journalctl -u docker`; escalate
Version unchanged after install	apt held the package, or the repo served the same version	Check `apt-mark showhold` and the repo list
Containers alive but their logs froze during the window	The daemon was down long enough to fill the log FIFO	Restart the affected containers; shorten future daemon downtime