Rollback — putting Docker back the way it was

The upgrade went in and something is wrong. Before reaching for a rollback, be clear about what a rollback actually restores, because “put it back” is not one operation and the cheapest version of it does not restore everything.

Three strategies, in increasing order of cost and completeness:

1. Package downgrade. Reinstall the previous versions. Fast, and restores the binaries and nothing else.
2. Snapshot restore. Roll the filesystem back to the pre-upgrade state. Restores everything on that filesystem, loses anything written since.
3. Rebuild from capture. Provision a host at the old version and replay the configuration capture. Slowest, most reliable, and the only one that works when the host itself is the problem.

Package downgrade

#!/usr/bin/env bash
set -euo pipefail

PREV='5:28.1.3-1~ubuntu.24.04~noble'

# Confirm the old version is still in the repository
apt-cache madison docker-ce | grep -F "$PREV" || { echo "previous version no longer published" >&2; exit 1; }

sudo apt-get install -y --allow-downgrades "docker-ce=$PREV" "docker-ce-cli=$PREV"

systemctl is-active docker
docker version --format 'server={{.Server.Version}}'

Three things to know about this command.

--allow-downgrades is required. Modern apt refuses to install an older version of an installed package without it, and the refusal message is not always obvious about the cause.

The daemon restarts again. The downgrade goes through the same postinstall path as the upgrade, so containers take a second interruption. If live restore is on and this is a patch-level downgrade, they survive it; if you are rolling back a line upgrade, they do not.

The old version has to still be published. Docker’s repository keeps a deep history — apt-cache madison docker-ce typically lists many prior versions — but this is a property of the repository, not a guarantee. Check before you commit to this as your rollback plan, not during the incident.

sudo apt-mark hold docker-ce docker-ce-cli containerd.io

# Confirm
apt-mark showhold

# Release the hold when you are ready to try again
# sudo apt-mark unhold docker-ce docker-ce-cli containerd.io

Record the hold somewhere your team reads. A held package is invisible to most patch reporting, so a host pinned during an incident and forgotten becomes a host that quietly stops receiving Docker security updates. Put an expiry date on it in the ticket.

What does not roll back with the packages

Snapshot restore

If you took a filesystem snapshot with the daemon stopped — the procedure in the capture lesson — this is the most complete rollback available short of rebuilding.

# Stop the socket first, or systemd re-activates the daemon
sudo systemctl stop docker.socket docker.service
sudo systemctl stop containerd.service

# Restore the snapshot (LVM example; adapt to your storage)
sudo lvconvert --merge /dev/vg0/docker-preupgrade

# Downgrade the packages to match the restored state
sudo apt-get install -y --allow-downgrades 'docker-ce=5:28.1.3-1~ubuntu.24.04~noble' 'docker-ce-cli=5:28.1.3-1~ubuntu.24.04~noble'

sudo systemctl start containerd.service docker.service
docker ps -a

The order matters: restore the state, then install the binaries that match it. Starting a new daemon against restored old state is the mismatch you are trying to avoid.

The LVM line is illustrative and the mechanics differ by storage stack. lvconvert --merge cannot complete while the origin volume is in use — it schedules the merge for the next activation, so the filesystem must be unmounted, or the host rebooted, before the rollback actually takes effect. ZFS (zfs rollback) and hypervisor snapshots have their own constraints. Rehearse whichever one you have on a host that is not production, because “restore the snapshot” is exactly the kind of step that turns out to need a reboot at the worst moment.

Rebuild from capture

The strategy that always works, and the reason the capture lesson comes before this one.

# On the replacement host, install the exact prior version
sudo apt-get install -y 'docker-ce=5:28.1.3-1~ubuntu.24.04~noble' 'docker-ce-cli=5:28.1.3-1~ubuntu.24.04~noble' containerd.io docker-buildx-plugin docker-compose-plugin
sudo apt-mark hold docker-ce docker-ce-cli

# Replay the captured daemon configuration
sudo install -m 0644 ./capture/daemon.json /etc/docker/daemon.json
sudo dockerd --validate
sudo systemctl restart docker

# Re-pull the exact images by digest from the capture
awk '$2 ~ /^sha256:/ {split($1,a,":"); print a[1]"@"$2}' ./capture/images.txt | xargs -r -n1 docker pull

# Bring the workload back from its declarative source
docker compose -f /srv/app/compose.yaml up -d

# Restore volume data from the backup, then verify
docker compose -f /srv/app/compose.yaml ps

This is slower than a downgrade and it is the only strategy that is unambiguously correct. Everything it depends on — the version string, the daemon config, the image digests, the Compose file, the volume backup — came out of the pre-upgrade capture.

Deciding which one

The first row is the most common and it is not a rollback at all. Check journalctl -u docker and dockerd --validate before deciding the upgrade was bad; a large share of “failed upgrades” are a daemon.json key that the new version stopped accepting, which is a two-minute fix.

Knowledge check