Pre-upgrade capture — what to save and what it is actually worth · Docker & Containers

The instinct before an upgrade is to “take a backup”, and the instinct is right. The usual execution is wrong: people either take nothing, on the grounds that the containers are stateless, or they try to snapshot /var/lib/docker, which is large, slow, inconsistent while the daemon runs, and of limited use afterwards.

What you actually need is five small artefacts. They fit in a few kilobytes and they are what turns “the upgrade broke the host” into “rebuild it from these”.

The five artefacts

Artefact	Why	Size
`daemon.json` and systemd drop-ins	The daemon will not behave the same without them	bytes
Compose files and deploy manifests	The declarative source of every container	kilobytes
Image digest inventory	Lets you re-pull byte-identical images	kilobytes
Container specification dumps	Reconstructs anything not in a Compose file	kilobytes
Volume data backup	The only irreplaceable item	large

Only the last one is a backup in the ordinary sense, and it belongs to the backup part of this course rather than to the upgrade procedure. The other four are configuration capture, and they are cheap enough that there is no excuse for skipping them.

Capturing them

Read-only / Safepre-upgrade capture— Everything except volume data. Read-only; safe on a live host.

#!/usr/bin/env bash
set -euo pipefail

STAMP=$(date -u +%Y%m%dT%H%M%SZ)
OUT="/var/backups/docker-preupgrade/$STAMP"
mkdir -p "$OUT"

# 1. Daemon configuration and any unit customisation
cp -a /etc/docker/daemon.json "$OUT/" 2>/dev/null || echo 'no daemon.json' > "$OUT/daemon.json.absent"
systemctl cat docker.service    > "$OUT/docker.service.txt"
systemctl cat containerd.service > "$OUT/containerd.service.txt"

# 2. Component versions, so you know what you are rolling back to
docker version > "$OUT/version.txt"
docker info    > "$OUT/info.txt"
dpkg -l 'docker-*' 'containerd*' > "$OUT/packages.txt"

# 3. Image inventory with digests
docker image ls --digests --no-trunc --format '{{.Repository}}:{{.Tag}} {{.Digest}} {{.ID}}' > "$OUT/images.txt"

# 4. Full specification of every container, running or not
docker ps -a --format '{{.Names}}' > "$OUT/container-names.txt"
docker ps -aq | xargs -r docker inspect > "$OUT/containers.json"

# 5. Volume and network inventory (not the data - see below)
docker volume ls  --format '{{.Driver}} {{.Name}}'  > "$OUT/volumes.txt"
docker network ls --format '{{.Driver}} {{.Name}}'  > "$OUT/networks.txt"

# 6. Compose projects, if the host uses them
docker compose ls --format json > "$OUT/compose-projects.json" 2>/dev/null || true

echo "capture written to $OUT"
ls -la "$OUT"

containers.json is the important one and the one people leave out. It is the complete resolved specification of every container: image digest, environment, mounts, networks, resource limits, restart policy, labels. If the host has containers created by hand rather than by Compose — and almost every host has at least one — this file is the only record of how they were built.

The image digest inventory is the valuable one

Read-only / Safedigest inventory— Repository, tag and digest for every local image.

$ docker image ls --digests --format '{{.Repository}}:{{.Tag}} {{.Digest}}'

registry.example.com/myapp:release-2026-08 sha256:9f2b4d6e8c0a2f4b6d8e0c2a4f6b8d0e2c4a6f8b0d2e4c6a8f0b2d4e6c8a0f2b
registry.example.com/myapi:release-2026-08 sha256:3c5e7a9b1d3f5a7c9e1b3d5f7a9c1e3b5d7f9a1c3e5b7d9f1a3c5e7b9d1f3a5c
postgres:16.14-bookworm                    sha256:7b9d1f3a5c7e9b1d3f5a7c9e1b3d5f7a9c1e3b5d7f9a1c3e5b7d9f1a3c5e7b9d

Illustrative output

Digests, not tags. A tag is a pointer that can move; the digest is the image. Recovering with docker pull myapp:release-2026-08 gets you whatever that tag points at today. Recovering with the digest gets you the bytes that were running before the upgrade:

Service impact possiblerecover by digest— Re-pull exactly what was there, independent of tag movement.

docker pull registry.example.com/myapp@sha256:9f2b4d6e8c0a2f4b6d8e0c2a4f6b8d0e2c4a6f8b0d2e4c6a8f0b2d4e6c8a0f2b

Read-only / Safeverify recoverability— Confirm each image can actually be re-pulled before you need to.

while read -r ref digest _; do
case "$digest" in
  sha256:*) ;;
  *) echo "SKIP  $ref (no digest - built locally?)"; continue ;;
esac
repo="${ref%%:*}"
if docker manifest inspect "$repo@$digest" > /dev/null 2>&1; then
  echo "OK    $ref"
else
  echo "FAIL  $ref - not retrievable from registry"
fi
done < /var/backups/docker-preupgrade/latest/images.txt

Any FAIL line is a decision to make before the upgrade, not after: push the image, or accept that it is not recoverable and plan accordingly.

docker manifest inspect is still labelled experimental in the CLI help. It works and it is the shortest way to ask a registry whether a digest exists, but if you would rather not depend on an experimental subcommand, docker pull --quiet against the digest on a throwaway host answers the same question at the cost of downloading the layers. crane manifest and skopeo inspect are the non-Docker equivalents.

What a `/var/lib/docker` copy is worth

The honest assessment

Copying /var/lib/docker feels like the thorough option. Three things limit its value.

It is inconsistent unless the daemon is stopped. The daemon writes container state, layer metadata and the containerd content store continuously. A tar or rsync of a live /var/lib/docker captures a torn state that may not load. To get a consistent copy you must stop the daemon — which means the outage you were trying to avoid, before the upgrade rather than during it.

It is coupled to the daemon version. The layout of the image store, the containerd metadata database and the container state files are internal formats. Restoring a copy taken under one version into a different version is not a supported operation. It usually works between patch releases and is a coin flip across feature releases, in the direction you care about — restoring an older copy into a newer daemon after a failed upgrade.

It is large and slow. Tens to hundreds of gigabytes, most of which is image layers you can re-pull in minutes.

When it is worth it: a filesystem-level snapshot — LVM, ZFS, a hypervisor snapshot — taken with the daemon stopped, immediately before a line upgrade you consider risky. That is fast, consistent, and restores the whole host state including the daemon version. It is a rollback mechanism, not a backup.

When it is theatre: tar czf docker-backup.tar.gz /var/lib/docker on a running host. Slow, inconsistent, and unlikely to restore usefully.

Service impact possibleconsistent snapshot— If you take one, take it properly. This stops containers.

# Stop the daemon AND the socket, or systemd restarts it on the next API call
sudo systemctl stop docker.socket docker.service
sudo systemctl stop containerd.service

# Now the tree is quiescent - snapshot at the filesystem layer
sudo lvcreate --snapshot --size 20G --name docker-preupgrade /dev/vg0/docker

sudo systemctl start containerd.service
sudo systemctl start docker.service

docker.socket before docker.service is not optional. The socket unit activates the daemon on demand, so stopping only the service leaves systemd ready to start it again the moment anything touches /var/run/docker.sock — including your own monitoring agent.

What if you skip the capture entirely?

Most of the time, nothing, because most upgrades work. The capture is insurance and insurance usually goes unclaimed.

The claim, when it comes, looks like this: the daemon will not start after the upgrade, the cause is a daemon.json key you cannot remember the previous value of, and the file has been overwritten by a package prompt you answered without reading. You now need to reconstruct the daemon configuration of a production host from memory while it is down.

Or: the rollback works, the daemon starts, and three containers do not come back because they were created by hand two years ago with flags nobody recorded. The Compose file covers the other nine. Those three are the ones that took the site down.

Two minutes, a few kilobytes, and neither of those is possible.

The five artefacts

Capturing them

The image digest inventory is the valuable one

What a /var/lib/docker copy is worth

Knowledge check

What a `/var/lib/docker` copy is worth