Repository setup and version pinning

Installing the newest Docker is fine for a laptop. On a fleet it is a decision you make once and then cannot unmake, because a fleet with unpinned versions has no answer to the two questions that matter during an incident: what changed, and is this host the same as that host.

Adding the official apt repository

sudo apt-get update
sudo apt-get install -y ca-certificates curl

sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg \
  -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

sudo tee /etc/apt/sources.list.d/docker.sources >/dev/null <<EOF
Types: deb
URIs: https://download.docker.com/linux/debian
Suites: $(. /etc/os-release && echo "$VERSION_CODENAME")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt-get update

Signed-By is the part that matters. It scopes that key to that repository only, so a compromise of the Docker key cannot be used to sign a package that apt accepts as coming from Debian. The older pattern of dropping keys into /etc/apt/trusted.gpg.d/ trusted every key for every repository, and is why that directory is deprecated.

Verify the key before you trust it

Downloading a key over TLS proves you reached download.docker.com. It does not prove you reached the real one — that is what the fingerprint is for.

$ gpg --show-keys /etc/apt/keyrings/docker.asc

pub   rsa4096 2017-02-22 [SCEA]
    9DC858229FC7DD38854AE2D88D81803C0EBFCD88
uid                      Docker Release (CE deb) <docker@docker.com>
sub   rsa4096 2017-02-22 [S]

Compare 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 against the value Docker publishes. This check belongs in your provisioning code, not in a human’s memory:

EXPECT=9DC858229FC7DD38854AE2D88D81803C0EBFCD88
GOT=$(gpg --show-keys --with-colons /etc/apt/keyrings/docker.asc \
    | awk -F: '/^fpr:/ {print $10; exit}')

if [ "$GOT" != "$EXPECT" ]; then
  echo "Docker signing key fingerprint mismatch: got $GOT" >&2
  exit 1
fi
echo "signing key verified"

Reading a Docker version string

Before you can pin, you have to be able to read what you are pinning to.

$ apt-cache madison docker-ce | head -5

docker-ce | 5:28.5.2-1~debian.12~bookworm | https://download.docker.com/linux/debian bookworm/stable amd64 Packages
docker-ce | 5:28.5.1-1~debian.12~bookworm | https://download.docker.com/linux/debian bookworm/stable amd64 Packages
docker-ce | 5:28.4.0-1~debian.12~bookworm | https://download.docker.com/linux/debian bookworm/stable amd64 Packages
docker-ce | 5:28.3.3-1~debian.12~bookworm | https://download.docker.com/linux/debian bookworm/stable amd64 Packages
docker-ce | 5:28.3.2-1~debian.12~bookworm | https://download.docker.com/linux/debian bookworm/stable amd64 Packages

Pinning: three mechanisms, one right answer per situation

apt-mark hold — the interactive brake

sudo apt-mark hold docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin

apt-mark showhold

Hold is a flag in the local dpkg database. It is easy, it is what most people reach for, and it has two properties that matter on a fleet.

It is invisible: nothing in your configuration repository records it. A rebuilt host has no hold, and drifts. And it is blunt — it prevents any change, so the host also stops receiving the security update you did want, and a plain apt upgrade will report packages “kept back” without explaining why.

Hold is right for a host you are actively debugging, or as a belt-and-braces addition to a declared version. It is not a fleet strategy on its own.

An apt preferences pin — the declarative version

# /etc/apt/preferences.d/docker-ce.pref
Package: docker-ce docker-ce-cli
Pin: version 5:28.5.2-1~debian.12~bookworm
Pin-Priority: 1001

A priority above 1000 permits a downgrade as well as blocking an upgrade, which is what makes this a rollback mechanism and not just a brake. The file is a file: it lives in Git, it deploys with the host, and a rebuilt host comes back with the same pin.

Just install the exact version

The simplest form, and the one Docker’s own documentation uses:

VERSION_STRING=5:28.5.2-1~debian.12~bookworm

sudo apt-get install -y \
  docker-ce="$VERSION_STRING" \
  docker-ce-cli="$VERSION_STRING" \
  containerd.io docker-buildx-plugin docker-compose-plugin

$ docker version --format 'client={{.Client.Version}} server={{.Server.Version}} api={{.Client.APIVersion}}/{{.Server.APIVersion}}'

client=28.5.2 server=28.1.1 api=1.51/1.49

Hold the whole set, or hold none of it. The five packages are a unit.

RPM systems

sudo dnf install -y python3-dnf-plugin-versionlock

VERSION_STRING=3:28.5.2-1.el9
sudo dnf install -y \
  docker-ce-"$VERSION_STRING" \
  docker-ce-cli-"$VERSION_STRING" \
  containerd.io docker-buildx-plugin docker-compose-plugin

sudo dnf versionlock add docker-ce docker-ce-cli
dnf versionlock list

The controlled upgrade

1. Read the release notes for every version between the one you run and the one you want. Docker publishes them per major line; skipped releases are the case live restore explicitly does not cover.
2. Confirm the target is a patch bump within the same major line if you intend to keep containers running. A major upgrade is a container-restarting change.
3. Confirm live restore is already active: docker info --format "{{.LiveRestoreEnabled}}". Enable it with systemctl reload docker if not — that costs nothing.
4. Record the current state: docker version, docker info, dpkg -l | grep -E "docker|containerd", and the PID of a representative container.
5. Release the hold, install the exact target version, re-apply the hold. Do not run a bare apt upgrade.
6. Verify: client and server versions match your intent, the container PID you recorded is unchanged, and the application answers a real request.
7. Roll one host. Wait long enough to see a full traffic cycle. Only then roll the rest.

TARGET=5:28.5.2-1~debian.12~bookworm

sudo apt-mark unhold docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin
sudo apt-get update
sudo apt-get install -y docker-ce="$TARGET" docker-ce-cli="$TARGET"
sudo apt-mark hold docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin

docker version --format 'client={{.Client.Version}} server={{.Server.Version}}'

The rollback is the same command with the previous version string, which is why recording it in step 4 is not busywork. Docker’s repository retains old releases, so a downgrade is normally available — but confirm with apt-cache madison before you rely on it, not during the incident.

Verification that can fail

WANT_VERSION=28.5.2
GOT=$(docker version --format '{{.Server.Version}}')

if [ "$GOT" != "$WANT_VERSION" ]; then
  echo "engine drift: running $GOT, expected $WANT_VERSION" >&2
  exit 1
fi

if ! apt-mark showhold | grep -qx docker-ce; then
  echo "docker-ce is not held; this host will drift" >&2
  exit 1
fi

echo "engine $GOT pinned and held"

apt-cache policy docker-ce is the companion for a human: it shows the installed version, the candidate, and which repository each came from, which is the fastest way to see that a host is silently tracking the wrong suite.

The same principle, one level up: images

Everything above applies to the Docker Engine package. The identical argument applies to the images you run, and the mechanism is a digest rather than a version string.

IMAGE=nginx:1.27
docker pull "$IMAGE"
docker image inspect --format '{{index .RepoDigests 0}}' "$IMAGE"

That prints something of the form nginx@sha256: followed by 64 hex characters. Using that reference in a Compose file or a docker run binds you to exactly that content, whatever the publisher later does to the tag.

Knowledge check