Immutable digests — why tags are not enough · Docker & Containers

Tags are mutable. Digests are not. This is the most important property of a Docker registry, and the one whose consequences are least often thought through.

docker pull nginx:1.27
# Whatever image the publisher has assigned to 1.27, right now

docker pull nginx@sha256:0000000000000000000000000000000000000000000000000000000000000000
# Always the exact image whose manifest hashes to that value

The interesting question is not “which is safer” — everyone knows the answer. It is “what does it actually look like when a tag moves under a running system”, because the answer is not an error message.

What a moved tag does to a running fleet

Suppose registry.example.com/api:1.4.0 is deployed to twelve hosts. Somebody rebuilds and re-pushes 1.4.0 — a hotfix, a CI re-run, a latest-style convenience tag on a release branch.

Nothing happens. That is the problem.

Running containers do not consult the registry. Each of the twelve hosts keeps running the image it already has, because a container is bound to an image ID at create time, not to a tag. The new image arrives one host at a time, whenever something causes that host to re-create the container:

A restart: unless-stopped container after a host reboot — the container is re-created only if it was removed, so often not.
A deploy of an unrelated service that runs docker compose up -d for the whole file.
An autoscaler adding a node.
An operator running docker compose pull && docker compose up -d because a log looked odd.

So over the following week the fleet drifts into two populations. Some hosts serve the old bytes, some the new, both reporting api:1.4.0 in every dashboard, every docker ps, and every log line.

The bug that only happens on some requests

The report is always the same shape: an intermittent failure with no pattern. One in four checkouts fails. A header is missing sometimes. A metric has two clusters of latency. It never reproduces in staging and it never reproduces on the host you happen to SSH into.

The first diagnosis is a race condition in the application, and a team can lose a week to that. The second is a load balancer or a caching layer. Both are wrong, and both are reasonable, because every tool you would normally reach for reports that all twelve hosts are running api:1.4.0 — which is true, and useless.

The tell is that the failure correlates with the host, not with the request. Once someone thinks to bucket the errors by container ID instead of by endpoint, the pattern is instant and obvious.

There is no error, no alert and no log entry for this condition, because from the daemon’s point of view nothing went wrong. A tag pointed somewhere; now it points somewhere else. That is what tags are for.

The one command that finds it

Read-only / Safefleet digest census— Run on every host and collect. More than one distinct image ID for a single tag is a split fleet.

docker ps --format '{{.Names}}' | while read -r c; do
docker inspect --format '{{.Name}} {{.Config.Image}} {{.Image}}' "$c"
done

/api registry.example.com/api:1.4.0 sha256:4f2a...c19d
/web registry.example.com/web:1.4.0 sha256:9b71...02af

Illustrative output

.Config.Image is the reference as written at create time — the tag. .Image is the image ID the container is actually running. Collect the pair from every host and group by tag:

Read-only / Safeaggregated across twelve hosts— Two distinct IDs for one tag. This is the whole diagnosis.

$ sort fleet-census.txt | awk '{print $2, $3}' | sort | uniq -c

      7 registry.example.com/api:1.4.0 sha256:4f2a...c19d
    5 registry.example.com/api:1.4.0 sha256:81e0...77b3
   12 registry.example.com/web:1.4.0 sha256:9b71...02af

Illustrative output

Seven hosts on one image, five on another, one tag. If you take one check away from this lesson, take this one: the count of distinct image IDs per deployed tag should be exactly one, and it is trivial to assert.

Finding the digest to pin

There are two digests for a multi-platform image and picking the wrong one is the usual mistake.

Read-only / Safeindex digest and platform digests— The top-level Digest is the index. The entries beneath it are the per-platform manifests, each with its own digest.

docker buildx imagetools inspect docker.io/library/alpine:latest

Name:      docker.io/library/alpine:latest
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300

Manifests:
Name:      docker.io/library/alpine:latest@sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3
MediaType: application/vnd.docker.distribution.manifest.v2+json
Platform:  linux/amd64

Name:      docker.io/library/alpine:latest@sha256:e047bc2af17934d38c5a7fa9f46d443f1de3a7675546402592ef805cfa929f9d
MediaType: application/vnd.docker.distribution.manifest.v2+json
Platform:  linux/arm/v6

Pin the index digest — the Digest: at the top — when you want one reference that works on every architecture in your fleet. Pin a platform manifest digest only when you deliberately want to force one architecture, and be aware that the reference will then fail to run on any other.

Pinning in Compose

services:
  api:
    # api:1.4.0 — keep the human-readable version in a comment
    image: registry.example.com/api@sha256:4f2ac19d0000000000000000000000000000000000000000000000000000c19d
    labels:
      org.opencontainers.image.version: "1.4.0"

pull_policy: always does not mean 'fail if not local'

A common misreading, and it matters because it changes what happens during a registry outage.

Compose’s pull_policy values, per the Compose specification:

Value	Behaviour
`always`	Compose always pulls the image from the registry
`missing`	Pulls only if the image is absent from the local cache (the default)
`never`	Never pulls; relies on the local cache
`build`	Builds the image, rebuilding if it is already present
`daily`, `weekly`, `every_<duration>`	Checks the registry if the last pull was longer ago than the interval

if_not_present is a backwards-compatible alias for missing.

So always makes the registry a hard dependency of every up. It is the right setting when you want to guarantee freshness, and the wrong one when you want a deploy to survive a registry outage. The specification also notes that “the latest tag is always pulled even when the missing pull policy is used” — another reason not to deploy latest.

With a digest pin, always is close to free: the registry confirms the digest exists and the daemon transfers nothing, because content addressed by a digest it already holds cannot have changed.

Two lookups, one of which is mutable

Resolving registry.example.com/api:1.4.0 is two registry operations, and only the first one is mutable.

Lookup one — the tag index. GET /v2/api/manifests/1.4.0 asks the registry which manifest that tag currently names. The registry answers with the manifest body and a Docker-Content-Digest header giving its digest. Re-pointing the tag is a single push: the registry writes a new entry in its tag index. Nothing is versioned, nothing is logged in the image, and the old manifest is not touched.

Lookup two — content addressing. Everything from the manifest downwards is addressed by the SHA-256 of its own bytes. The manifest lists the config blob and each layer blob by digest. The manifest’s own digest is a hash over that list. Change one byte of one layer and that layer’s digest changes, so the manifest changes, so the manifest digest changes.

That is the whole guarantee, and it is why it is worth stating precisely: a digest does not mean “the registry promised not to change it”. It means the registry cannot change it without producing a different digest, and a client that fetched by digest verifies the bytes it received against the digest it asked for. A malicious registry serving different content under the same digest is caught by arithmetic, not by policy.

It also explains why signatures attach to the digest. cosign sign on a tag would be a signature over a pointer that somebody else controls.

Note the layer blobs are shared across manifests by digest, which is why re-pushing a rebuilt image usually transfers only the changed layers, and why the moved-tag failure is so cheap for an attacker or so easy to do by accident.

Keeping pins current

The objection to digest pinning is real: an unreadable 64-hex string that no human will ever update means you stop getting security rebuilds.

The answer is to make bumping the pin a routine automated change rather than a manual one. Whatever tool you use, the shape is the same: resolve the tag to a digest, open a change, let CI verify it, merge.

Read-only / Saferesolve the current digest for a set of tags— Emits digest-pinned references for review. Feed the output into a pull request rather than applying it directly.

for ref in \
docker.io/library/nginx:1.27 \
docker.io/library/postgres:16 \
; do
DIGEST=$(docker buildx imagetools inspect "$ref" \
  | awk '/^Digest:/ {print $2; exit}')
printf '%s -> %s@%s\n' "$ref" "${ref%%:*}" "$DIGEST"
done

docker.io/library/nginx:1.27 -> docker.io/library/nginx@sha256:0000...0000
docker.io/library/postgres:16 -> docker.io/library/postgres@sha256:1111...1111

Illustrative output

The point of routing it through review is that the digest bump becomes the artefact you can point at afterwards. “Which base image change introduced this” is answerable from your own history instead of from the publisher’s.

What if the registry is replaced or compromised?

Digest pinning changes the failure from silent substitution to noisy unavailability, which is a large improvement and not a complete defence.

What it stops. An attacker with push access to the registry cannot make your pinned reference resolve to their image. They would have to find a SHA-256 preimage, and the client verifies the received bytes against the requested digest, so a substituted blob is rejected locally rather than trusted on the registry’s say-so.

What it does not stop. Denial of service. The attacker — or a disk failure, or an over-eager garbage collection — can delete the blobs your digest names, and then every host that does not already have them is stuck. Your running containers keep running; your next deploy, your next new node and your next host rebuild do not.

What it says nothing about. Whether the image was safe in the first place. A digest pin guarantees you keep getting the same image. If that image was backdoored at build time, you now get the backdoor reproducibly.

The mitigations are the boring ones: mirror the digests you depend on into a second registry, keep a pull-through cache with a long TTL so warm content survives an upstream outage, and pair digest pinning with signature verification so “the same image” is also “an image from a publisher I trust”.

What a moved tag does to a running fleet

The one command that finds it

Finding the digest to pin

Pinning in Compose

Keeping pins current

Knowledge check