Tags — mutable references to digests · Docker & Containers

A tag is a string a publisher attaches to a digest in their registry. That is the entire definition. It carries no version semantics, nothing validates it, and the person who set it can change it at any moment without telling you.

Every problem in this lesson follows from the gap between that reality and the way a tag looks in a Compose file — where nginx:1.27 sits in exactly the position a version number sits, and behaves nothing like one.

What a tag is to the daemon

A tag is a registry-side pointer, resolved once

There are two distinct things called tagging, and conflating them causes real confusion.

Locally, docker tag adds a name to an image that is already on the host. It copies nothing and creates no layer — it writes an entry in the daemon’s reference store pointing a name at an existing image ID. Ten tags on one image consume the disk of one image.

Remotely, a tag is a mapping the registry maintains from a name to a manifest digest. docker push myapp:1.0.0 uploads any missing blobs and then sets that mapping. Pushing again with different content and the same tag rewrites the mapping — the old digest is still there, now unnamed, until garbage collection.

The resolution happens exactly once, at pull time. After that the tag is out of the picture entirely:

docker pull nginx:1.27 asks the registry what 1.27 means right now and fetches that digest.
docker create records the resolved image, and docker inspect shows .Image as a hash, not as the tag you typed.
docker start and docker restart replay the stored specification. Neither re-reads the tag, so neither can pick up a moved tag. This is the entire mechanism behind “I deployed and nothing changed”.

There is no such thing as “a container running the 1.27 tag”. There are only containers running digests, and a tag that pointed at one of them at some moment in the past.

Tag conventions, and how far to trust them

Tag style	Stable in practice?	Notes
`nginx:1`	No	Tracks the whole major line. Moves on every minor release
`nginx:1.27`	No	Moves on every patch release within the minor
`nginx:1.27.4`	Usually	Rarely re-pointed, but nothing prevents it
`nginx:1.27.4-alpine`	Usually	Same, plus a base-image variant that can be rebuilt
`nginx:latest`	No	See below
`nginx:stable`, `nginx:mainline`	No	Rolling channel names by design
`nginx@sha256:...`	Yes	Content-addressed. Cannot change

The middle rows deserve a caveat that people skip. Even a fully-qualified patch tag like 1.27.4-alpine is commonly rebuilt — same application version, new base image, because the base picked up a security fix. That is the publisher doing exactly the right thing, and it still means the digest under that tag changed without the name changing. Most Docker Official Images are rebuilt on a schedule for precisely this reason.

So “pin to a patch version” is a real improvement over latest and is not the same thing as a pin. Only a digest is a pin.

Why `latest` is worse than a badly chosen tag

latest is not special to the daemon in any way that means “newest”. It is an ordinary string. It is special in four ways that are all operational.

1. It is the implicit default. docker run nginx means nginx:latest. So does image: nginx in a Compose file. Nobody chose it; they just did not type a tag, which means latest ends up in production without a decision ever being made about it.

2. Compose treats it differently from every other tag. The Compose specification states plainly: “The latest tag is always pulled even when the missing pull policy is used.” missing is the default policy. So a service on latest re-resolves from the registry on every docker compose up, while a service on 1.27 uses the cached image. Two services in the same file, two different caching behaviours, and nothing on the page says so.

3. It has no agreed meaning. Some publishers point it at the newest stable release. Some point it at the newest build of any kind, including release candidates. Some point it at whatever their CI last pushed. Some tagged it once in 2019 and never again, so latest is the oldest image in the repository. There is no way to tell which, short of reading the publisher’s documentation.

4. It destroys the ability to roll back. This is the one that costs a night.

The rollback that had nothing to roll back to

The sequence is always the same, and it is a bad deploy compounded by a naming problem.

Production runs myapp:latest. It has for two years and it has been fine.
CI pushes a broken build and moves latest to it.
The deploy pulls, containers are recreated, the service breaks.
Somebody says “roll back”, and the room discovers there is no name for the previous image. latest is the new one. The old digest is still in the registry, unnamed, and nobody wrote it down.

At this point the recovery paths are all bad. You can hunt the digest out of a host that has not pulled yet — docker image inspect on any node that missed the deploy still knows it, which is why you should look before restarting anything. You can search the registry’s tag list for a versioned tag pointing at the same content, if CI happened to push one. You can rebuild from the previous commit, which is twenty minutes you do not have.

Or you can find the digest is gone entirely, because the registry garbage collected an untagged manifest.

The prevention is one line in CI: always push an immutable tag alongside the mutable one. Tag every build with the commit SHA or a build number as well as latest. That costs nothing — the blobs are already uploaded, and a second tag is a pointer — and it means “roll back” is always a name you already have.

Configuration changeCI tagging— One build, two tags. The immutable one is what production references.

SHA=$(git rev-parse --short HEAD)
REPO=registry.example.com/myorg/myapp

docker buildx build \
--tag "$REPO:$SHA" \
--tag "$REPO:latest" \
--push \
.

# Record what production should reference. This line is the rollback plan.
crane digest "$REPO:$SHA"

Detecting tag drift across a fleet

The point of understanding all this is being able to answer one question during an incident: is every host running the same image?

Read-only / Safedrift check— Compare what each container is bound to against the digest the tag resolves to now. Exits non-zero on drift.

DRIFT=0

for id in $(docker ps -q); do
  name=$(docker inspect --format '{{.Name}}' "$id")
  ref=$(docker inspect --format '{{.Config.Image}}' "$id")
  bound=$(docker inspect --format '{{.Image}}' "$id")
  local_now=$(docker image inspect --format '{{.Id}}' "$ref" 2>/dev/null || echo "absent")

  if [ "$bound" != "$local_now" ]; then
      echo "DRIFT $name ref=$ref bound=$bound tag-now=$local_now" >&2
      DRIFT=1
  fi
done

[ "$DRIFT" -eq 0 ] && echo "every running container matches its tag on this host"
exit "$DRIFT"

That compares the container to the local image store. To compare the host to the registry — the check that catches “this host has not pulled since Tuesday” — resolve the tag remotely without pulling:

Read-only / Saferemote drift— What the registry says the tag means right now, versus what this host has.

IMAGE=registry.example.com/myorg/myapp:1.4

REMOTE=$(crane digest "$IMAGE")
LOCAL=$(docker image inspect --format '{{if .RepoDigests}}{{index .RepoDigests 0}}{{end}}' "$IMAGE" \
      | sed 's/.*@//')

echo "registry: $REMOTE"
echo "host:     $LOCAL"
[ "$REMOTE" = "$LOCAL" ] || echo "this host is running a superseded image" >&2

Run that across the fleet and group the results. If it returns more than one distinct digest, you have a fleet running two versions of one name, and every “it reproduces on some nodes” report you have makes sense.

Choosing a pinning strategy

Pin proportionately. The strongest option is not always worth its friction.

services:
  # Dev: mutable, convenient, re-resolved on every up.
  scratchpad:
    image: nginx:latest

  # Most teams, most services: a patch tag plus a recorded digest in
  # the change ticket. Readable in review, and you can still audit.
  web:
    image: nginx:1.27.4

  # Anything in the request path, anything handling data, anything
  # you would have to explain in an incident review.
  payments:
    image: registry.example.com/myorg/payments@sha256:0c1a2f9b7d3c4e5a6f7089abcdef0123d5f28ef21aabd54d6a48d8b9d3b8e5b1

A pure-digest manifest is unreadable in a pull request — nobody can tell sha256:0c1a... from sha256:0c1b... at a glance. The usual answer is a comment carrying the human-readable version next to the digest, or a digest-resolving step in CI that rewrites tags to digests at deploy time. Both work. What does not work is deciding digests are too awkward and going back to tags for the payments service.

What happens if you re-tag an image locally and push it?

docker tag locally is free and creates no data. Pushing is where the effects are, and there are two worth predicting.

Pushing an unchanged image under a new tag transfers almost nothing. The blobs already exist in the registry, so the push is a few HEAD requests and a manifest PUT. This is why tagging a release from an existing build is instant — and why “the push was suspiciously fast” is not evidence that something went wrong.

Pushing changed content under an existing tag rewrites the pointer silently. There is no confirmation, no warning, and no record in the registry of what the tag used to mean. The previous digest becomes untagged content, still pullable by digest for as long as retention keeps it.

The compounding effect people miss: because a tag is just a pointer, two tags can point at the same digest, and deleting one does not delete the content. But on a registry configured to garbage collect untagged manifests, deleting the last tag pointing at a digest does make that content collectable — including content some deployment somewhere pinned by digest. That is the mechanism behind a digest pin that worked for a year and then started returning a manifest error.

The habit that prevents all of it: treat a release tag as write-once. Push it, never move it, and let the mutable convenience tags (latest, stable, edge) be the only ones that ever change.

Why this matters

Tag drift is the most common supply-chain incident there is, and it almost never presents as a supply-chain incident. It presents as a bug that reproduces on some nodes and not others, or a deploy that changed nothing, or a rollback with nowhere to roll back to. In every case the underlying fact is the same: a name in somebody else’s database moved, and nothing on your side noticed.

The fix is not primarily technical. It is a habit: every artefact that reaches production gets an immutable name at build time, and the deployment references content rather than a name somebody else controls. That costs one extra --tag in CI and one digest in a manifest.

What it buys is that “roll back” is always available, “is every host the same?” is always answerable, and every scan, signature and SBOM you produce actually describes the thing that is running.

What a tag is to the daemon

Tag conventions, and how far to trust them

Why latest is worse than a badly chosen tag

Detecting tag drift across a fleet

Choosing a pinning strategy

Knowledge check

Why `latest` is worse than a badly chosen tag