Immutable tag fallback — when :latest is the only option

A registry outage freezes every deploy that pulls a fresh artifact. The digest-pinned deploys — the ones that reference an image by sha256:abc... — hold the line because the kubelet resolves them from the node’s image cache. The tag-pinned deploys — the ones that reference :v1.2.3 or :latest — are the ones that block. The fallback is to use a mutable tag when no digest-pinned path is available, and the production discipline is to do so under rules that prevent the fallback from becoming the default.

flowchart LR
    A["registry outage"] --> B["digest pin serves from cache"]
    A --> C["tag pin fails"]
    C --> D["fallback: :latest"]
    D --> E["non-critical only"]
    D --> F["audit log per pull"]
    D --> G["exit when registry returns"]
    B --> H["critical workloads hold"]
    E --> I["scope limit"]
    F --> J["rollback evidence"]
    G --> K["fallback closed"]

When :latest is the only option

The conditions under which a mutable tag fallback is acceptable:

• The registry is unreachable for the team’s digest-pinned path. The cross-region replica is also down, the pull-through cache is cold, and the kubelet’s image cache is missing the digest the deploy needs.
• The workload is non-critical. A logging sidecar, a debug dashboard, a staging deployment — not a production payment service, not the control plane, not the signature-verification admission controller.
• The fallback is bounded. The team has a written exit condition — “exit the fallback when the registry returns” — and a deadline — “exit the fallback within four hours regardless”.

When all three conditions are met, the fallback is the bridge that keeps non-critical workloads alive while the primary registry recovers. When any one condition is violated, the fallback is a workaround that has outlived its window.

The rules for using mutable tags under failure conditions

The production rules — five of them — that govern the fallback:

1. Scope. The fallback applies to non-critical workloads only. Production-critical services stay on digest pins and absorb the deploy freeze.
2. Audit. Every pull against a mutable tag is logged with the resolved digest, the timestamp, the requesting workload, and the operator who authorised the fallback. The audit log is the evidence that the fallback was used and the proof of what ran.
3. Verification. The mutable tag is verified with crane manifest digest before the fall-back pull, so the resolved digest is recorded before the pull.
4. Bounded duration. The fallback has a written deadline — four hours is a common default — past which the fallback is closed regardless of registry state.
5. Exit condition. The fallback exits the moment the primary registry returns. The team does not wait for the deadline if the registry recovers first.

Verifying the fallback tag

The mutable tag is verified before the pull, so the resolved digest is recorded before the image lands on the node.

# 1. Read the digest the tag resolves to
DIGEST=$(crane manifest digest registry.example.com/app:latest)
echo "$DIGEST"

# 2. Pull by digest — not by tag — so the kubelet uses
#    the content-addressed path, not the mutable tag
docker pull registry.example.com/app:$DIGEST

# 3. Tag the pulled image locally so the workload
#    references the verified digest
docker tag registry.example.com/app:$DIGEST \
    registry.example.com/app:fallback-$TIMESTAMP

The pull-by-digest is the production primitive. The tag :latest is the discovery mechanism; the digest is the identity. A pull that resolves the tag, records the digest, and pulls by digest is a pull that the audit log can trace.

# The full fallback sequence
TIMESTAMP=$(date -u +%FT%TZ)
DIGEST=$(crane manifest digest registry.example.com/app:latest)
docker pull registry.example.com/app:$DIGEST
docker tag registry.example.com/app:$DIGEST \
    registry.example.com/app:fallback-$TIMESTAMP
echo "fallback pull $TIMESTAMP digest=$DIGEST" \
    >> /var/log/registry-fallback.log

What digest pins buy during the outage

Digest pins hold the line because the kubelet resolves the digest against the node’s image cache. A pod that references registry.example.com/app:$DIGEST looks up the digest in the cache; if the cache has the layers, the pod starts without touching the registry. A pod that references registry.example.com/app:v1.2.3 requires the registry to resolve the tag, which fails when the registry is unreachable.

The fallback does not apply to digest-pinned workloads. The digest-pinned workloads absorb the deploy freeze for the duration of the outage. The fallback applies to tag-pinned workloads that have no other path.

Production discipline

1. Digest-pinned workloads hold the line during the outage. The fallback is for tag-pinned workloads that have no other path.
2. The fallback is non-critical only. Production payment, control plane, and signature-verification workloads do not enter the fallback.
3. The fallback pulls by digest, not by tag. The tag is for discovery; the digest is the pull reference.
4. Every fallback pull is logged with the resolved digest, the timestamp, and the operator. The audit log is the rollback evidence.
5. The fallback has a deadline and an exit condition. The bridge is closed when the registry returns or when the deadline expires.

Cross-course references

• Git, CI/CD & GitOps — Part XLV-02 (Digests and Content-Addressing) covers the digest pin the fallback relies on.
• Git, CI/CD & GitOps — Part XLV-05 (Immutable Tags and Digest Pinning) covers the immutable-tag discipline the fallback relaxes.
• Git, CI/CD & GitOps — Part XCVII-05 (Recovering the Artifact Registry) covers the recovery the fallback bridges.

Quiz