Application rollback — promoting the previous digest, the registry as the source of truth

Application rollback is the simplest of the five mechanisms and the most often misunderstood. A team that says “roll back the application” means “point the Deployment at the previous digest”. The digest is what runs; the tag is the handle the team used to find it. The registry holds the bytes; the manifest pins the digest; the cluster pulls. Rolling back is editing the pin and letting the cluster re-pull.

How application rollback works

A Deployment references an image. The reference can be a tag (api:v1.4.2) or a digest (api@sha256:9f2b...). The two are not equivalent.

flowchart LR
    A["Manifest"] -->|"image: api:v1.4.2"| B["Tag resolves at pull"]
    B --> C["Latest bytes for that tag"]
    A -->|"image: api@sha256:9f2b..."| D["Digest is the identity"]
    D --> E["Exactly these bytes"]

A tag is mutable. A digest is not. A reference by tag asks the registry, at pull time, “what do you have for this name?”; a reference by digest asks “do you still have these bytes?”. The second question is the only one a rollback can answer.

The rollback procedure:

1. Identify the previous digest the Deployment was pinning. The previous commit holds the value; git log and the Deployment’s rollout history name it.
2. Edit the manifest to point at the previous digest.
3. Commit the change. Let the GitOps controller sync, or apply with kubectl apply -f deployment.yaml.

The cluster pulls the bytes from the registry, the new ReplicaSet comes up with the previous digest, and the rollout completes. The bytes that come down are exactly the bytes the previous digest identified, because the registry is content-addressed: the same digest means the same bytes, regardless of what tags point at them.

The registry is the source of truth

The registry does not forget. A digest, once uploaded, is identified by the SHA-256 of its bytes. Re-uploading bytes under the same digest is a no-op; uploading different bytes under the same digest is rejected. The registry’s immutability is what makes the rollback possible: the team can ask, six months later, “do you still have sha256:9f2b…?” and the registry answers yes or no with no ambiguity.

This is why digest pinning matters. A team that deploys api:latest has no way to ask “what was running three deploys ago?”; the tag is whatever someone pushed most recently. A team that deploys api@sha256:9f2b... knows exactly what is running and can ask the registry to confirm that the bytes are still there.

kubectl set image deployment/$NAME api=api@sha256:$PREVIOUS_DIGEST
kubectl annotate deployment/$NAME \
  kubernetes.io/change-cause="rollback to api@sha256:$PREVIOUS_DIGEST"

The kubernetes.io/change-cause annotation tracks the change in the Deployment’s revision history (the old --record flag that once set it was removed in kubectl 1.33). A more GitOps-native approach is to edit the manifest, commit, and let Argo CD or Flux sync.

What application rollback can and cannot do

The boundary the application rollback crosses is the container image. What it can do:

• Restore the previous bytes the application runs.
• Roll forward without rebuilding: the registry still holds the digest; the cluster pulls and runs.

What it cannot do:

• Undo database writes the previous application’s brief run produced. The bytes are rolled back; the data they wrote is not.
• Restore a deleted digest. If the registry’s garbage collection has removed the previous digest, the rollback fails: the cluster cannot pull bytes that no longer exist.
• Roll back a multi-image Deployment partially. A Deployment that runs api and worker as two containers requires two digest edits; rolling back one and not the other leaves them mismatched.

When application rollback fails

The failure modes:

• Garbage-collected digest. The registry has a retention policy. The previous digest was pruned because no tag pointed at it and no Deployment pulled it for N days. The rollback is impossible without rebuilding the previous image from the previous commit.
• Overwritten tag. The team uses a CI pipeline that re-tags latest on every build. The previous tag’s bytes may have been replaced. The digest is the only thing that survives; pinning by tag is the failure.
• Image removed by registry policy. Compliance or vulnerability scanning may have removed the image. The rollback then requires a rebuild from source.

The defence against all three is the same: digest pinning, plus a registry retention policy that holds the N most recent digests per repository for at least the rollback window the team commits to.

Production discipline

1. Pin by digest, deploy by digest, roll back by digest. Tags are hints for humans; digests are the contract for automation.
2. Set a registry retention policy that holds every digest a Deployment has ever pulled, for the rollback window the team commits to (commonly 30 or 90 days).
3. Record the digest in the rollout history. Use the Deployment annotation or the GitOps sync record; the next rollback needs the previous digest’s exact value.
4. Treat a digest-not-found error as a retention bug, not a deployment bug. If the rollback fails because the digest is gone, the fix is in the registry policy, not in the deployment procedure.

Cross-course references

• This course, Part XLV (Immutable identity) covers content addressing and digest pinning in depth.
• Kubernetes for Production Sysadmins - Part XIV (Workloads) covers image references and the pull-on-rollout mechanism.
• This course, Part LIII (Container supply chain) covers image provenance and the registry as a trust boundary.

Quiz