Skip to main content
RunBook Academy

← All break/fix scenarios in Git, CI/CD & GitOps

advancedcicd-rollback~30 min

Failed rollback (new revision has the same bug)

Reported symptoms

  • ●`argocd app rollback <app>` reports `Sync Status: Synced` and `Health Status: Healthy`
  • ●The cluster is running the same image digest as before the rollback (`kubectl get deploy -o jsonpath={.spec.template.spec.containers[0].image}` returns `<reg>/<repo>@sha256:<digest>` that matches the pre-rollback digest)
  • ●The Application `targetRevision` points at a git commit whose manifest declares `<image>:<tag>` and the tag was overwritten during the forward deploy (the tag was mutated, not the digest)
  • ●`git log -- overlays/prod/<app>/deployment.yaml` shows the rollback commit (`Revert "..."`) referencing the same tag, and the tag in the registry now points at the buggy image
  • ●The Application still exhibits the bug after rollback; customer-impacting symptoms continue
  • ●`argocd app history <app>` shows the rollback as a successful sync; the manifest in git is at the previous commit, but the deployed artifact (image) is unchanged
  • ●`crane manifest <reg>/<repo>:<tag> | jq .config.digest` returns the same digest as the forward deploy
  • ●No immutable-tag protection was in place; the tag was a moving pointer that CI could overwrite

Evidence

  • · `argocd app manifests <app> | yq .spec.template.spec.containers[0].image` returns `<reg>/<repo>:<tag>` (mutable tag)
  • · `crane manifest &lt;reg&gt;/&lt;repo&gt;:&lt;tag&gt; | jq .digest` returns the same `sha256:<...>` as `kubectl get deploy -o jsonpath={.spec.template.spec.containers[0].image}` after the rollback
  • · `git -C gitops-repo show HEAD:overlays/prod/&lt;app&gt;/deployment.yaml` references the same tag as the buggy forward deploy; the manifest was reverted but the tag was not
  • · `argocd app history &lt;app&gt;` shows the rollback as a sync to the previous commit, with `Health: Healthy` after the sync — the controller does not check that the image digest changed
  • · `gh api repos/&lt;org&gt;/&lt;repo&gt;/actions/runs?per_page=5` shows a CI run that pushed `&lt;reg&gt;/&lt;repo&gt;:&lt;tag&gt;` with the same tag during the forward deploy
  • · The CI pipeline push step did NOT use an immutable tag (`&lt;git-sha&gt;` or `&lt;short-sha&gt;`); it used a moving tag (`latest`, `prod`, `&lt;version&gt;`) that CI overwrote
  • · `regctl manifest ls &lt;reg&gt;/&lt;repo&gt;` shows tags that were pushed with the same name and different digests in the last 24 hours, confirming tag overwrite
  • · `gh attestation verify &lt;image&gt;@sha256:&lt;digest&gt;` returns provenance referencing the same buggy build
Diagnosis and resolutionclick to reveal

Root cause

Rollback restored the git manifest to a previous commit, but the manifest referenced a mutable tag (`latest`, `prod`, `&lt;version&gt;`) that the CI pipeline had overwritten during the forward deploy. The rollback reverted the manifest but did not revert the image content, because the registry tag pointed at the same buggy digest. The structural failure is the use of mutable tags in production: a mutable tag is a moving pointer that resolves at pull time, and a rollback that points the manifest at the same tag is a rollback that resolves to the same image. Argo CD can only roll back what the manifest says; if the manifest says `latest`, Argo CD cannot know that the previous sync and this sync point at different digests, because the tag itself hides the difference.

Remediation

Stop the rollback — it did not work. Roll forward to a known-good image: build a new commit that explicitly references the last-known-good digest, push the new image to a new tag (`&lt;reg&gt;/&lt;repo&gt;:rollback-&lt;timestamp&gt;`), and update the manifest to reference that tag. The manifest must reference the digest (`&lt;reg&gt;/&lt;repo&gt;@sha256:&lt;known-good-digest&gt;`) for the rollback to be robust against future tag overwrites. Then fix the structural failure: move production to immutable tags. Configure the CI pipeline to push images with a tag that is a function of the git commit (`&lt;reg&gt;/&lt;repo&gt;:sha-&lt;short-sha&gt;`, `&lt;reg&gt;/&lt;repo&gt;:&lt;semver&gt;+&lt;short-sha&gt;`), and update the GitOps manifest to reference that tag. Pin the manifest to the digest (`@sha256:...`) so the manifest itself is immutable: a future push that re-tags the image cannot change what the cluster pulls. Add an admission policy (Kyverno/OPA) that rejects any Production manifest whose image reference is a tag-only mutable reference; the policy must require `@sha256:...` for any `image:` field in a Production overlay. Then audit every Application: every mutable tag in a Production overlay must be converted to a digest-pinned reference.

Verification

The cluster is running the known-good digest: `kubectl get deploy -o jsonpath={.spec.template.spec.containers[0].image}` returns `&lt;reg&gt;/&lt;repo&gt;@sha256:&lt;known-good-digest&gt;`. The Application is `Synced` and `Healthy`, and the bug is no longer reproducible. `git diff HEAD~1 HEAD -- overlays/prod/&lt;app&gt;/` shows the image line now references the digest, not the tag. `crane manifest &lt;reg&gt;/&lt;repo&gt;:&lt;tag&gt;` returns the digest, and a fresh `crane pull` resolves to the same digest the cluster is running. The admission policy rejects a synthetic PR that re-introduces a mutable tag in a Production overlay. The CI pipeline pushes images with a `&lt;sha&gt;-&lt;short-sha&gt;` tag and the manifest references the digest; a re-push of the same short-sha with a new image produces a different digest, and the manifest is unchanged.

Prevention

Mutable tags are an anti-pattern in production. Promote images by digest, not by tag. The CI pipeline builds images and pushes them with an immutable tag (`sha-&lt;git-sha&gt;`) and the manifest pins the digest (`@sha256:<...>`). The manifest is the only thing the GitOps controller reverts on rollback, and a manifest that references a digest is unambiguous: revert to a previous manifest, and the digest referenced by that manifest is the digest the cluster pulls. Enforce this structurally with an admission policy that rejects any Production manifest whose `image` reference is tag-only: the policy must require `@sha256:&lt;digest&gt;` for any image in a Production overlay. Audit the registry quarterly: any tag whose manifest digest has changed in the audit window is a violation. Drill the rollback: simulate a buggy forward deploy, perform a `argocd app rollback`, and verify the cluster is running a different digest. The principle is that rollback is a manifest change, and a manifest that resolves at pull time is not a manifest that can be rolled back.

A rollback reverts the manifest, not the image. If the manifest references a mutable tag, the rollback reverts to a manifest that points at the same image — and the bug returns. Promote images by digest in production, pin the manifest to the digest, and enforce the pattern with an admission policy. The rollback that works is the one where the previous manifest referenced a different digest than the current manifest; if both reference the same tag, the rollback did not happen.