As neededgitops-rollback

Production Rollback Checklist

16 items ·11 critical ·5 warn ·0 info

How to use this checklist

This is the checklist for a rollback under time pressure. The order is deliberate: classify, identify the target, rehearse the command in your head, then act. A rollback done in the wrong order is a rollback that destroys evidence, drops data, or rolls forward instead of back.

The rehearsed-rollback item is not optional. A team whose first production rollback is the actual production rollback is a team that will lose more during the rollback than the bug cost.

Sign-off

Every critical item must pass before the rollback is declared complete. A failing critical item means the rollback is incomplete and the incident is still open; it is not a note for later. Record the date, the operator, and the disposition of every item that did not pass.

Critical11 items

The failure is classified as something a rollback can fix (bad config, bad manifest, bad image, bad schema). A rollback against a data-corruption incident without a tested data restore is itself a second incident.
A previous-good artifact, commit, or sync state is identified before the rollback command runs. "Rollback to whatever was there before" is not a command; it is a guess.
```
argocd app history &lt;app&gt;; kubectl rollout history deploy/&lt;name&gt;
```
The rollback target is pinned by digest / commit / revision number, not by tag or by a moving ref. The same artifact must be reachable in the registry after the rollback as was there before.
A human with rollback authority (or break-glass) has approved the rollback. The approval is recorded against the change ticket, the chat, or the incident channel — not just spoken.
The exact rollback command has been rehearsed against a non-production cluster of the same shape within the last 90 days. A rollback run for the first time during an incident will go wrong in a new way.
Data implications are reviewed: does the rollback drop or rewrite data, does it require a schema revert, does it lose writes that happened after the bad deploy? A data-safe rollback is a forward-compatible one.
The rollback includes a post-rollback validation step (smoke test, health probe, sync state check, error rate check). A rollback that ends in a broken state is two failures, not one.
```
kubectl rollout status deploy/&lt;name&gt; --timeout=10m; argocd app wait &lt;app&gt; --health
```
If the rollback used the controller (Git revert + sync), Git and the cluster agree afterwards. If the rollback used an out-of-band `kubectl apply` or cluster edit, the change is reconciled back into Git so the controller does not fight it.

The previous artifact has not been garbage-collected from the registry. A retention policy that deleted the previous release before the rollback makes it impossible.

crane digest ghcr.io/&lt;org&gt;/&lt;image&gt;@&lt;previous-digest&gt;; cosign verify --certificate-identity &lt;identity&gt; --certificate-oidc-issuer &lt;issuer&gt; ghcr.io/&lt;org&gt;/&lt;image&gt;@&lt;previous-digest&gt;

Recovery is confirmed through the user-visible symptom and through the original alert, not through the component that was rolled back. The error rate is back to baseline and the firing alert has resolved.
The change ticket records the rollback: previous digest, new digest, pipeline run URL, the user who approved, the validation result, and the time-to-recovery. A rollback without evidence cannot be analysed.

Warning5 items

In-flight requests, long-running jobs, queued messages, and pending uploads are considered. The rollback either drains gracefully, completes in-flight work, or acknowledges that some requests will be dropped.
A "rollback" that is actually a forward fix (deploy v2.1 over v2.0) is treated as a new deploy, not as a rollback: it goes through pre-deploy, gets a new ticket, and is signed off separately.
Stakeholders are updated on the rollback itself: started, expected duration, validated, resolved. The cadence is fixed (every 5 minutes during the rollback) and continues past the rollback if user impact persists.
A decision is recorded during the incident: rollback only, or rollback + forward fix. If forward-fix, the bad change stays reverted and a separate PR introduces the corrected change through the normal flow.
A follow-up action is filed: post-mortem, additional test that would have caught this, additional rollback drill, additional observability. The action has an owner and a date.