Revert versus redeploy — when to roll back, when to fix forward

Revert and redeploy are not synonyms. A revert is a commit that undoes the change in the source of truth; a redeploy is a new release that ships a fix forward. Both end with “the bad code is no longer running in production,” but they get there by different paths and have different operational properties.

The decision between them is one of the most consequential calls in an incident. A revert is the right call when the bad code must be out of production now and there is no fix ready. A redeploy is the right call when a fix is known, tested, and the blast radius of leaving the bad code running for the next hour or two is acceptable. Both are valid; neither is universally correct; the worst case is to do neither, leaving the bad code running while the team argues.

What a revert does to production

A revert is a commit. Once that commit is pushed to the branch that production tracks (typically main or a release branch), the same CI/CD pipeline that ships every other change ships the revert. A GitOps controller reading the branch sees the new commit, computes the diff between the previous desired state and the new desired state, and reconciles production to the reverted state. The production system is now running the tree at HEAD minus the changes from the reverted commit:

flowchart LR
    A["bad commit on main"] --> B["production runs bad code"]
    B --> C["git revert bad-commit"]
    C --> D["revert commit on main"]
    D --> E["CI pipeline runs on revert commit"]
    E --> F["GitOps controller reconciles"]
    F --> G["production runs reverted code"]

The timeline from “revert commit pushed” to “production runs reverted code” is exactly the same as the timeline for any other change. There is no special fast path. The audit pipeline is satisfied: the revert went through code review, CI, and a deployment. The auditor six months later can see the revert in the history and reconstruct what was rolled back.

The cost of a revert is what it always is: the inverse of the original change is now in production, which may have its own problems. If the bad commit introduced a feature that other code depends on, the revert breaks that dependency. If the bad commit removed a feature users rely on, the revert brings it back. A revert is not free; it is the cheapest immediate rollback, not the cheapest long-term state.

What a redeploy does to production

A redeploy is a new release that fixes the problem forward. The bad code is no longer running because it has been replaced by fixed code, not because it has been undone. The source of truth goes from “bad commit + nothing else” to “bad commit + fix commit”; production tracks the new state.

flowchart LR
    A["bad commit on main"] --> B["production runs bad code"]
    B --> C["fix commit on main"]
    C --> D["CI pipeline runs on fix commit"]
    D --> E["GitOps controller reconciles"]
    E --> F["production runs fixed code"]

A redeploy is the normal path for every change in a healthy pipeline. The bad code is replaced by good code; the history shows both commits; the production timeline is “bad, then fixed.” There is no rollback commit because there was no rollback — the fix is the change.

The cost of a redeploy is time. The fix must be written, tested, reviewed, and shipped. The pipeline that catches the bad code in the first place should catch the fix; the deployment that ships the bad code should ship the fix. If the fix is small and the tests are good, this can take minutes. If the fix is non-trivial or requires coordination, it can take hours.

When to revert, when to redeploy

The decision tree is short. It depends on three questions:

flowchart TD
    A["Production incident or risk?"] -- "no" --> Z["Do nothing — fix in the normal cadence"]
    A -- "yes" --> B{"Is a fix ready and tested?"}
    B -- "yes" --> Y["Redeploy the fix"]
    B -- "no" --> C{"Can production wait for the fix?"}
    C -- "yes" --> X["Write and ship the fix; no revert needed"]
    C -- "no" --> W["Revert now, then redeploy the fix when ready"]

The branches:

• No incident. Do nothing. The bad code is in production but is not causing harm; the normal cadence of review and ship will produce a fix in the next cycle. Reverting without an incident is bureaucratic overhead.
• Incident with a ready fix. Redeploy. The fix is the right state for production; the revert would be a temporary state that has to be undone by the redeploy anyway. Ship the fix.
• Incident without a ready fix, but production can wait. Write and ship the fix; no revert needed. The cost of waiting is acceptable; the cost of a revert-then-redeploy sequence is not.
• Incident without a ready fix, and production cannot wait. Revert now, then redeploy the fix when ready. The immediate rollback removes the bad code; the subsequent redeploy restores the feature (or replaces it with a fixed version) without a window of vulnerability.

GitOps and the rollback

A GitOps controller (ArgoCD, Flux, or equivalent) reads the desired state from Git and reconciles production to match. The desired state is whatever is at the branch tip the controller tracks. A revert commit pushed to that branch is a new desired state; the controller sees it, computes the diff, and applies it to production. No special “rollback” mechanism is required.

COMMIT=abc1234
git revert -m 1 $COMMIT
git push origin main
# GitOps controller picks up the new commit on its next sync
# Production reconciles to the reverted tree

The same is true for a redeploy: a fix commit pushed to the branch is a new desired state, and the controller reconciles production to the fixed tree. The controller does not care whether the new commit is a revert, a fix, or a feature; it cares only that the branch tip has moved.

For Kubernetes Deployments specifically, the rollback story is two-layered:

# Layer 1: git-level rollback (the source of truth)
git revert -m 1 $COMMIT
git push origin main

# Layer 2: Deployment-level rollback (an emergency escape hatch)
kubectl rollout undo deployment/service-x
# This rolls back the Deployment's last ReplicaSet without
# touching Git. Use only when GitOps reconciliation is broken
# or the rollback must happen faster than the next GitOps sync.

kubectl rollout undo is a redeploy of the previous ReplicaSet without a corresponding Git commit. It is faster than a full GitOps reconcile but breaks the GitOps invariant: production no longer matches the branch tip. The discipline is to follow a kubectl rollout undo with a git revert as soon as the incident is stabilised, so production and Git re-converge.

UnderTheHood: image tags and the redeploy boundary

For containerised services, a redeploy is usually a new image tag. The Deployment manifest in Git pins an image reference; updating the pin to a new tag (or digest) is a new commit, and the GitOps controller reconciles to the new image. The choice between revert and redeploy at the Git level maps onto a choice at the image level:

• Revert at Git level: the manifest in Git goes back to pinning the previous image tag. Production reconciles to that previous tag.
• Redeploy at Git level: the manifest in Git pins a new image tag. Production reconciles to that new tag.

The image tag boundary is where the “fix forward” path lives. If the bad release is v3.2.0 and the fix is v3.2.1, a redeploy is v3.2.0 → v3.2.1. A revert is v3.2.0 → v3.2.0-rolled-back (where the rolled-back tag is either the previous v3.1.x or a newly built v3.2.0 with the bad change reverted). The image registry is where the rollback boundary is enforced.

Production discipline

The production discipline has five rules:

1. Revert for security exposures and data-integrity risks. The bad code is removed immediately; the fix is shipped as a follow-up. The revert-then-redeploy sequence is two commits in the history, both with explicit messages.
2. Redeploy for correctness bugs and performance regressions. The fix is the right state for production; no temporary rollback is needed. The history shows the bad commit and the fix commit, and nothing else.
3. Never use kubectl rollout undo (or equivalent) without a follow-up git revert. A rollback that is not in Git is a rollback the next incident responder cannot reproduce.
4. Pin images by digest in repositories where the rollback boundary matters. Tags are mutable; digests are not. The rollback commit in Git must reference an image that cannot have been replaced.
5. Document the decision. The incident postmortem records whether the rollback was a revert or a redeploy, and why. The next incident responder needs the rationale.

Cross-course references

• Git, CI/CD & GitOps — Part XIV (Revert 1-5) — the mechanics of git revert, -m, -n, and the audit-trail framing are the substrate for the operational decision in this lesson.
• Kubernetes for Production Sysadmins — Part XV (Deployments) — kubectl rollout undo and ReplicaSet history are the Deployment-level rollback mechanism; this lesson is the Git-level counterpart.
• Ansible for Production Sysadmins — Part XXXVII (RepoArch) — the same revert-vs-redeploy choice applies to configuration management: revert the playbook in Git, or ship a fixed playbook forward.
• Terraform for Production Sysadmins — Part IX-XII (State) — a revert in Git is not enough for Terraform; the state file must be reconciled separately. Plan the rollback as a coordinated Git + state operation.

Quiz