The rollback decision framework — when to revert Git, when to deploy a hotfix, when to forward-fix

The first minutes of an incident are a decision under uncertainty. The on-call engineer knows a bad change is in production, knows the change is hurting users, and has to choose between three paths: revert the bad commit in Git, ship a hotfix that supersedes the bad change, or write a forward-fix that addresses the data the bad state wrote. The wrong choice costs time and trust; the right choice is the one that restores the service in the fewest minutes with the smallest blast radius. The framework in Part LXXXV-05 is the signals the engineer reads to make the choice: the data question, the urgency, and the fix availability.

The three paths

A rollback is one of three operations, each with a different cost and a different reach:

flowchart LR
    A["Incident detected"] --> B{"Data written?"}
    B -- "No" --> C{"Fix known?"}
    B -- "Yes" --> F["Forward-fix"]
    C -- "Yes, urgent" --> D["Hotfix"]
    C -- "Yes, not urgent" --> E["Git revert"]
    C -- "No" --> G["Suspend + investigate"]

• Git revert. git revert <bad-commit> plus a controller rollback (or reconcile). The branch is restored to a known-good commit; the cluster is restored to the previous synced revision. The path is the fastest when the data half is untouched (LXXXV-02). The path is the slowest when the bleeding must stop in seconds and the commit-and-reconcile cycle is too slow.
• Hotfix. A new commit on the branch that supersedes the bad one - the bad change is left in history but its effect on the cluster is undone by the new commit. The hotfix is the right path when the bad change is fundamentally wrong (the schema is wrong, the algorithm is wrong) and the engineer has a known-good replacement.
• Forward-fix. A compensating change that addresses the data the bad state wrote. The path is the only safe path when the bad state has written data that cannot be left in place. The forward-fix is the slowest path; it accepts the duration of the data corruption as the cost of recovery.

The decision signals

The framework reduces to three signals. Each signal has a discrete set of values; the combination of values identifies the path.

Signal 1: the data question

The data question is the discriminator. Did the bad change write data that the rolled-back state cannot read or that must remain in place?

• No data written. The bad change touched manifests or configuration; no rows, no files, no secrets have been written. The rollback is a pure state operation; git revert plus a controller rollback restores the system.
• Data written, data shape compatible. The bad change wrote data that the rolled-back state can read (or that can be migrated to the rolled-back shape). The rollback is git revert plus a forward-fix for the data. The forward-fix is small.
• Data written, data shape incompatible. The bad change wrote data that the rolled-back state cannot read and that cannot be migrated to the rolled-back shape. The rollback is the forward-fix path: keep the bad manifests, write a compensating migration that brings the data into a shape the application can read.

Signal 2: the urgency

The urgency is the controller. How fast must the bleeding stop?

• Seconds. The service is fully down. The path is the fastest one available: a controller rollback if Argo CD, a kubectl rollout undo if the rollback is at a single Deployment. The Git revert follows as soon as the bleeding has stopped.
• Minutes. The service is degraded. The path is git revert plus a controller-side nudge (argocd app sync, flux reconcile). The CI and the controller’s webhook are fast enough that the revert reaches production in under a minute.
• Hours. The service is impaired but not down. The path is the slowest one available: a hotfix that supersedes the bad change, with full review and CI. The cost is acceptable because the bleeding does not require seconds-to-minutes recovery.

Signal 3: the fix availability

The fix availability is the constraint. Does the engineer have a known-good replacement?

• Replacement known. The engineer knows what the manifest should look like (or what the migration should be). The path is the hotfix: a new commit on the branch with the correct value.
• Replacement unknown. The engineer does not yet know the correct value; the investigation is in progress. The path is the revert: restore the previous known-good state while the engineer investigates.

The decision matrix

The three signals combine into a small decision matrix:

The “Incompatible” row is the one most teams underestimate. A bad change that wrote data the rolled-back state cannot read cannot be rolled back by reverting Git; the rollback must keep the new state and fix the data. The hotfix path is the wrong choice here because the bad manifests are still in production; the Git revert path is the wrong choice because the data is incompatible with the reverted manifests.

The order of operations for each path

Each path has a sequence; the sequence is what makes the rollback close the loop.

Git revert path

argocd app rollback $APP_NAME
git revert $BAD_COMMIT
git push origin main
argocd app sync $APP_NAME

The Argo CD rollback restores the cluster in seconds. The Git revert restores the branch. The push and the sync confirm the convergence. The order is cluster first, branch second.

Hotfix path

git checkout -b hotfix/$INCIDENT main
# edit manifests
git commit -m "hotfix: revert bad probe path"
git push origin hotfix/$INCIDENT
# open PR, get review, merge
argocd app sync $APP_NAME

The hotfix is a new commit that supersedes the bad one. The PR wraps the hotfix in review and CI; the merge triggers the sync. The bad commit remains in history; the new commit’s manifests are what the controller applies.

Forward-fix path

git checkout -b forward-fix/$INCIDENT main
# add a compensating migration
git commit -m "forward-fix: re-add dropped column from JSONB"
git push origin forward-fix/$INCIDENT
# open PR, get review, merge
flux reconcile kustomization $KS_NAME

The forward-fix keeps the new manifests (they are the correct ones) and adds a compensating change for the rows the bad state wrote. The PR wraps the migration in review and CI; the reconcile triggers the controller to apply the new state.

The hotfix is not a replacement for a fix-forward

A hotfix is a change that supersedes the bad change. It is not the same as a fix-forward: the hotfix is the production recovery; the fix-forward is the prevention of recurrence.

flowchart LR
    A["Incident"] --> B["Hotfix"]
    B --> C["Service restored"]
    C --> D["Fix-forward"]
    D --> E["Recurrence prevented"]

The hotfix restores the service; the fix-forward is the PR that adds the test, the lint rule, the review checklist, the pre-staged rollback branch, the change-cause annotation. The hotfix is the change that goes into the incident record; the fix-forward is the change that goes into the team’s weekly meeting.

Common failure modes

Three failure modes are specific to the decision framework:

• Reverting without asking the data question. The on-call engineer sees a 5xx spike and reverts the schema migration. The reverted application cannot read the new rows. The service stays down.
• Hotfixing without investigating. The on-call engineer ships a hotfix that “fixes” the symptom without understanding the cause. The same incident recurs the next time the hotfix is bypassed. The fix-forward PR is the place to investigate the cause.
• Forward-fixing without keeping the new state. The on-call engineer writes a forward-fix but also reverts the manifest, leaving the cluster on the bad state with no compensating migration. The service is on a half-rollback.

Production discipline

1. Ask the data question before choosing the path. A rollback that ignores the data half is incomplete.
2. Match the urgency to the controller, not to the path. Seconds means controller rollback first, Git revert second. Minutes means Git revert first, controller second.
3. Never skip the PR review when the bleeding allows. The review is the audit trail; the incident channel is the exception, not the rule.
4. Always open a fix-forward PR after the hotfix. The fix-forward is the change that prevents recurrence.

Cross-course references

• This course, Part LX (Decision framework) - the rollback-versus-forward-fix decision in detail.
• This course, Part LIX (Rollback across artifact boundaries) - the five boundaries the rollback must cross.
• This course, Part LXXXV-02 (Data vs state rollback) - the data question.

Quiz