Git revert versus controller rollback — what Git undoes and what the controller undoes

In a GitOps system the word rollback hides two operations that must compose. A git revert adds a new commit to the branch; the new commit’s tree is the inverse of the previous commit’s tree. An argocd app rollback (or its kubectl equivalent) tells the controller to re-apply a previous synced revision to the cluster. The two operations target different artifacts and they are not substitutes. A team that runs only the Git revert and waits for the controller to converge is letting the controller do the cluster-side work, which is fine when the controller is the one that needs to act. A team that runs only the controller rollback and walks away has fixed the cluster but not the branch; the next reconciliation will re-apply the bad change. Part LXXXV is the discipline of running both, in the right order, and recognising the failure mode of running only one.

The two artifacts a rollback must touch

The GitOps model has two artifacts that record what should be running: the branch in the Git repository and the running state in the cluster. A change that touched both must be undone in both.

flowchart LR
    A["Bad commit on branch"] --> B["Bad state in cluster"]
    B -. "Git revert" .-> C["Good commit on branch"]
    C -. "controller applies" .-> D["Good state in cluster"]
    C -. "no controller action" .-> E["Drift: branch good, cluster bad"]
    B -. "controller rollback only" .-> F["Drift: cluster good, branch bad"]

The four endpoints are the four states the system can be in:

• Both good. The branch and the cluster agree on the desired state; no rollback needed.
• Branch bad, cluster bad. A bad commit has been applied; the rollback must fix both. The order is the Git revert first (the controller needs a good commit to converge toward) and the controller’s next reconciliation as a consequence.
• Branch bad, cluster good. A controller rollback (or an out-of-band intervention) reverted the cluster but the branch still points at the bad commit. The next reconciliation will re-apply the bad change. The branch must be reverted in Git to keep the controller honest.
• Cluster good, branch bad. The symmetric case: a Git revert has been committed, but the controller has not yet reconciled. The cluster is on the old bad state; the rollback is incomplete until the next reconcile.

The production discipline is that the branch and the cluster must converge to the same state, and a rollback that leaves them apart is a rollback that the next reconcile will undo.

What git revert actually does

git revert <commit> does not delete the commit and does not rewrite history. It computes the inverse of the diff between <commit>’s parent and <commit>, applies that inverse to the working tree, and produces a new commit on the current branch with the inverse as its change. The original commit remains in the history; the new commit cancels its effect on the tree.

git revert $BAD_COMMIT
git push origin main

The command is additive: it produces a new commit that is recorded in the history alongside the one it undoes. The auditability properties the course established in Part I hold

• the original change is still in history, the rationale for the revert is captured in the new commit’s message, and the branch tip is moved forward, not rewritten. A pull request that wraps the revert preserves the review and CI discipline.

git revert is the right command for a shared branch. The alternative - git reset --hard and a force-push - rewrites history, breaks every collaborator who has pulled the original tip, and destroys the audit trail. The two commands are not interchangeable on a pushed branch.

What the controller rollback actually does

argocd app rollback $APP_NAME does not touch Git. It writes a sync to the Argo CD controller that points the Application at a previous deployed revision from its history, and the controller re-applies that revision’s manifests to the cluster. The Application object now records the rollback as the most recent operation; the cluster’s resources converge to the previously synced state; Git is unchanged.

argocd app rollback $APP_NAME
argocd app rollback $APP_NAME $HISTORY_ID

The first form rolls back to the previous deployment. The second rolls back to a specific history ID - a number Argo CD assigns to each successful sync and that argocd app history prints. The ID is a positional argument, not a flag, and it is not optional when rolling back past more than one revision; without it, the rollback targets the previous sync, not the one the operator intends.

One precondition: the rollback refuses to run while automated sync is enabled - the controller would otherwise re-sync to the branch tip and undo it on the next tick. Disable automated sync first with argocd app set $APP_NAME --sync-policy none, and re-enable it with --sync-policy automated only after the Git revert has landed on the branch.

The controller’s rollback operates on the applied state: the manifests the controller last wrote to the cluster, not the manuscripts the engineer might have edited locally. If the cluster has drifted from the branch since the last sync, the controller rollback restores to the last synced state, not to the branch tip.

Why the two are not substitutes

The two operations touch different artifacts and solve different problems. git revert solves the source-of-truth problem: the branch must record that the bad change was undone, by whom, and under whose review. The controller rollback solves the runtime-state problem: the cluster must converge to a known-good state in seconds, not in the minutes the commit-and-reconcile cycle takes.

The two are composed, not chosen between:

• Git revert only. The branch records the undo; the controller will reconcile on the next tick and re-apply the inverse. This is the standard path when the controller is healthy and the reconciliation interval is short. It is the slowest path to recovery because the cluster must wait for the next tick, but it is the cleanest: the branch is the source of truth, the cluster follows.
• Controller rollback only. The cluster is restored to a known-good state in seconds; the branch still has the bad commit and will re-apply it on the next reconciliation. This is acceptable only when followed immediately by a Git revert (or a new commit that supersedes the bad one) before the next tick. A controller rollback without a follow-up Git revert is a half-life rollback.
• Both, in order. The fastest path to a stable end state. For Argo CD the sequence starts by disabling automated sync (argocd app set $APP_NAME --sync-policy none) - the rollback refuses to run while it is enabled. The controller rollback restores the cluster; the Git revert aligns the branch; automated sync is re-enabled once the revert has merged; the next reconciliation confirms the convergence and the loop is closed. This is the production pattern for an incident in which the cluster cannot wait for the next tick and the audit trail must be closed in the same hour.

The order of operations

When both are needed, the order matters less than the completeness - but the convention is cluster first, branch second. The on-call engineer’s first obligation is to stop the bleeding; the cluster rollback does that. The branch revert is the durable record of why the cluster was rolled back; it is the audit trail.

sequenceDiagram
    participant Op as On-call
    participant C as Controller
    participant G as Git
    Op->>C: argocd app set $APP_NAME --sync-policy none
    Op->>C: argocd app rollback $APP_NAME
    C-->>Op: cluster restored
    Op->>G: git revert $BAD_COMMIT
    G-->>Op: new commit on branch
    Op->>G: push, open PR
    Op->>C: argocd app set $APP_NAME --sync-policy automated
    Op->>C: reconcile (next tick confirms)

The sequence ends when the controller reconciles and finds no diff. The branch and the cluster agree; the audit trail records both operations; the next change can build on the reverted tip without inheriting the bad state.

Production discipline

1. Always run both halves. A controller rollback without a Git revert (or a Git revert without a controller rollback you can confirm) is an incomplete rollback. The next reconcile will surface it.
2. Cluster first, branch second. The cluster is the live system; the branch is the record. Restore the live system, then update the record.
3. Disable automated sync before the controller rollback; re-enable it after the revert merges. Argo CD refuses to roll back while automated sync is enabled, and re-enabling it before the branch is reverted re-applies the bad commit.
4. Wrap the Git revert in a pull request when possible. The review and CI pipeline that protect every other change protect the revert too. A revert merged without review is a revert whose rationale is unrecorded.
5. Confirm the convergence. After the next reconcile, the diff must be empty. A non-empty diff after a rollback means one half of the rollback was missed.

Cross-course references

• This course, Part LIX (Rollback across artifact boundaries) - the five boundaries a rollback must cross.
• This course, Part XIV (Reverting commits) - git revert in detail, including -m 1 for merge commits and -n for staged inverses.
• This course, Part LXXV (Drift) - emergency drift and the post-incident repair.

Quiz