Flux rollback via revert — `git revert` plus reconcile, and the workflow the controller follows

Flux does not have a native rollback command. Where Argo CD exposes argocd app rollback that targets a history entry without touching Git, Flux treats the rollback as a Git-driven operation: the rollback is a git revert on the branch, and the controller’s reconciliation is what applies the reverted manifests to the cluster. The model is deliberate. Flux’s design philosophy is that Git is the only source of truth and the controller’s job is to make the cluster match Git - never the other way around. A native rollback command that bypassed Git would violate that philosophy. The on-call engineer who uses Flux must therefore compose the rollback from a git revert and a flux reconcile, in that order.

Why Flux has no native rollback

The contrast with Argo CD is intentional. Argo CD records every sync in the Application’s history and exposes a command that targets a previous sync; the command is a controller-only operation. Flux records every applied revision in the Kustomization’s .status.history field, but the field is informational. The controller has no command that targets a previous revision; the controller reconciles to the Git revision the source-controller has fetched.

flowchart LR
    subgraph Argo["Argo CD model"]
        H1["Application.history"] --> R1["argocd app rollback APPNAME N"]
    end
    subgraph Flux["Flux model"]
        G1["Git branch tip"] --> R2["git revert"]
        R2 --> S1["flux reconcile kustomization"]
    end

The Flux model has consequences the on-call engineer must internalise:

• The cluster cannot be rolled back faster than the Git branch can be updated. A git revert is a commit, the commit must be pushed, the source-controller must fetch it, and the kustomize-controller must reconcile. The latency is the round-trip from the on-call engineer’s commit to the controller’s next tick. With webhooks it is seconds; with a manual reconcile after a push, it is the time to run the command.
• The branch and the cluster cannot diverge on purpose. The controller has no “rolled back to revision N” mode that the next reconcile would respect; every reconcile targets the branch tip. A divergence between the branch and the cluster is drift, and the controller will correct it in the branch’s direction.
• The history is audit, not control. .status.history records what the controller applied and when. It does not drive the controller’s behaviour; the branch drives the controller’s behaviour.

The .status.history field

A Kustomization’s status carries a history of every successful reconciliation. The field is informational: the controller appends an entry on every successful apply and uses the field to detect drift between the last applied revision and the current source revision.

flowchart LR
    K["Kustomization api"] --> H1["status.history - last applied revision 8a3f9d2"]
    K --> H2["status.lastAppliedRevision - 8a3f9d2"]
    K --> H3["status.observedGeneration - 7"]

Inspecting the history:

kubectl get kustomization $KS_NAME -n flux-system -o yaml

The output includes a .status.history list with entries shaped like:

- lastAppliedRevision: 7c2e8b1/abc...
  digest: sha256:24a0c4b4...
  timestamp: 2024-01-15T11:14:41Z
  metadata:
    revision: 7c2e8b1

The history is a record, not a control surface. The on-call engineer reads it to identify the last good revision; the rollback is a git revert that targets the bad commit, not a Flux command that targets a history entry.

The rollback workflow

The Flux rollback is a Git revert plus a reconcile. The sequence is:

git revert $BAD_COMMIT
git push origin main
flux reconcile kustomization $KS_NAME

The first command produces a new commit on the branch that inverts the bad commit. The second pushes the revert to the remote; the source-controller will pick it up on the next fetch. The third forces the kustomize-controller to reconcile immediately rather than waiting for the next interval.

sequenceDiagram
    participant Op as On-call
    participant G as Git
    participant SC as source-controller
    participant KC as kustomize-controller
    participant K as Cluster
    Op->>G: git revert $BAD_COMMIT
    G-->>Op: new commit on branch
    Op->>G: push
    G-->>SC: webhook (if configured)
    SC->>SC: fetch new artifact
    Op->>KC: flux reconcile kustomization $KS_NAME
    KC->>SC: read artifact at new tip
    KC->>K: apply reverted manifests
    K-->>KC: resources converged
    KC-->>Op: Ready=True

The webhook is optional. If the Git repository is configured with a webhook receiver in Flux, the source-controller fetches the new artifact within seconds of the push. If the repository has no webhook, the source-controller fetches on its interval (default 1 minute). The flux reconcile command forces a fetch regardless; it tells the controller “tick now, do not wait”.

When to wrap the revert in a pull request

The discipline of wrapping every change in a pull request extends to the revert. The argument is the same: the pull request is where the review and CI confirm the change is sound. A revert merged without review is a revert whose rationale is unrecorded.

git checkout -b revert/$BAD_COMMIT main
git revert $BAD_COMMIT
git push origin revert/$BAD_COMMIT

A pull request from revert/$BAD_COMMIT to main runs the standard CI: lint, validate, dry-run. The merge triggers the controller’s reconciliation. The audit trail is the pull request, the CI run, the merge commit, and the Kustomization’s .status.history entry.

The exception is the incident in which the bleeding cannot wait for a pull request. A revert committed directly to main is acceptable when the on-call engineer judges the review delay to be more expensive than the review’s absence. The discipline is to record the direct commit in the incident channel and to backfill the review after the bleeding has stopped.

Suspending reconciliation during a slow revert

If the Git revert takes longer than the reconciliation interval (for example, the on-call engineer is investigating the cause before committing the revert), the controller will keep reconciling the current bad state on every tick. The fix is to suspend reconciliation until the revert is ready.

flux suspend kustomization $KS_NAME
git revert $BAD_COMMIT
git push origin main
flux reconcile kustomization $KS_NAME
flux resume kustomization $KS_NAME

The suspend prevents the controller from re-applying the bad manifests while the engineer is working. The resume re-enables reconciliation. The reconcile after the resume forces a tick that picks up the reverted tip. The discipline is that suspension is per-Kustomization and time-bounded; the on-call engineer resumes as soon as the revert is committed.

What the rollback does and does not do

The Flux rollback restores the cluster’s resources to the manifests at the branch tip after the revert. It does not touch the Git branch beyond the revert commit. It does not touch the source-controller’s artifact cache beyond the new fetch. It does not roll back the cluster’s data; rows in a database that were written under the bad revision remain after the rollback.

The boundaries the rollback does not cross define its reach. A team that uses the rollback as the only recovery step has restored one artifact while leaving the others drifted.

Common failure modes

Three failure modes are specific to the Flux rollback:

• Reverting without forcing a reconcile. The on-call engineer commits and pushes the revert; the controller’s next interval is five minutes away; the cluster stays on the bad manifests for five minutes. The fix is flux reconcile kustomization $KS_NAME after the push, which forces the controller to tick now.
• Forgetting to resume after a suspend. The on-call engineer suspends the Kustomization to investigate, commits the revert, but forgets to resume. The Kustomization is suspended; the new commit is in Git; the cluster is on whatever state it was in when the suspend was issued. The fix is flux resume kustomization $KS_NAME.
• Reverting past an out-of-band change. The bad commit also touched a Secret the controller does not manage. The revert restores the manifest; the Secret is not on the branch; the cluster still has the bad Secret value. The fix is to treat the Secret as a separate artifact with its own rollback surface (LXXXV-02).

Production discipline

1. Always revert in Git, never in the cluster. Flux’s model is that Git is the only source of truth. A revert that touches only the cluster is not a Flux rollback; it is drift.
2. Force a reconcile after the push. The reconcile ticks the controller now rather than waiting for the next interval.
3. Wrap the revert in a pull request when possible. The review and CI discipline applies to the revert.
4. Inspect .status.history to identify the last good revision. The history is informational; reading it is how the on-call engineer knows what to revert from.

Cross-course references

• This course, Part LXXVIII (Flux architecture) - the source-controller and kustomize-controller whose reconciliation the revert triggers.
• This course, Part LXXXV-01 (Git revert vs controller rollback) - the two halves of a rollback.
• This course, Part XIV (Reverting commits) - git revert in detail.

Quiz