Self-heal versus control — when automatic reconciliation fights the operator

Self-heal is the controller’s revert-on-diff behaviour: when the controller sees that the observed state has diverged from the desired state, it patches the cluster back. Auto-sync is the controller’s apply-on-Git-change behaviour: when a new commit appears in the Git source, the controller renders and applies. They are independent toggles, and the production discipline depends on understanding the difference.

stateDiagram-v2
    [*] --> Synced
    Synced --> OutOfSync: Observed diverges from desired
    OutOfSync --> Synced: Self-heal patches cluster back
    OutOfSync --> Suspended: Operator suspends for incident
    Suspended --> OutOfSync: Operator resumes
    OutOfSync --> Committed: Operator commits change in Git
    Committed --> Synced: Controller applies new desired state

The two toggles, independently

A production GitOps Application has four combinations of the two toggles:

• Auto-sync off, self-heal off. The controller reports drift but never applies. Used in early-stage or audited environments where every apply requires human approval.
• Auto-sync on, self-heal off. The controller applies Git changes, but does not revert manual edits. A new commit is applied; a manual edit is surfaced as drift but not reverted. The most common production setting.
• Auto-sync off, self-heal on. Rare. The controller reverts manual edits but does not apply Git changes. Used when the Git source is reviewed out of band and the controller’s job is to enforce the cluster’s match to whatever the source-of-truth is.
• Auto-sync on, self-heal on. The fully automatic configuration. New commits are applied and manual edits are reverted. The highest drift-recovery rate and the highest rate of fighting legitimate operator actions.

The production rule is one of the middle two combinations. Self-heal off with auto-sync on is the safe default; self-heal on is reserved for Applications whose resources should never be hand-edited.

The silent failure

The dangerous failure mode is the silent one: self-heal is on, an operator makes a hand-edit, the controller reverts it on the next tick, the operator does not notice because they are working on something else, and the operator’s work is lost. The Application is Synced, the diff is empty, and the on-call engineer who made the hand-edit is the only person who knows work was lost.

The silent failure is rare in environments with strong monitoring — the alert fires on the brief OutOfSync window — but common in environments where alerts are tuned to sustained drift only. A 30-second OutOfSync tick that reverts to Synced is below the alerting threshold and the loss is invisible.

The mitigation is one or both of:

• Lower the sustained-drift window to one tick. A controller that runs every 90 seconds with a sustained window of one tick pages on every reversion. The trade-off is alert volume; the gain is visibility into every fight.
• Require commit-before-apply as an operational rule. The GitOps model’s rule is that every change is committed first; any hand-edit is by definition an exception. The exception is logged, the commit follows, and the silent failure becomes a documented one.

The commit-not-fight discipline

The production rule that prevents most fights is that every operator action against a resource is preceded by a commit to Git. The commit is pushed; the controller renders the new desired state; the controller’s apply and the operator’s intent align. The self-heal toggle does not need to be touched.

The commit-not-fight discipline has three operational requirements:

• A working Git push from inside the incident. A laptop with credentials, a CI pipeline that can be triggered by hand, or a runbook that walks the engineer through the commit-and-wait cycle.
• A reconciliation interval shorter than the incident’s decision loop. If the controller runs every 90 seconds and the engineer can commit in 60, the controller is the faster path. If the controller runs every 10 minutes and the engineer can commit in 60 seconds, the engineer is faster — but their work is at risk.
• A documented exception path. When the commit is not possible (no laptop, no credentials, no network), the exception is the suspend-self-heal path. The exception is documented in a runbook that the on-call engineer can follow without paging the platform team.

argocd app sync "$APP_NAME"

The sync command forces an out-of-cycle reconciliation: the controller reads the Git revision, renders the desired state, queries the observed state, and applies the patch if there is a diff. The sync is the production escape hatch when the engineer has committed a fix and wants the controller to apply it now rather than at the next tick.

Production discipline

1. Self-heal is on by default, suspendable per Application per incident. The default is to enforce the declared state; the exception is documented and time-bounded.
2. Auto-sync and self-heal are tuned independently. A team that sets both to on has chosen the most aggressive configuration; the choice must be deliberate.
3. The commit is the production rule, not the exception. A team that finds itself suspending self-heal frequently has an operational problem the GitOps model is supposed to solve. The fix is to make committing faster, not to disable self-heal.

Cross-course references

• Kubernetes for Production Sysadmins - Parts on server-side apply and field managers cover the API-level mechanics that make self-heal work.
• Linux for Production Sysadmins - Parts on configuration management cover the equivalent discipline in file-level configuration.
• Terraform for Production Sysadmins - Parts on drift remediation cover the analogous model in declarative cloud infrastructure.

Quiz