Reconciliation fights emergency manual fix

Reported symptoms

●The on-call engineer applied a manual change via `kubectl edit` or `kubectl patch` to a Production Application during an incident
●The change worked: the symptom abated, alerts cleared, the on-call declared mitigation
●Within `timeout.seconds` (default 5 seconds) the change was reverted by Argo CD and the symptom returned
●`argocd app get <app>` shows the sync status flapping: `OutOfSync` moments after each edit, then `Synced` again once the controller self-heals the live state back to git
●The on-call engineer repeated the manual change; the same revert happened
●The Application spec has `automated.selfHeal: true` and `automated.prune: true`
●GitOps discipline is otherwise strong: there are no other drift events in the last 90 days
●The team has a documented break-glass procedure, but it does not mention "disable Argo CD sync"

Evidence

· `kubectl get events -n argocd --field-selector reason=SyncOperationStarted` shows a `Sync` event within seconds of each manual `kubectl edit`
· `kubectl get events -n <app-ns> --field-selector reason=DeploymentUpdated` interleaves: one by the on-call SA, one by `argocd-application-controller` within `timeout.seconds`
· `argocd app diff <app>` run right after the edit shows the live/git divergence, and `argocd app history <app>` shows only repeated sync operations to the same git revision — the manual edit creates no history entry and no git revision; the controller log shows a self-heal sync to that same revision after each edit
· `kubectl get application <app> -n argocd -o yaml` returns `spec.syncPolicy.automated` as `{selfHeal: true, prune: true}`
· The kube audit log shows the manual edit's `user.username=<on-call>` followed by `user.username=argocd-application-controller` with the inverse `spec.template.spec.containers[0].resources.limits.memory`
· `git log --oneline -- overlays/prod/<app>/` does not contain the on-call's change; the manifest is unchanged
· The on-call engineer did not edit the GitOps repo first, or did edit it but the PR did not merge before the reconcile loop reverted
· `argocd app manifests <app>` does not contain the on-call's resource changes; they were only ever in the cluster

Diagnosis and resolutionclick to reveal

Root cause

`selfHeal: true` is the controller''s policy: any deviation between cluster state and git-declared state is reverted to the git state. During an incident, an engineer reached for `kubectl` because the change was needed faster than a PR could be reviewed and merged. The controller then ran the reconciliation loop, saw the deviation, and applied git — exactly as configured, exactly as expected, exactly as unhelpful during an incident. This is not a bug; it is a structural conflict between two policy goals: GitOps discipline (cluster must match git) and incident response (a human must be able to apply a fix faster than the PR pipeline allows). The conflict is real, and it must be resolved structurally, not procedurally — relying on the on-call to remember to disable sync is the wrong place to encode the policy because the on-call is the person least able to do a procedural task at 3am.

Remediation

Stop the loop immediately. For a standalone Application, disable automated sync through the supported CLI: `argocd app set <app> --sync-policy none`. If the whole Application reconciliation loop must be paused, set the documented annotation with `kubectl annotate application <app> -n argocd argocd.argoproj.io/skip-reconcile=true --overwrite`; status and health will be stale while this annotation is present. For an ApplicationSet-managed Application, change the ApplicationSet template/source because editing only the generated child may be overwritten. Then apply the hotfix again, this time knowing the controller will not revert it. Capture the change as a PR in the GitOps repo: open the PR with the same resource changes the on-call made, get an out-of-band review (incident channel, on-call manager, post-incident review), and merge. Once merged, remove `argocd.argoproj.io/skip-reconcile` if it was used, restore the declared automated-sync policy, explicitly refresh the Application, and let Argo CD reconcile to the new git state. Codify the exact pause command, owner, expiry, and restoration steps in the incident runbook. If scheduled deny windows are required, define them on the AppProject; a sync window is not an Application field and an unpredictable incident cannot be represented by a permanent after-hours exception.

Verification

`argocd app get <app>` shows `Sync Status: Synced` to the new manifest that contains the on-call''s hotfix. `kubectl get deploy <name> -o jsonpath={.spec.template.spec.containers[0].resources.limits.memory}` equals the value in the GitOps repo. The kube audit log shows no more writes from the on-call SA after the merge — the on-call is out of the loop. `argocd app history <app>` ends with a single sync to the new SHA and the repeated self-heal syncs to the old revision have stopped. The documented pause control is absent at the end, health has been refreshed, and the incident record contains who paused reconciliation, why, and when it was restored.

Prevention

Reconcilers and incident response share a boundary. Provide a documented, least-privilege break-glass path for a standalone Application and a separate path for ApplicationSet-managed Applications. Alert on the documented `skip-reconcile` annotation and on automated-sync policy changes, attach an expiry to the incident action, and make restoration an incident exit gate. Use AppProject sync windows only for genuine scheduled change policy, not as a fictional Application-level incident switch. Finally, drill the scenario: simulate an incident, run the break-glass procedure, capture the manual change as a PR, re-enable sync, and verify the cluster matches git at the end of the drill. The principle is that GitOps discipline and incident response are both correct; the policy must say which one wins during a declared incident, and the answer must be encoded in the system.

Self-heal is a policy and break-glass is a controlled exception. Use the documented automated-sync or skip-reconcile controls, distinguish standalone Applications from ApplicationSet-managed children, and make restoration and refresh explicit incident exit criteria.