Argo CD Application OutOfSync after manual change

Reported symptoms

●`argocd app get <app>` shows `Sync Status: OutOfSync` and a one-line `Status Revision Difference`
●The diff reported by Argo CD is exactly `replicas: 3 to 6` against a Deployment the team owns
●`kubectl get deploy <name> -o jsonpath={.spec.replicas}` returns `6` while the GitOps repo manifest sets `replicas: 3`
●`git log --oneline -- overlays/prod/<app>/deployment.yaml` shows no recent change to `replicas`
●No `Sync` operation has run since the incident: `argocd app get <app>` re-reports the same diff on every reconcile cycle (`timeout.reconciliation`, default 3 minutes), and the last recorded sync predates the drift
●The kube audit log shows `verb=update, user=<sre-username>, object=Deployment/<app>` from the day of the incident, no subsequent update from any other principal
●The Application has `automated.selfHeal: false` but `automated.prune: true`: drift is detected on the reconcile interval, but nothing re-applies the desired state
●`argocd app history <app>` shows no sync record for the change; the diff is "live" against the cluster, not a manifest Argo CD applied

Evidence

· `argocd app manifests <app> | yq .spec.replicas` returns `3`
· `kubectl get deploy <name> -n <ns> -o yaml | yq .spec.replicas` returns `6`
· `kubectl get events -n <ns> --field-selector involvedObject.name=<deploy>` shows `ScalingReplicaSet` events but no recent `Updated` event from the controller
· `argocd app diff <app>` returns only the one-line diff: `spec.replicas: 3 vs 6`
· `kubectl get deploy <name> -o jsonpath={.metadata.annotations}` does NOT contain `argocd.argoproj.io/tracking-id` set to the tracked resource ID, indicating the manifest is being tracked
· `kubectl auth can-i update deploy/<name> --as=system:serviceaccount:argocd:argocd-application-controller` returns `yes`: the controller could re-apply the desired state, but with `selfHeal: false` it only does so during a sync, and no new git revision has arrived to trigger one
· The kube-apiserver audit log records `user.username=<sre>` as the only principal that wrote to `spec.replicas` in the last 30 days
· `git log --all --diff-filter=M -- overlays/prod/<app>/deployment.yaml | head -5` shows the last commit to the file was months ago, well before the incident

Diagnosis and resolutionclick to reveal

Root cause

Argo CD''s reconciliation loop re-compares the cluster against the desired state in git on every reconcile interval (`timeout.reconciliation`, default 3 minutes), so the manual `kubectl scale` is re-reported as `OutOfSync` again and again — but detection is all that happens. With `automated.selfHeal: false`, automated sync fires only when a NEW git revision arrives; it never re-applies the current revision just because the live cluster drifted. No commit has touched the manifest since the incident, so no sync has run and the drift sits untouched. `automated.prune: true` changes nothing here: pruning only deletes resources no longer in git, it does not restore field-level values on existing resources. This is one-time drift — no actor is re-applying it — so a single successful sync (or `selfHeal: true`) would clear it completely; the diff persists only because nothing triggers that sync. The structural failure is that the incident response reached for `kubectl` instead of for git, and the GitOps controller then had no way to express the change as a code-review-able artefact.

Remediation

Decide which side is authoritative. If the manual scale was a legitimate emergency response that should remain, commit the new value to git: update the manifest to `replicas: 6`, open a PR, get a post-incident review, and let the next sync reconcile. If the manual scale was wrong or temporary, revert the cluster to match git: run `argocd app sync <app>` (a single successful sync fully clears one-time drift) or `kubectl scale deploy <name> --replicas=3`. Either way, the diff becomes zero. Then add `selfHeal: true` to the Application spec (and `prune: true` if not already set) so future drift is reverted automatically, with the change captured in the controller log as a `Sync` operation. For production Applications where auto-revert of a manual change is undesirable, set `selfHeal: false` and rely on a drift-detection alert that pages on `argocd app list | grep OutOfSync` so a human resolves the divergence within a service-level objective.

Verification

`argocd app get <app>` shows `Sync Status: Synced` after the chosen resolution. `argocd app diff <app>` returns empty. `kubectl get deploy <name> -o jsonpath={.spec.replicas}` equals the value in the GitOps manifest. The kube audit log for the Deployment shows no more writes from the original `<sre-username>` for the same field. With `selfHeal: true`, manually scaling to a value different from git is detected and reverted on the next reconcile cycle (`timeout.reconciliation`, default 3 minutes), and the revert appears in `argocd app history <app>` as a sync.

Prevention

Make the git path the only path. In a GitOps-managed cluster, the Application''s `destination` namespace and the `Repositories` RBAC should grant only the GitOps controller write access; human principals should have read access for debugging and break-glass access via a time-limited `RoleBinding`. Add an admission policy (Kyverno or OPA) that rejects `update` or `patch` on `apps/Deployment` for any principal that is not `system:serviceaccount:argocd:argocd-application-controller`, so a manual `kubectl scale` is rejected at the API server rather than silently creating drift. For the rare case where a manual change is necessary (an emergency fix to a value that cannot wait for a PR), use a documented break-glass procedure: open a tracker issue, edit the manifest, merge a PR with the change, and let Argo CD reconcile — the manual `kubectl edit` is replaced by a code change that is auditable and rollback-able. Drift should be detected and surfaced, not reverted silently — that distinction is the policy boundary between GitOps and Kubernetes-as-a-database.

Drift is the signal that git and the cluster disagree. The recovery is not to silence the signal but to resolve the disagreement by changing exactly one side: either commit the cluster change to git or revert the cluster to git. The choice should be auditable, and the policy should make the cluster-only path structurally impossible for everyone except the GitOps controller.