Self-heal — when automatic correction is right and when it is wrong

Self-heal is the option that reverts drift. Automated sync alone applies Git changes; automated sync with self-heal also reverts out-of-band changes. The two together is the strongest convergence guarantee a GitOps controller can give, and it is also the option with the largest blast radius for operator intent.

What self-heal does

A controller with self-heal enabled watches the cluster on every reconcile tick. If it sees a resource whose live state differs from the rendered desired state for reasons the controller did not cause (a manual kubectl edit, an out-of-band Helm install, a sidecar injected by a service mesh), it re-applies the desired state. The drift is corrected silently, within the controller’s reconcile interval.

argocd app set payment-api \
  --sync-policy automated \
  --self-heal

flowchart LR
    R["Reconcile tick"] --> D["Compute diff"]
    D -->|"diff from Git"| A1["Apply Git change"]
    D -->|"diff from out-of-band"| A2["Revert drift"]
    D -->|"no diff"| OK["Ready=True"]
    A1 --> OK
    A2 --> OK

In Flux, the equivalent is the spec.selfHeal field on a Kustomization:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: payment-api
spec:
  interval: 10m0s
  prune: true
  selfHeal: true

selfHeal: true is independent of prune. The two are orthogonal: prune deletes orphan resources, self-heal reverts out-of-band edits. A Kustomization with prune: true and selfHeal: false will not delete orphaned resources and will not revert drift; one with both will do both.

When self-heal is right

Self-heal is the right answer when the chart is the single source of truth for live state and any divergence is a bug:

• Stateless workloads. A Deployment whose pods are managed end-to-end by the chart cannot be improved by a manual kubectl scale. Self-heal reverts the scale.
• Admission policies and CRDs. A ValidatingWebhookConfiguration whose CA bundle is managed by the controller must be the controller’s copy. Self-heal reverts a manual rotation.
• Observability stack. DaemonSets like Fluent Bit and node-exporter have no legitimate hand-edit reason; self-heal reverts any edit.
• System namespaces. kube-system, flux-system, argocd - the workloads in these namespaces are owned by the platform team and the chart is the truth.

When self-heal is wrong

Self-heal is wrong when the team has legitimate reasons to edit live state:

• Incident response. An on-call engineer scaling a Deployment during an outage is making a real-time decision. Self-heal reverts the scale on the next reconcile; the engineer is in a race with the controller.
• One-off certificates. A team rotating a TLS certificate manually for a one-off incident has a temporary divergence. Self-heal reverts the rotation.
• Operator-owned fields. Some resources have fields the controller should not own: the kubectl.kubernetes.io/last-applied-configuration annotation, certain finalizers, the resourceVersion. Server-side apply tracks ownership, but legacy client-side apply can lose the ownership signal.

How to disable self-heal per-Application

The escape hatch when self-heal is wrong for one Application but right for the rest of the fleet:

argocd app set payment-api \
  --sync-policy automated

Note the absence of --self-heal. Argo CD treats automated sync without --self-heal as automated sync without drift correction. In Flux:

spec:
  selfHeal: false

The default for Flux is selfHeal: false; Argo CD’s default is no self-heal unless --self-heal is passed. Both controllers ship safe-by-default and require explicit opt-in.

Suspending self-heal during an incident

The discipline for an on-call engineer who needs to make a manual edit and keep it:

1. Suspend the Application’s reconciliation. Argo CD: argocd app set payment-api --pause-reconciliation. Flux: flux suspend kustomization payment-api.
2. Make the manual edit. The controller is paused; the edit stays.
3. Reconcile when ready. Argo CD: argocd app set payment-api --resume-reconciliation. Flux: flux resume kustomization payment-api.
4. Commit the edit to Git. The durable fix. The next automated sync will apply the committed state.

The pause/resume cycle is the on-call engineer’s contract with the controller during an incident.

Production discipline

1. Self-heal is enabled per Application, not per cluster. A cluster with mixed workloads has Applications that need self-heal and Applications that do not.
2. Self-heal is disabled during active incident response. Pause the reconciliation; the on-call owns the cluster until the incident is closed.
3. Self-heal is paired with a corrected Git tree. A typo in the chart under self-heal is a typo that propagates indefinitely.
4. Self-heal is paired with argocd-notifications or Flux alerts. A silent self-heal hides a problem from the operator; an alerting one surfaces it.

Cross-course references

• Git, CI/CD & GitOps for Infrastructure Engineers - Part LXXV-01 (What drift is) is the failure mode self-heal corrects; Part LXXVII-04 (Sync policies and windows) is the policy frame this lesson deepens.
• Kubernetes for Production Sysadmins - Part XVI (Operators and Controllers) covers the reconcile loop that self-heal implements.

Quiz