Uncontrolled drift — when the cluster forgets Git

A GitOps controller without self-heal is a watchdog that barks but does not bite. The controller polls the cluster, compares the live state to the desired state in Git, and reports OutOfSync when the two diverge. With self-heal disabled, the report is the entire response.

The three controls drift needs

Drift is manageable when the controller combines three controls. Remove any one and the pattern becomes unsafe.

    flowchart LR
        A["Cluster diverges from Git"] --> B["Self-heal reconciles"]
        B --> C["Cluster matches Git"]
        D["IgnoreDifferences"] --> B
        E["Reconciliation budget"] --> F["Alert on persistent drift"]
        F --> G["Operator investigates"]

• Self-heal. The controller’s reconcile loop applies the desired state when the live state diverges. Argo CD’s automated.selfHeal: true; Flux’s Kustomization controllers’ implicit reconcile-on-drift.
• IgnoreDifferences. A per-field declaration naming the fields that legitimately differ from Git.
• Reconciliation budget. A monitoring signal that alerts on persistent reconcile failure.

The team that has none has a cluster that drifts without bound and an audit trail that records the divergence only after an incident.

Why self-heal is not optional

A controller without self-heal is a controller that requires a human to be the reconciler. A kubectl scale deployment from three to ten replicas, never reverted, is a divergence that persists for months because no one reads the controller’s report.

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  syncPolicy:
    automated:
      selfHeal: true

The flag above is the difference. With the flag, the next reconcile applies the desired state and reverts the cluster-side change. Without the flag, the reconcile records the change and waits for a human.

IgnoreDifferences for legitimate divergence

A Kubernetes Service gets a clusterIP assigned at create time by the API server. The value is not in the manifest; the controller’s diff would mark the Service OutOfSync on every reconcile. The fix is an ignoreDifferences declaration:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  ignoreDifferences:
    - group: ""
      kind: Service
      jsonPointers:
        - /spec/clusterIP
    - group: autoscaling
      kind: HorizontalPodAutoscaler
      jsonPointers:
        - /spec/replicas

The declaration names the fields that legitimately diverge. The un-disciplined use of ignoreDifferences is the same anti-pattern as disabling self-heal globally.

The reconciliation budget as an alert

Self-heal applies the desired state; the cluster matches Git. When the reconcile fails - a manifest the controller cannot apply, a webhook that rejects the resource - the controller retries. Argo CD records the failure in argocd_app_sync_total{result="error"}; Flux records it in flux_reconcile_total{result="error"}. A reconciliation budget alerts when the rate of failures exceeds a threshold.

- alert: GitOpsReconcileFailing
  expr: |
    sum(rate(argocd_app_sync_total{result="error"}[15m])) > 0.1
  for: 15m
  labels:
    severity: page

The alert fires when the controller cannot recover on its own. The team’s response is to read the controller’s logs, identify the failing Application, and decide whether to fix the manifest.

Production discipline

1. Self-heal is enabled by default. The opt-out is per-resource, with an ignoreDifferences declaration.
2. ignoreDifferences entries are reviewed.
3. A reconciliation budget alerts on persistent failure.
4. Drift incidents trigger a follow-up.

Cross-course references

• This course, Part LXXVII (ArgoCD) - the reconcile loop in detail.
• This course, Part LXXXI (SyncedAndHealthy) - the synced/healthy matrix.

Quiz