What drift is — the actual diverging from the desired

Drift is the central failure mode of a GitOps system. The previous part established the reconciliation loop and the diff between observed and desired state. This part begins where that diff is non-empty: when what is actually running in the cluster has diverged from what the GitOps controller says should be running. The word drift is the operational name for that divergence.

flowchart LR
    G["Git commit"] --> R["Rendered desired state"]
    R -->|"diff"| D{"Equal?"}
    K["Live API response"] -->|"normalise"| D
    D -->|"yes"| OK["Converged"]
    D -->|"no"| DR["Drift detected"]
    DR --> A["Alert"]
    DR --> S{"Self-heal on?"}
    S -->|"yes"| SH["Reconcile back"]
    S -->|"no"| HOLD["Hold and surface"]

Defining drift precisely

Drift is not a difference between the YAML in Git and the YAML someone ran by hand. Drift is a difference between what the cluster API returns right now and what the rendered manifests at the recorded Git revision say the API should return. Two things follow from this definition:

• Drift is observed, not declared. The controller learns about drift by issuing live queries and comparing their responses to its normalised desired state. A hand-applied resource with identical YAML still drifts if the ownership annotations do not match.
• Drift is per-application. An Argo CD Application owns a set of resources; drift is reported per Application, not per cluster. A different Application on the same cluster is its own diff and its own reconciliation loop.

argocd app diff "$APP_NAME"

This Argo CD CLI command renders the desired state from the recorded Git revision, queries the live state, and prints the normalised diff. An empty result means no drift. Any non-empty line is drift.

The four categories

Drift has four operational categories, and each has a different signature:

• Manual drift. A human applied a change directly to the cluster. The cause is an operator action — kubectl apply, kubectl edit, a console change, a Helm install from the CLI.
• Accidental drift. A controller, scheduler, or side-effect altered the resource. The cause is system behaviour, not intent — a HPA scaling a Deployment’s spec.replicas, a mutating webhook injecting a sidecar, a CSI driver resizing a PersistentVolumeClaim.
• Emergency drift. An incident responder broke the model intentionally to stop the bleeding. The cause is a deliberate break of the contract, made under time pressure, with the intent to repair it later.
• Cascading drift. A change to one resource propagates to another through a controller, webhook, or cross-resource reference. The cause is the dependency graph, and the drift is visible only after the propagation completes.

The rest of this part treats each category separately. The takeaway for this lesson is that “drift” is not one phenomenon but four, and the detection, alerting, and remediation differ in each.

Why drift is surfaced, not silently corrected

A naive GitOps setup reconciles on every tick and silently corrects any drift it finds. A mature setup surfaces drift before correcting it, because silent correction has two costs:

• The correction can hide the cause. If the controller reverts a manual change made by an on-call engineer to fix an incident, the engineer has lost their work and the incident is unresolved.
• The correction can fight a legitimate action. If the on-call engineer is currently applying a fix and the controller is racing them, the operator sees a flaky state they cannot diagnose.

The standard answer is: detect, alert, and then reconcile. Self-heal is a separate flag from auto-sync, and the production discipline is to keep self-heal on for resources that should never be touched by hand and off for resources that operators may legitimately modify during incidents.

Cross-course references

• Kubernetes for Production Sysadmins - Parts on field-manager ownership cover the API-level mechanics that make drift visible to the controller.
• Terraform for Production Sysadmins - Parts on configuration drift cover the same problem in a declarative-infrastructure context, with different detection mechanisms.
• Ansible for Production Sysadmins - Parts on configuration drift cover the analogous problem in configuration management.

Quiz