Drift detection — finding and correcting out-of-band changes

Drift detection is one of GitOps’ most valuable features. This lesson walks what drift is, how the controller detects it, the difference between desired and unexpected drift, and the operational discipline.

What is drift

flowchart LR
    A["Git: desired state"] --> B[GitOps controller]
    C["Cluster: actual state"] --> B
    B -->|compare| D{Drift?}
    D -->|Yes| E[Report or correct]
    D -->|No| F[In sync]

Drift is when the cluster’s actual state differs from Git’s desired state. The GitOps controller detects drift by comparing the manifests in Git to the resources in the cluster.

argocd app diff billing

===== Deployment/billing =====
  replicas: 3 (Git)
- replicas: 5 (cluster)

The diff shows the difference: Git has 3 replicas; the cluster has 5. The controller reports the drift and (if selfHeal is enabled) corrects it.

How drift happens

flowchart LR
    A[How drift happens] --> B[kubectl apply from laptop]
    A --> C[kubectl scale from laptop]
    A --> D[kubectl edit from laptop]
    A --> E[Helm install bypassing Git]
    A --> F["kustomize build / kubectl apply"]
    A --> G[CI pipeline with cluster credentials]
    A --> H[Auto-scaling changing replicas]

Drift comes from:

• kubectl apply / scale / edit from a laptop. Imperative changes bypass Git.
• Helm install bypassing Git. A Helm release applied without a Git-tracked values file.
• CI pipeline with cluster credentials. Push-based CD bypassing the GitOps controller.
• Auto-scaling. HPA changes replica counts; if Git specifies a fixed count, drift is reported even though it’s expected.

The first three are process failures. The fourth (HPA) is expected drift that the GitOps controller should be configured to ignore.

Desired vs unexpected drift

flowchart TD
    A[Drift] --> B{Expected?}
    B -->|Yes HPA, PDB, kubectl debug| C["Desired drift: ignore or report"]
    B -->|No manual edits| D["Unexpected drift: alert and correct"]

Two types:

• Desired drift. Expected changes that the controller should not revert. Examples:
  • HPA scaling replicas.
  • Pod restarts (Pod IP changes).
  • PDB replacing unhealthy Pods.
  • kubectl debug ephemeral containers.
• Unexpected drift. Process failures. Examples:
  • kubectl apply from a laptop.
  • kubectl edit to bypass review.
  • Helm install without Git tracking.
  • CI pipeline with cluster credentials.

The controller should be configured to ignore desired drift and to alert on unexpected drift.

Self-healing vs alerting

flowchart LR
    A[Drift detected] --> B{selfHeal enabled?}
    B -->|Yes| C[Revert cluster to Git state]
    B -->|No| D["Report drift, alert"]
    C --> E["Audit: who caused the drift?"]
    D --> E

The controller can:

• Self-heal. Automatically revert the cluster to Git state. The drift disappears.
• Alert. Report the drift; humans investigate.

Self-heal is appropriate for production manifests where drift is a process failure. Alerting is appropriate when drift may be legitimate (debug Pods, HPA scaling) or when the team needs to investigate before reverting.

Quiz

The operational discipline

Drift detection in production rests on five non-negotiable elements:

• Configure ignoreDifferences. Fields managed by Kubernetes controllers (HPA replicas, pod-template hash) should be ignored.
• Alert on drift. Drift is a process failure signal. Alert on uncorrected drift.
• Investigate drift immediately. Every drift event is a clue to a process failure. Find the cause.
• selfHeal for production. Production manifests should be self-healed; alerting alone is not enough.
• Document expected drift. The runbook should list expected drift sources (HPA, PDB) and the ignoreDifferences configuration.

Drift is the signal that the GitOps process is failing. The discipline is to alert, investigate, and prevent future drift.