Configuration drift — detecting and correcting divergence

Configuration drift is when the cluster’s actual state diverges from the Helm release state. This lesson walks the sources, detection, correction, and the operational discipline.

The drift problem

flowchart LR
    A["Helm release: desired state"] --> B["Cluster: actual state"]
    A -.->|drift| B
    C["kubectl edit / apply"] -.->|bypass| B

Helm manages a release; the cluster has the actual resources. Drift occurs when the cluster state changes without Helm knowing:

• kubectl edit / apply from a laptop. Bypasses Helm.
• CI pipeline with cluster credentials. Applies manifests directly, so the release Secret never records the change.
• Auto-scaling changes replicas. If Helm’s values specify a fixed count, HPA-induced drift is reported.

Detection with helm diff

# Compare the current cluster state with the rendered chart
helm diff upgrade myrelease mychart/ -f values.yaml

# Output:
# mychart/templates/deployment.yaml:
#   spec:
#     replicas: 3 (desired)
# -   replicas: 5 (actual)

The helm-diff plugin shows what would change. If there are unexpected changes, drift is present.

Detection with Argo CD selfHeal

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Argo CD’s selfHeal continuously reconciles the cluster state toward the desired state in Git. Drift is detected and corrected automatically.

Sources of drift

flowchart LR
    A[Sources of drift] --> B[kubectl edit from laptop]
    A --> C[kubectl apply from laptop]
    A --> D[CI pipeline bypassing Helm]
    A --> E["Auto-scaling (HPA)"]
    A --> F[Out-of-band scripts]

The sources:

• kubectl edit / apply from laptop. Imperative changes bypass Helm.
• CI pipeline with cluster credentials. Pushes manifests without Helm.
• Auto-scaling. HPA changes replicas; if Helm’s values specify a fixed count, drift is reported.
• Out-of-band scripts. CronJobs or operators that modify resources directly.

Correction

# Bring the cluster back to the desired state
helm upgrade myrelease mychart/ -f values.yaml --atomic --wait

The helm upgrade with the correct values renders the desired state and applies it. Drift is corrected.

For HPA-induced drift, configure Helm to ignore the replicas field (Helm 3 has no direct equivalent of Kustomize’s ignoreDifferences; the workaround is to not manage replicas from Helm when HPA is enabled).

Quiz

The operational discipline

Configuration drift in production rests on five non-negotiable elements:

• Never bypass Helm. All changes go through helm upgrade.
• Enable Argo CD selfHeal. Drift is detected and corrected automatically.
• Monitor drift. Alert on drift that is not auto-corrected.
• Document expected drift. HPA, etc. — list the sources of expected drift.
• Investigate every drift event. A drift event is a signal of a process failure.

Drift is the symptom of a process failure. The discipline is to prevent the process failure, not just to correct the drift.