Drift detection and remediation — keeping live and manifest in sync

Drift is the gap between what you declared and what the cluster has. In a well-managed cluster, the gap is zero: every field on every object matches a manifest in source control, and controllers fill in the fields that operators did not declare. In a poorly-managed cluster, the gap grows: ad-hoc kubectl edits, controllers that add unexpected fields, manual scaling, and forgotten experiments all accumulate. This lesson covers how to detect drift, how to remediate it, and how to alert on sustained divergence.

What drift looks like

Drift is any of:

• A field declared in the manifest but different in the live state.
• A field declared in the manifest but missing in the live state.
• A field present in the live state but declared in the manifest as null/removed.
• A field present in the live state but not declared in the manifest (this is the most insidious kind, because client-side apply preserves it).

flowchart LR
    M[Manifest in Git] -->|declared| D[Detected drift]
    L[Live state] -->|observed| D
    D -->|whichever differs| F[Flag the field]

The four drift categories:

Type 3 is the most common in practice. A controller added a field the manifest did not declare. Client-side apply preserves it; SSA detects the conflict.

Sources of drift

The production sources of drift, in order of frequency:

1. Direct kubectl writes. kubectl edit, kubectl set, kubectl scale, kubectl patch. These bypass GitOps and the next apply will revert them.
2. Controller additions. HPA writes spec.replicas. Service mesh injects sidecars. Cert-manager writes TLS annotations. These are intentional and SSA-friendly.
3. Helm post-rendering. Helm hooks, post-install Jobs, and helm upgrade with new values produce fields not in the original manifest.
4. Operators (CRD controllers). Custom controllers write status fields, add finalizers, and modify specs. These are legitimate and should be reflected in the manifest.
5. Webhooks. Mutating webhooks add fields based on inputs that are not visible to kubectl.
6. Forgotten experiments. Someone ran kubectl exec to debug and the side effects stuck (annotations, labels).

The first category is the dangerous one — it represents unauthorised changes to production. The others are legitimate and should be accounted for in the manifest.

Detection: kubectl diff

The basic detection mechanism:

kubectl diff -f manifests/

This compares every manifest in the directory against the live state and prints a unified diff of the changes. Any non-empty output is drift.

For CI:

# .github/workflows/drift-check.yml
name: Drift check
on: [schedule, workflow_dispatch]
jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: azure/setup-kubectl@v4
    - name: Diff against prod
      run: |
        kubectl diff -f manifests/
      env:
        KUBECONFIG: ${{ secrets.PROD_KUBECONFIG }}

A non-empty diff fails the workflow. This catches drift on a schedule (e.g., every 6 hours) and on demand.

Detection: GitOps controllers

GitOps controllers (ArgoCD, Flux) detect drift continuously. ArgoCD polls the cluster every 3 minutes by default and compares the live state against the Git state. Drift is shown in the ArgoCD UI and emitted as a metric (argocd_app_info with sync_status: OutOfSync).

flowchart LR
    Git[Manifest in Git] --> Argo[ArgoCD]
    Live[Live state] --> Argo
    Argo -->|diff| Compare[Comparison]
    Compare -->|matches| Synced[Synced]
    Compare -->|differs| OutOfSync[OutOfSync]
    OutOfSync -->|alert| Slack[Slack alert]

The GitOps model is continuous reconciliation: the controller detects drift and reverts it automatically (self-heal) or surfaces it for human review.

Detection: kube-state-metrics and alerts

kube-state-metrics exposes the live state as Prometheus metrics. Some metrics expose drift directly:

• kube_deployment_status_replicas vs kube_deployment_spec_replicas — divergence in replica counts (Deployment controller may be lagging or the HPA may have moved them).
• kube_deployment_metadata_generation — increments on spec changes; if it keeps growing without the controller reconciling, the controller is stuck.
• kube_pod_container_status_restarts_total — restarts; not drift but a related symptom.

Alerts:

- alert: KubernetesDeploymentReplicasMismatch
  expr: |
    kube_deployment_spec_replicas{namespace="prod"}
    !=
    kube_deployment_status_replicas{namespace="prod"}
  for: 15m
  annotations:
    summary: "Deployment {{ $labels.deployment }} has spec != status replicas"

This catches sustained divergence (15 minutes) between the operator’s intent (spec) and the controller’s observation (status).

Remediation: the reconcile loop

The right remediation is a reconcile loop: a controller that detects drift and re-applies the desired state. GitOps controllers do this; so does the Deployment controller internally.

flowchart LR
    State[Observe state] --> Diff[Compare to desired]
    Diff -->|matches| Done[No-op]
    Diff -->|differs| Act[Apply desired state]
    Act --> State

The pattern:

1. Observe the live state (LIST/WATCH on the API).
2. Compare against the desired state (the manifest).
3. If they differ, apply the desired state.
4. Repeat forever.

This is the Kubernetes-native pattern; every controller does it. GitOps controllers extend it from cluster-level controllers (Deployment, StatefulSet) to manifest-level controllers (the GitOps agent itself).

Remediation: controlled reverts

When drift is detected but the GitOps controller cannot or should not auto-revert (e.g., the change was authorised), the human path is:

# 1. Capture the live state
kubectl get deployment web -o yaml > /tmp/web-drift.yaml

# 2. Decide: revert or accept?
# - If the change was unauthorised: revert
# - If the change was a controller adding a legitimate field:
#   update the manifest to declare it, then re-apply

# 3. If reverting: apply the manifest
kubectl apply -f manifests/deployment.yaml

# 4. Verify
kubectl diff -f manifests/deployment.yaml
# (empty output = no drift)

The decision matrix:

The hardest case is the last: webhooks that add fields based on inputs the manifest does not see. These require coordination with the webhook author.

Alerting on sustained drift

Production discipline is to alert when drift persists:

# ArgoCD: applications out of sync for more than 30 minutes
argocd_app_info{sync_status="OutOfSync"} == 1
# Joined with argocd_app_sync_status{sync_status_code="OutOfSync"} and a `for:` clause

- alert: ArgoCDAppOutOfSync
  expr: |
    argocd_app_info{sync_status="OutOfSync"} == 1
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "ArgoCD application {{ $labels.name }} is out of sync"

The for: 30m clause prevents alerts on transient drift (a controller briefly out of sync during a rollout). Sustained drift is the signal that something is wrong.

Cross-course references

• The Terraform course part XVII-Terraform-Drift covers state-vs-config drift; the same discipline applies at the cluster level.
• The Ansible course part XXXVI-Ansible-Drift covers configuration drift; cluster drift is the same problem at a different layer.
• The GitOps course (CIII-Kubernetes-GitOps) covers ArgoCD and Flux in depth; both implement continuous reconciliation.

Quiz

Production discipline

• Detect drift continuously. A daily CI job is the minimum; ArgoCD’s 3-minute polling is the standard.
• Distinguish legitimate from unauthorised drift. HPA, service mesh, and webhook additions are not drift; kubectl edits and manual scales are.
• Enable GitOps selfHeal for production applications where the controller can safely revert unauthorised changes. For high-risk applications, disable selfHeal and require manual sync.
• Alert on sustained drift. A for: 24h clause on ArgoCD OutOfSync catches drift that persists past a normal rollout window.
• Make drift a runbook signal, not a notification. A drift alert that no one acts on is worse than no alert. The runbook should specify the action: investigate, revert, or reflect in manifest.