Accidental drift — controllers, schedulers, and side effects

Accidental drift is the category produced when a cluster component — a controller, a scheduler, a webhook, a sidecar injector — alters a resource without an operator issuing the command. The diff is identical to a manual edit from the GitOps controller’s perspective. The cause is invisible to the diff engine, and identifying it requires reading the audit log and the controller logs, not the diff.

flowchart LR
    M["Manifest in Git"] --> C["Controller applies"]
    C --> K["Resource on cluster"]
    H["HPA"] -->|"scales replicas"| K
    W["Mutating webhook"] -->|"injects sidecar"| K
    S["Scheduler"] -->|"assigns node"| K
    V["CSI driver"] -->|"resizes volume"| K
    K -->|"query"| O["Observed state"]
    O -->|"diff vs desired"| DR["Drift reported"]

The four common sources

Four sources account for most accidental drift in a production cluster:

• HorizontalPodAutoscaler. The HPA writes spec.replicas on the Deployment it scales. If the Deployment has a desired replica count in Git, the HPA’s value drifts from it. The diff will show one line for spec.replicas; the cause is the autoscaler, not an operator.
• Mutating webhooks. An admission webhook can inject a sidecar container, an annotation, or a label on every resource created in a namespace. The diff will show the injected field; the cause is the webhook.
• Scheduler-driven defaults. When a Pod’s spec.nodeName is unset in the manifest and a scheduler plugin or a defaulting controller fills it in, the diff will show spec.nodeName. The Pod’s status fields are populated by the kubelet and ignored by the diff, but some defaulters operate on the spec.
• CSI driver side effects. A PersistentVolumeClaim’s spec.resources.requests.storage can be expanded by the CSI driver in response to a volume expansion request, leaving the PVC spec larger than the manifest in Git.

In each case the GitOps controller sees a one-line diff and reports drift. The cause is in the cluster, and the remediation is to fix the upstream cause, not to re-apply the resource.

Why the diff engine surfaces it

The diff engine’s job is to compare observed and desired state. It has no signal that the difference was produced by a controller rather than an operator. If the diff engine ignored controller-driven changes, it would also ignore operator-driven changes, and the entire drift detection system would be useless. The classification is outside the diff engine; the diff engine’s only job is to surface the difference.

The classification is done by the on-call engineer reading the diff and asking “who else writes this field?”. If the field is spec.replicas on a Deployment owned by an HPA, the answer is the HPA. If the field is metadata.annotations and the value matches an admission webhook’s sidecar pattern, the answer is the webhook. If neither pattern matches, the answer is an operator.

Reading the diff to classify

The diff line itself usually contains enough context to classify the drift source. A spec.replicas change on a Deployment that has a HPA reference is almost always HPA. A sidecar container with a name matching an admission webhook’s known injection pattern is almost always the webhook. A spec.nodeName filled in on a Pod whose manifest omits it is almost always a defaulting controller.

When the diff does not classify cleanly, the next step is the audit log. The Kubernetes audit log records every write to the API server, including the user identity. A field manager identity of kube-controller-manager or horizontal-pod-autoscaler indicates a controller wrote the field. A field manager identity of kubectl-edit indicates an operator. The classification moves from “diff says X” to “diff says X and the audit log says Y wrote it”.

Production discipline

1. Audit every controller that writes to resources you own. A production GitOps repository should list every controller, webhook, and defaulting service that can write to its resources, and the fields each one writes. The list is the basis for classifying future drift.
2. Treat HPA writes as ownership transfers. When an HPA is created for a Deployment, spec.replicas is now owned by the autoscaler. The Deployment’s manifest should not set it; the HPA should set minReplicas and the autoscaler should be free to scale within maxReplicas.
3. Disable diff ignore for fields the controllers write. The diff engine’s normalisations exist to suppress noise, but they can also hide accidental drift. Production audits should run a comparison that does not strip the fields the controllers are known to write.

Cross-course references

• Kubernetes for Production Sysadmins - Parts on admission controllers, HPAs, and CSI drivers cover the cluster components that produce accidental drift.
• Terraform for Production Sysadmins - Parts on resource drift cover the analogous problem in declarative infrastructure provisioning.
• Ansible for Production Sysadmins - Parts on fact-driven drift cover the equivalent in configuration management.

Quiz