Manual drift — the operator edits the cluster directly

Manual drift is the category produced when a human acts on the cluster directly. The most common form is kubectl edit, the second is kubectl apply -f, and the third is a console patch through a UI like the OpenShift web console or Lens. The reconciliation loop does not care which path produced the change — it only sees the live state and the diff against the rendered desired state.

sequenceDiagram
    participant Op as Operator
    participant K as Cluster API
    participant C as GitOps controller
    Op->>K: kubectl edit deployment/web
    K-->>Op: OK
    Note over K: Observed state updated
    C->>K: Get/List (next tick)
    K-->>C: Live state with edit
    C->>C: Diff vs rendered desired
    C-->>C: OutOfSync
    alt self-heal on
        C->>K: Apply desired (patch)
        K-->>Op: Edit reverted
    else self-heal off
        C-->>Op: Alert only
    end

The canonical signature

The signature of manual drift in an Argo CD Application is specific and recognisable. The Application status moves to OutOfSync, the sync status records the diff between the desired and observed states, and the live resource carries the operator’s edit. The Argo CD CLI makes the diff explicit:

argocd app diff "$APP_NAME"

The output names each resource and each field that differs. For a kubectl edit that added an environment variable, the diff will show one block under spec.template.spec.containers[0].env listing the new entry. For a kubectl scale the diff will show spec.replicas. For a label added through the OpenShift console the diff will show metadata.labels.

The sync status alone is enough to detect manual drift; the diff explains what changed. The combination is the canonical production signal.

Why the audit trail is incomplete

A manual edit has a partial audit trail by definition. The Kubernetes API server records the change in its audit log — who issued the request, from which client, at what timestamp. The GitOps controller records the diff it observed. But neither record captures why the change was made. The why is in the engineer’s head, or in a Slack thread, or in an incident channel that is not part of the durable audit.

This is why the production rule is that every change, even during an incident, must be committed to Git. The commit is the only durable record of intent. The audit log records that a change was made; the commit records that it was approved. When the change is reverted by the controller and the operator re-applies, the commit is the only way to break the loop.

The race between hand-applies and self-heal

When self-heal is on, the controller reverts the manual edit within one reconciliation interval. The operator sees their work disappear. The standard response is to re-apply, which triggers another revert. The result is a flaky state where the resource oscillates between the operator’s edit and the controller’s reversion on every tick.

The production discipline for this race is one of three options:

• Commit the change in Git so the desired state matches. This is the preferred path. The change is now durable, auditable, and self-heal will preserve it because it is the declared state.
• Disable self-heal for the duration of the incident. The controller still reports drift, but does not revert. After the incident the operator commits the change and re-enables self-heal.
• Accept the revert and lose the work. The wrong choice in almost every case, but documented here for completeness — the controller will win eventually because it runs on every tick.

How to detect manual drift from outside

The Application status is the immediate signal, but a stronger production check is to query the Application object directly and inspect the conditions and sync status fields. This works across all Applications in a cluster, not just the one an operator is debugging:

kubectl -n argocd get application -o yaml

The output lists every Application in the argocd namespace, each with its status.sync.status (Synced or OutOfSync), its status.conditions, and its status.operationState. A production dashboard surfaces the count of OutOfSync Applications over time, the count of resources per Application in drift, and the age of each drift — old drift is drift that has been ignored, which is its own operational problem.

Production discipline

1. Treat the cluster as read-only except for the controller. Operators interact with the desired state through Git and the GitOps toolchain, not through direct cluster writes. Direct writes are reserved for documented emergencies.
2. Treat every manual edit as a future incident. If an operator edits a resource directly, the next incident is the one where that edit is lost. The remediation is to commit the change before walking away.
3. Track the age of every drift. A 30-second-old drift is the operator’s current work; a 30-day-old drift is a piece of the declared state the cluster no longer matches. The two require different responses.

Cross-course references

• Kubernetes for Production Sysadmins - Parts on audit logging cover the cluster-side record of the manual edit; the GitOps side adds the commit-level record.
• Linux for Production Sysadmins - Parts on configuration drift cover the equivalent problem in configuration files.
• Terraform for Production Sysadmins - Parts on Terraform state cover the same race between manual operations and declared state.

Quiz