GitOps reconciliation and drift — recap

GitOps is not a product; it is a set of four properties that any system can implement: the desired state is declarative, the desired state is versioned in Git, the controllers pull the desired state, and reconciliation runs continuously. Get any one of the four wrong and the system is no longer GitOps — it is “configuration in Git” with some pieces missing.

The four principles and what breaks when each one is violated

flowchart TD
    A["Desired state"] -->|declarative| B["YAML, manifests, Helm chart"]
    B -->|versioned| C["Git commit, signed, immutable ref"]
    C -->|pulled| D["Controller reads Git"]
    D -->|reconciled| E["Loop: observed = desired"]
    style A fill:#dff
    style B fill:#dff
    style C fill:#dff
    style D fill:#dff

• Declarative. The desired state is expressed as data, not procedure. A Deployment manifest is declarative; a bash script that creates a Deployment is not. Violation: imperative operators that produce state without leaving a record.
• Versioned. The desired state lives in Git, with every change a commit and a pull request. Violation: manifests edited in place via kubectl edit or applied from a laptop.
• Pulled. The controller in the cluster fetches the desired state from Git; the deploy system does not push manifests into the cluster’s API. Violation: CI systems that kubectl apply against the cluster API.
• Continuously reconciled. The controller runs the diff-and-apply loop on a schedule (or on Git webhook) for as long as the application exists. Violation: one-shot apply pipelines that do not retry.

The reconciliation loop

The loop, in five steps, is identical in Argo CD and Flux:

1. Fetch. The controller reads the desired state from Git (Argo CD: from the configured repo URL; Flux: via the GitRepository and Kustomization CRDs).
2. Observe. The controller reads the observed state from the cluster (via the Kubernetes API).
3. Diff. The controller computes the difference between desired and observed.
4. Apply. The controller applies the diff to the cluster (Argo CD: server-side apply; Flux: server-side apply with owner annotations).
5. Report. The controller updates the status (Application.Status in Argo CD, Kustomization.Status in Flux) to reflect whether the observed state now matches the desired state.

flowchart LR
    A[Fetch] --> B[Observe]
    B --> C[Diff]
    C -->|"desired != observed"| D[Apply]
    D --> B
    C -->|"desired = observed"| E[Healthy]

The synced and healthy matrix

A controller’s status is two orthogonal fields, and a failure in one is not a failure in the other:

• Synced / NotSynced. Does the observed state in the cluster match the desired state in Git? A NotSynced status means a sync is pending or has failed.
• Healthy / Degraded. Are the resources in the desired state actually working? A Degraded status means the resources are applied but failing (a CrashLoopBackOff, an ImagePullBackOff, a failed readiness probe).

The four combinations map to four operational states:

Drift — the three types and the matching controls

Drift is any change to the observed state that did not come through the reconciliation loop:

• Manual drift. Someone ran kubectl edit, kubectl scale, or kubectl patch to change a resource directly. The next reconciliation will revert the change if self-heal is enabled; otherwise it persists.
• Accidental drift. A node failure, a network partition, or a controller crash left the cluster in a partial state. Self-heal converges back to the desired state on the next reconciliation.
• Emergency drift. A break-glass change made during an incident, applied via kubectl apply or a hot-patch. The change is real and may need to stay; the after-incident work is to reconcile Git to match the cluster, or to revert the cluster to match Git.

Production discipline

The five rules that recur across every GitOps deployment:

1. The controller is the only writer. Any change to a managed resource that does not go through the controller is drift. The cluster’s API is reached by the controller and by emergency break-glass procedures, full stop.
2. Self-heal on, prune off by default. Self-heal converges drift; prune removes orphans. The first is a guardrail; the second is a footgun.
3. Break-glass is logged. A break-glass change bypasses Git; the after-incident work is to put the change back into Git so the audit trail survives.
4. kubectl edit is not a workflow. If a change is worth making in the cluster, it is worth committing in Git so the next reconciliation does not undo it.
5. The status fields are the dashboard. Synced and Healthy are not metrics; they are the controller telling you whether the cluster matches what you said it should match. An Application that is NotSynced for an hour is an Application whose desired state has drifted from its observed state.

Cross-course references

• Kubernetes for Production Sysadmins — Controllers and reconciliation loops are the core pattern of Kubernetes itself; the GitOps controller is a specialisation of the same loop.
• Terraform for Production Sysadmins — Terraform apply is a one-shot reconciliation; Terraform drift detection is the same question as Argo CD’s drift detection, with a different cadence.

Quiz