Continuously reconciled — observed versus desired, and the loop

The fourth OpenGitOps principle - continuously reconciled - is the safety net. The first three principles give the controller something to converge against (declarative desired state), something it can trust (versioned and immutable), and a direction of change (pull, not push). The fourth principle is the loop itself: the controller observes, computes a diff, and converges, on a cadence, forever.

A reconciliation tick in one picture:

flowchart LR
    A["Git ref"] -->|"desired state"| R["Reconciler"]
    C["Cluster API"] -->|"observed state"| R
    R -->|"diff"| D{"Empty?"}
    D -->|"yes"| OK["Healthy"]
    D -->|"no"| P["Patch / create / delete"]
    P -->|"applied"| C
    P -->|"re-observe"| C

The reconciler reads the desired state from Git, reads the observed state from the cluster API, computes a diff, and acts. If the diff is empty, the system is healthy. If the diff is non-empty, the reconciler applies the changes and re-observes. The tick ends when the diff is empty, and the loop restarts on the next interval.

What the reconciler actually does

The reconciler’s contract is short:

1. Read. Pull the desired state from the source (Git ref, OCI artifact, Helm chart).
2. Observe. Query the cluster API for the live state of the resources the desired state describes.
3. Diff. Compute the delta between desired and observed.
4. Apply. Create, update, or delete resources to converge.
5. Re-observe. Confirm the cluster matches the desired state. If not, retry with backoff.

The diff is the work; the cluster API call is the cost; the re-observe is the verification. A reconciler that does not re-observe is a one-shot apply - which is not GitOps, it is a push from a different direction.

Three modes of reconciliation

Different teams need different levels of automation. GitOps controllers expose three modes that trade autonomy for safety:

1. Read-only. The controller reports the diff between desired and observed but does not apply. A human reviews the diff and clicks “Sync” to apply. Useful for production systems where every change must be eyeballed.
2. Sync-wave / manual approval. The controller computes the diff and applies it, but waits for an approval before destructive operations (deletes, certain kinds of updates). Useful for production systems where the loop is trusted but the deletes are not.
3. Automated self-heal. The controller applies any change that converges toward the desired state, including re-creating resources that have drifted or been deleted. Useful for development and for systems where the desired state is genuinely the contract.

argocd app list

This Argo CD CLI command lists every Application and its current sync status - Synced, OutOfSync, or Unknown. An OutOfSync Application is one where the observed state diverges from the desired state; the controller knows about the divergence and is waiting (in read-only mode) or has already applied (in automated mode).

Drift, and how the loop corrects it

Drift is the divergence between the cluster state and the desired state. Drift has three causes:

• Manual edits. An operator runs kubectl edit to fix an incident and forgets to commit the change. The next reconcile reverts the manual edit.
• External actors. A load balancer, an autoscaler, a cloud provider changes a piece of state the controller thought it owned. The next reconcile reverts it.
• Bugs. A controller bug that applies the wrong state. The next reconcile corrects it.

In all three cases, the loop is the recovery mechanism. Without the loop, drift accumulates and the cluster diverges from the record. With the loop, drift is bounded by the reconciliation interval.

What “continuously” means in practice

The reconciliation interval is configurable. Common values:

• 30 seconds. Hot production; tight latency budget for hot-fixes; high API-server load.
• 3 minutes. Default for most controllers; comfortable balance between latency and load.
• 10 minutes. Staging environments where the latency budget is loose.

The interval is the upper bound on how long a commit can take to land. A 3-minute interval means a commit can take up to 3 minutes to be applied. A 30-second interval means a commit is usually applied within a minute.

The lower bound is the cost of the reconcile. Each tick produces API-server load: a List of every resource the Application owns, plus a Get per resource to compute the diff. A cluster running 1000 Applications at a 30-second interval produces a steady-state API-server load that has to be sized for.

Production discipline

1. Two metrics on a dashboard. Reconciliation latency (commit SHA to applied state, in seconds) and drift count (number of Applications in OutOfSync for longer than one tick) are the operational health signals.
2. Automated self-heal in dev, read-only in prod by default. The promotion gate is the lever; the cluster only converges unattended when the team has earned the trust.
3. Sync waves for order-sensitive resources. CRDs and namespaces must exist before the workloads that depend on them. Sync waves enforce the order at apply time; without them, the controller can race.

Cross-course references

• Ansible for Production Sysadmins - Part XXXIX covers the AWX reconciliation loop, which is the same model for configuration management.
• Terraform for Production Sysadmins - Part XXIX covers the operator-pattern Terraform Cloud run, which is the pull-mode reconciliation for IaC.
• Kubernetes for Production Sysadmins - The controllers-and-operators pattern is the same loop the GitOps controller runs at the cluster level.

Quiz