The reconciliation loop — observed, desired, difference, action

The reconciliation loop is the unit of work in a GitOps system. Every controller - Argo CD, Flux, the Kubernetes controllers underneath both - runs the same loop. One tick reads the desired state, reads the observed state, computes a difference, and acts on the cluster to converge. The loop is what distinguishes a GitOps controller from a push-based CD pipeline: it never terminates. It pauses when the diff is empty and resumes on the next interval.

flowchart LR
    O["Observe cluster"] --> D{"Diff empty?"}
    R["Read desired from Git"] --> D
    D -->|"yes"| W["Wait for next tick"]
    D -->|"no"| A["Apply / patch / delete"]
    A --> O
    W --> O

The diagram compresses the loop into four arrows. The four phases are what every tick does, in this order, on every controller.

The four phases

A single reconciliation tick has four phases, and each one has a cost.

1. Read. The controller reaches its source - Git for declarative manifests, an OCI registry for packaged charts, an S3 bucket for Terraform state - and pulls the desired state at a specific revision. The revision is recorded; it is the source of truth for what “desired” means this tick.

2. Observe. The controller queries the cluster API for the live state of every resource the desired state describes. The query is a List of each resource kind, followed by a Get per resource to fetch the full object. This phase is where most of the tick’s latency lives.

3. Diff. The controller compares the desired state from step 1 against the observed state from step 2. The diff is structured, not textual: it normalises both sides, ignores server-default fields, and reports only meaningful changes. An empty diff means the cluster is converged; a non-empty diff means the controller has work to do.

4. Apply. The controller applies the diff to the cluster. The apply is not a blind kubectl apply; it is the controller’s own apply path, which records annotations (the controller’s identity, the source revision) on every resource it owns. The annotations are what allow the next tick to identify which resources the controller should observe.

argocd app diff "$APP_NAME"

This Argo CD CLI command runs a diff without applying. It is a useful way to see what phase 3 would report for a single application without paying the cost of phase 4.

Tick versus sync

A tick is the full loop. A sync is only phase 4. The distinction matters because:

• A controller can run a tick that produces an empty diff. Nothing is synced, but the tick still happened - and the loop is still alive. An empty diff is a healthy tick.
• A controller can be forced to sync out of band - an operator clicks “Sync”, a CI job runs argocd app sync - without a tick happening. The sync applies the diff; the next tick confirms.
• A controller can fail in any phase. A failed read is different from a failed observe is different from a failed apply. The operational signal depends on which phase broke.

The loop never terminates

The decisive property of the loop is that it never terminates. A tick ends when the diff is empty; the loop does not. On the next interval - three minutes later by default - the tick runs again. The next tick may find an empty diff (still converged) or a non-empty diff (something drifted).

This is why reconciliation is “continuously” reconciled. The controller does not wait for a webhook, a pipeline run, or a human click. It ticks on its own cadence and converges opportunistically. The cadence is the only thing the operator controls; the work the controller does each tick is deterministic.

Production discipline

1. Watch the loop, not the activity. A controller whose ticks are finding empty diffs is healthy. Alert on “no successful reconcile in N intervals” instead of “no activity in N intervals”.
2. Treat each phase as a distinct failure domain. Read fails look like credential or source problems. Observe fails look like RBAC or API-server problems. Apply fails look like resource validation problems. The remediation differs per phase.
3. Resist out-of-band convergence. A manual kubectl apply during an incident buys time and costs drift. The cost is bounded by the reconciliation interval; the time bought is bounded by the operator’s ability to commit the fix in Git.

Cross-course references

• Kubernetes for Production Sysadmins - The controllers-and-operators pattern in the Kubernetes course is the same loop running at the cluster level for built-in resources like Deployment and Service.
• Ansible for Production Sysadmins - Parts on AWX cover the configuration-management analogue of the same loop.
• Terraform for Production Sysadmins - Parts on the operator pattern cover the pull-mode reconciliation for infrastructure.

Quiz