Argo CD reconciliation mechanics — the three components, the cache, the diff

Argo CD splits reconciliation across three components and uses a cache to make ticks fast. Understanding the split is the difference between “Argo CD is broken” and “the application controller is waiting on the repo server, which is waiting on a stale cache entry”. Each component has a job; the cache ties them together.

flowchart LR
    G["Git repo"] -->|"fetch"| RS["Repo server"]
    RS -->|"manifests + cache"| AC["Application controller"]
    AC -->|"observe"| K["Cluster API"]
    K -->|"observed state"| AC
    AC -->|"apply / sync"| K
    AC -->|"status"| UI["Argo CD UI / CLI"]

The diagram shows the three components - repo server, application controller, cluster API - and the cache that sits between the repo server and the application controller. The operator-facing surface is the UI and the CLI.

The three components

The repo server. The repo server is the component that knows how to render manifests from a source. It runs helm template, kustomize build, plain YAML, and the various plugin types. It fetches from Git on its own cadence and stores rendered manifests in memory, indexed by revision and path. The repo server is stateless from the application’s perspective - any application controller can query any repo server.

The application controller. The application controller is the component that drives reconciliation. There is one per Argo CD Application object in the cluster (or, in sharded setups, a subset). It owns the tick: it queries the repo server for the desired state at a revision, queries the cluster API for the observed state, computes the diff, and applies the sync. It also reports status to the Argo CD UI.

The cluster API. The cluster API is not an Argo CD component

• it is the destination Kubernetes API server. Argo CD calls it to observe and apply. The credentials Argo CD uses are a ServiceAccount token scoped to the namespaces and resource kinds the Application declares.

The repo server cache

The repo server caches rendered manifests. The cache is keyed by Git revision and source path, and invalidated when:

• A new commit lands on the watched branch or tag (the repo server’s Git fetch detects it).
• The cache entry expires on its TTL (default 24 hours).
• An operator forces a refresh.

The cache makes ticks fast. An application controller running a tick against an unchanged revision does not pay the cost of re-rendering the manifests; it gets the cached output in milliseconds. Without the cache, every tick would pay the render cost - which is the dominant cost for large Helm charts or complex Kustomize overlays.

The cache also introduces a failure mode: a stale cache entry can report a different diff than a fresh render would. The argocd app refresh command forces a refresh; it invalidates the cache entry for the application’s source and triggers a new Git fetch.

The diff path

The diff path is the heart of the tick:

1. Application controller queries the repo server at the recorded source revision. The repo server returns the cached (or freshly rendered) manifests.
2. Application controller queries the cluster API for the live state of every resource the manifests describe.
3. Application controller runs the diff engine. The diff engine normalises both sides and produces a structured diff.
4. If automated self-heal is enabled and the diff is non-empty, the application controller schedules a sync.

argocd app diff "$APP_NAME"

This CLI command runs the diff engine against the current state without applying. It is the right command for “show me what the next tick will do” without paying the apply cost. The output shows normalised differences only; defaulted fields and status fields are excluded.

argocd app sync "$APP_NAME" --prune --self-heal

This CLI command runs a sync with two important flags. --prune deletes resources that exist in the cluster but not in the desired state (for example, a resource that was deleted in Git but is still in the cluster). --self-heal re-applies the desired state even if the cluster already matches it; this is useful for re-injecting controller annotations after a hand apply. Both flags make the sync more aggressive; both are appropriate for production with the right safeguards (RBAC scoping, sync windows, drift detection alerts).

Sync windows and waves

Argo CD applies sync windows and sync waves on top of the mechanics:

• Sync windows. A window is a time range during which syncs are allowed or denied. Production windows might be “deny syncs between 09:00 and 17:00 on weekdays” to force human review during business hours.
• Sync waves. A wave is an integer annotation that orders syncs within an application. Resources in wave -5 apply before resources in wave 0, which apply before wave 5. The application controller waits for one wave to converge before starting the next. CRDs and namespaces typically get negative waves; workloads typically get wave 0.

Production discipline

1. Watch the repo server’s cache hit rate. A healthy Argo CD installation has a high cache hit rate; a low rate suggests either rapid source churn (which is fine) or excessive refreshes (which is wasteful). The repo server exposes Prometheus metrics for this.
2. Use --prune deliberately. Pruning deletes cluster resources that are not in the desired state. This is the right behaviour in steady state but is dangerous during a partial Git history rewrite, where a transient state may temporarily lack resources that exist in the cluster.
3. Run sync windows in production. A sync window that denies automatic syncs during business hours forces a human review gate. The window does not prevent manual syncs; it prevents the automated self-heal loop from reverting an out-of-band fix that an operator has not yet committed.

Cross-course references

• Kubernetes for Production Sysadmins - Parts on RBAC and admission controllers cover the cluster-side mechanics Argo CD relies on.
• Terraform for Production Sysadmins - Parts on the operator pattern cover the Terraform analogue of the repo server / controller split.
• Helm for Production Sysadmins - Parts on Helm release management cover the render-and-cache mechanics.

Quiz