Observed versus desired — what "actual" and "what should be" mean

The reconciliation loop compares two things. The observed state is what the cluster API returns right now, queried live. The desired state is what the rendered manifests at a specific Git revision say it should be. The two are not the same shape, and the asymmetries between them are what make the diff non-trivial.

flowchart LR
    G["Git commit SHA"] -->|"render"| M["Manifests YAML"]
    M -->|"parse + normalise"| D["Desired state object"]
    K["Cluster API Get/List"] -->|"response"| O["Observed state object"]
    D -->|"diff engine"| R{"Equal?"}
    O --> R
    R -->|"yes"| OK["Converged"]
    R -->|"no"| A["Patch needed"]

The diagram shows two paths arriving at the diff engine. One path goes through Git and a render step. The other goes through the cluster API. The output of each path is a structured object; the diff engine compares them.

What “desired” really means

The desired state is not the YAML in the Git repository. The YAML in the repository is input; the desired state is what that input renders to after the controller’s toolchain processes it.

A few examples of the gap between repository YAML and rendered desired state:

• Helm charts. A values.yaml and a Chart.yaml produce rendered manifests through helm template. The desired state is the rendered output, not the chart sources.
• Kustomize overlays. A kustomization.yaml with patches and resources produces a final YAML through kustomize build. The desired state is the built output.
• Plain manifests. A directory of Deployment, ConfigMap, and Service YAML files is its own desired state once parsed.
• OCI artifacts. A packaged chart or kustomize output stored in an OCI registry. The desired state is the contents of the artifact at a specific tag or digest.

The render step is part of what makes the desired state trusted. The same inputs, run by the same toolchain at the same version, produce the same outputs - and the controller records the Git revision it used, so the desired state at any tick is reproducible.

What “observed” really means

The observed state is what the cluster API returns when the controller asks. The controller does not consult a cache, a backup, or its memory; it issues a live Get (for a single resource) or a List (for every resource the application owns) and uses the response.

The observed state has asymmetries the YAML in Git does not:

• Server-defaulted fields. When a Deployment is created without a spec.strategy block, the API server fills in RollingUpdate with sensible defaults. The observed state contains those defaults; the YAML in Git does not.
• Normalised ordering. Lists in the API response may have keys reordered, timestamps formatted, and empty fields removed. The YAML in Git may have the keys in author order.
• Controller-injected annotations. Every resource the GitOps controller applies carries annotations identifying the application, the source revision, and the controller’s identity. The YAML in Git does not.
• Status fields. Live status subresources - replicas ready, conditions, observed generation - are populated by other controllers and present in the API response but not in the input YAML.

argocd app diff "$APP_NAME"

This Argo CD CLI command shows the diff between the desired state (rendered from Git at the recorded revision) and the observed state (fetched live from the cluster API). The diff output excludes status fields and defaulted fields by default; this is why two YAML files that look identical can produce a clean “no diff” result while a hand-applied resource with the same YAML produces drift.

The diff engine normalises both sides

The diff engine cannot compare the YAML in Git directly to the API response. It has to construct two normalised objects first:

1. Normalise the desired state. Parse the rendered YAML into structured objects. Strip fields the API server does not persist. Expand shorthand. Resolve references.
2. Normalise the observed state. Strip status fields (they are not part of desired state). Strip defaulted fields the desired state does not specify. Strip controller-injected annotations.
3. Compare the normalised objects field by field. Report only meaningful differences.

The normalisation is why the diff is robust against the asymmetries. A Deployment with no spec.strategy block in Git will not diff against a Deployment whose API representation includes strategy: {type: RollingUpdate}—both sides are normalised to omit the field, and the diff is empty.

Why the asymmetry matters

The asymmetry between desired and observed has three operational consequences:

• A clean diff is not the same as no changes. The controller’s diff ignores defaulted fields and status fields, so a Deployment that was hand-edited to set spec.template.metadata.annotations will drift, but a Deployment whose replicas were scaled by the autoscaler will not - the scaling is recorded in status.replicas, which the diff ignores.
• Drift in defaulted fields is invisible to the diff. If an operator manually edits a defaulted field, the diff engine may or may not surface it depending on whether the field was in the desired state at render time. The rule is: if the field was in the desired state, drift is visible; if it was not, drift is invisible.
• Annotations travel with ownership. A resource the controller creates carries the controller’s annotations even if the YAML in Git does not specify them. A hand-applied resource with the same YAML does not carry them. The diff sees the annotations as part of the observed state and reports a difference.

Production discipline

1. Trust the rendered desired state, not the YAML in Git. When debugging a diff, the first question is “what did the render produce?”, not “what does the YAML say?”. helm template and kustomize build reproduce the render locally.
2. Understand what the diff engine ignores. Defaulted fields, status fields, controller annotations. A diff that looks clean may still hide drift in fields the engine ignores; an audit of those fields requires a different tool.
3. Treat out-of-band applies as ownership changes. A resource applied by hand with the same YAML as the GitOps controller’s desired state is still drift, because ownership is recorded in annotations the controller relies on.

Cross-course references

• Kubernetes for Production Sysadmins - Parts on server-side apply and field managers cover the API-level mechanics the controller relies on.
• Terraform for Production Sysadmins - Parts on state versus configuration cover the same asymmetry in a declarative-infrastructure context.
• Ansible for Production Sysadmins - Parts on configuration drift cover the analogous problem in configuration management.

Quiz