Failure modes in reconciliation — what stops the loop, and how to detect it

The reconciliation loop fails in distinct ways. Each failure maps to one of the four phases - read, observe, diff, apply - and each phase has a distinct signal and a distinct remediation. The production discipline is to diagnose by phase rather than restart the controller and hope.

flowchart LR
    R["Read"] -->|"fails"| RF["Credential or source error"]
    O["Observe"] -->|"fails"| OF["RBAC or API-server error"]
    D["Diff"] -->|"fails"| DF["Render or normalisation error"]
    A["Apply"] -->|"fails"| AF["Validation or admission error"]
    RF --> RMD["Inspect creds and source"]
    OF --> RDO["Inspect RBAC and API server"]
    DF --> DFR["Re-render and compare"]
    AF --> APF["Inspect resource and admission"]

The diagram shows the four phases and the failure mode each one produces. The right column shows the remediation category. A generic “restart the controller” is not on the list; it is sometimes necessary, but it is not a diagnosis.

Read failures

A read failure is the controller’s inability to pull the desired state from its source. The source could be Git, a Helm repository, an OCI registry, or an S3 bucket. Symptoms:

• The status subresource reports Last Attempted Revision: <none> or Last Applied Revision: <none>.
• The controller logs show errors mentioning git fetch, authentication, repository not found, permission denied, or rate limit.
• The Application is in Unknown or Progressing state for an extended period.

Common causes:

• Expired credentials. The deploy key or token used by the controller has rotated or expired. The source controller fetches successfully but receives a 401.
• Network reachability. The cluster cannot reach the Git server. This is common in air-gapped or restricted-egress clusters.
• Repository moved or deleted. The URL no longer resolves.
• Rate limits. The Git provider’s API rate limit has been exhausted.

Remediation: rotate the credentials, fix the network policy, update the URL, or wait for the rate limit to reset. The controller does not recover on its own in any of these cases - the operator must intervene.

Observe failures

An observe failure is the controller’s inability to query the cluster API for the live state. Symptoms:

• The controller logs show errors mentioning forbidden, unauthorized, connection refused, timeout, or connection reset.
• The status subresource shows a recent revision but the inventory is stale or empty.
• The Application is in Progressing or Unknown with no progress on the diff.

Common causes:

• RBAC denials. The controller’s ServiceAccount does not have get/list/watch on the resource kinds the application owns. This is the most common cause.
• API-server reachability. The controller cannot reach the API server; this is rare in-cluster but common if the controller is on a different cluster.
• CRDs missing. The controller is observing a custom resource type whose CRD has been deleted or not installed. The API server returns a “no matches for kind” error.

Remediation: fix the RoleBinding, restore the API-server connectivity, or re-install the CRD. A controller that has RBAC denials on one resource kind will still reconcile other resources; the failure is per-resource, not cluster-wide.

Diff failures

A diff failure is the controller’s inability to compute a structured comparison. This is rarer than read or observe failures because the diff is local computation. Symptoms:

• The controller logs show errors mentioning unmarshaling, parsing, kustomize, helm template, or schema.
• The status subresource shows the source revision loaded correctly but no inventory.
• The Application is in Progressing with no apply attempt.

Common causes:

• Render errors. A Helm chart or Kustomize overlay produces invalid YAML. The render step fails before the diff can run.
• Schema errors. A custom resource manifest does not match the CRD’s schema. The Kubernetes API rejects the apply during validation, but the diff engine can fail earlier.
• Unsupported fields. A field type the controller’s diff engine does not know how to compare (rare in modern controllers).

Remediation: re-render the manifests locally with the same toolchain version, fix the schema violation, or update the controller. A render failure points to a manifest problem, not a controller problem.

Apply failures

An apply failure is the controller’s ability to write to the cluster but its inability to land the changes. Symptoms:

• The status subresource shows Last Attempted Revision: <new> and Last Applied Revision: <old> - the controller tried to apply the new revision but fell back.
• The controller logs show errors mentioning validation, admission webhook, invalid, or conflict.
• The Application is in OutOfSync even after multiple ticks.

Common causes:

• Validation. A required field is missing or has an invalid value. The API server rejects the apply.
• Admission webhooks. A validating or mutating webhook rejects the apply. Common with policy controllers (OPA, Kyverno, Datadog).
• Resource conflicts. Another actor modified the resource between the controller’s observe and apply phases; the controller retries but keeps losing the conflict.
• Quota or limit-range exhaustion. The cluster has reached a quota on the resource type.

Remediation: fix the manifest, update the admission policy, resolve the resource conflict (the controller usually retries on its own), or request a quota increase.

Detecting a stuck loop

A stuck loop is one where the controller is alive but not making progress. The signals:

• No successful reconcile in N intervals. The metric “last successful reconcile timestamp” should be recent. If it drifts older than the interval, the loop is stuck.
• Status subresource frozen on one revision. The Last Applied Revision is unchanged across many ticks.
• Health endpoint unresponsive. The controller’s /healthz endpoint times out.
• Controller pod restarts. The pod’s restart count is climbing. The controller is crashing during the tick.

A stuck loop is distinct from a broken loop. A broken loop reports errors; a stuck loop reports silence. The remediation differs: a broken loop wants a phase diagnosis; a stuck loop wants a controller restart or replacement.

Operational levers when the loop is broken

Two levers are worth knowing:

argocd app sync "$APP_NAME" --prune --self-heal

Argo CD CLI for forcing a sync with pruning and self-healing. This is the right tool when the controller’s view of the diff is stale but the controller is otherwise healthy.

flux reconcile kustomization "$KS_NAME"

Flux CLI for forcing a fresh tick on a Kustomization. This is the right tool when the kustomize-controller has not picked up the source controller’s latest artifact.

Neither command fixes the underlying problem if the loop is genuinely broken; they only force a tick. The remediation for a genuinely broken loop is to read the controller logs, identify the failing phase, and fix the root cause.

Production discipline

1. Diagnose by phase, not by symptom. A controller that reports “OutOfSync” could be failing on read, observe, diff, or apply. The remediation is different per phase. Phase diagnosis is faster than generic restart loops.
2. Watch the four metrics. Read success rate, observe success rate, diff success rate, apply success rate. Each one is a separate metric in modern controllers. A spike in any one of them is the early signal of a phase-specific problem.
3. Treat a stuck loop as an incident. A controller that is silent for more than two intervals is not “fine”. It is either failing in a way that does not surface errors, or it is dead. Either case warrants investigation.

Cross-course references

• Kubernetes for Production Sysadmins - Parts on RBAC debugging cover the most common observe-phase failure.
• Terraform for Production Sysadmins - Parts on state lock contention cover the apply-phase analogue.
• Ansible for Production Sysadmins - Parts on playbook failure modes cover the equivalent in configuration management.

Quiz