Controller unavailable — the GitOps control plane is down

The first two failure modes are visible at the application level: the application shows Degraded or OutOfSync, the controller log carries an error message, and the operator has a clear starting point. The third failure mode is invisible at the application level: when the controller itself is not running, every application reports the last status it observed before the controller stopped - almost always Synced and Healthy. The cluster drifts in silence, and the operator’s only signal is the absence of activity.

The silent failure

A controller that is not running leaves no application-level symptom. argocd app get returns Synced and Healthy because the API server is serving the last cached state. The applications are not actually healthy - the controller is not checking them - but the API surface does not distinguish “last checked at 14:00 and was healthy” from “checked continuously”.

flowchart LR
    A[Application status] --> B{Controller running?}
    B -- yes --> C[Live reconciliation]
    B -- no --> D["Cached state from last check"]
    D --> E["Looks healthy - lies"]
    C --> F["Reports actual state"]

The diagnostic move is to read the controller, not the applications:

kubectl -n argocd get pods

A missing or CrashLoopBackOff controller is the obvious case. A controller that is Running but not progressing is the harder case, and it requires reading the controller log:

kubectl -n argocd logs statefulset/argocd-application-controller

A controller that is hung on a watch or stuck in a backoff will show no recent entries - the absence of log lines is the symptom.

The three shapes

There are three distinct shapes of controller failure:

• Pod missing or CrashLoopBackOff. Visible in kubectl get pods. Read the pod’s previous logs (--previous) to find the crash reason. Common causes: OOMKill, a misconfigured RBAC role, a missing CRD.
• Pod Running but no progress. Visible only by the absence of log entries or by the absence of recent Sync events on Application resources. Common causes: a deadlock against the API server, a hung Git fetch.
• HA shard mismatch. In high-availability mode the controller shards managed clusters across its StatefulSet replicas, driven by the ARGOCD_CONTROLLER_REPLICAS environment variable. When spec.replicas and the variable disagree, some clusters are assigned to a shard that does not exist and are silently never reconciled. The diagnostic is to compare the StatefulSet’s spec.replicas with the ARGOCD_CONTROLLER_REPLICAS value on the StatefulSet.

For Flux, the equivalent shapes are spread across the source controller, kustomize controller, and Helm controller. The diagnostic is kubectl get kustomizations -A and kubectl get helmreleases -A plus the relevant controller logs.

Why this failure is the audit failure

Every other failure mode produces a visible signal that an audit can reconstruct. A controller downtime produces no signal at all. The audit trail shows a gap between the last sync before the controller stopped and the first sync after it restarted, and the operator reading the audit six months later cannot tell whether the cluster was unchanged during the gap or whether it drifted silently.

The remediation is alerting that catches the gap while it is still recoverable.

Production discipline

The minimum alert set for controller downtime:

• Controller pod not running. Alert when kube_statefulset_status_replicas_ready{statefulset="argocd-application-controller"} == 0 for more than two reconciliation intervals.
• Controller running but no progress. Alert when no new Application sync event has been observed for more than three reconciliation intervals. The Prometheus query is increase(argocd_app_sync_total[10m]) == 0.
• HA shard mismatch. Alert when a shard shows no reconciliation activity. The Prometheus query is increase(argocd_app_reconcile_count[10m]) == 0 per controller replica - a shard with no reconcile activity means clusters assigned to a replica that does not exist.

These three alerts cover the three shapes. They must fire within one reconciliation interval of the controller stopping.

Cross-course references

• Linux for Production Sysadmins - Parts XII (RepoSecurity) and XXXIV (ConfigMgmt) cover monitoring the configuration management daemon itself, with the same silent-failure shape.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers monitoring the Ansible controller process with the same “alert on absence, not just on error” rule.

Quiz