Unhealthy deployment — synced but broken

The fifth failure mode exposes the boundary of what the GitOps controller actually checks. A controller can report Synced and still report Degraded, and a controller can report Synced and Healthy while the application inside the cluster is failing. The reconciliation loop converges the desired state of Kubernetes resources; the health of the application running inside those resources is a separate question, and the controller only answers it if the resource definitions include probes or health checks.

The two dimensions of status

Argo CD and Flux both track two orthogonal dimensions: sync status (does the cluster match the desired state?) and health status (is the application inside the cluster healthy?). A resource can be Synced and Degraded (the controller applied the manifest but the resource itself is failing) or Synced and Healthy (applied and running).

argocd app get guestbook

The output shows both. A broken application looks like:

Sync Status:      Synced
Health Status:    Degraded

with the Conditions block carrying messages like:

Deployment.apps/guestbook: observed generation does not match spec generation
Service: waiting for endpoint

flowchart LR
    A[Desired state in Git] --> B[Controller apply]
    B --> C[Resource in cluster]
    C --> D{Resource healthy?}
    D -- yes --> E["Synced + Healthy"]
    D -- no --> F["Synced + Degraded"]
    F --> G[Per-resource health message]

The gap between Kubernetes reconciliation and application health

The Kubernetes Deployment controller creates a Deployment with the configured number of replicas and updates it when the manifest changes. It will not detect that the application inside the Pod is failing its health check, because that check is run by the kubelet against the probe configured on the container, not by the Deployment controller.

The Argo CD health check reads the Kubernetes resource status fields. For a Deployment, it reads status.conditions and the rollout status. A Deployment that is Available: False because its Pods are CrashLoopBackOff will surface in Argo CD as Degraded. A Deployment whose Pods are running but whose readiness probe is failing will surface as Progressing and then Degraded if the probe continues to fail.

The operator’s next read is the resource’s own status:

kubectl describe deploy guestbook

which will show Available: False (UnavailableReplicas).

The rollout discipline

The reason an unhealthy deployment reaches the cluster is that the manifest change did not wait for the rollout to complete:

• Readiness probes configured. Every Deployment should have a readiness probe that actually checks application health.
• Argo CD sync options. Automated with self-heal=true are the defaults that let the controller react to drift.
• CI that runs kubectl rollout status against staging. A manifest that validates but fails to roll out is caught at the CI gate.

Flux’s equivalent

Flux reports health through Kustomization and HelmRelease status conditions. A Kustomization that applies successfully but whose child resources are unhealthy surfaces as Ready=False with reason HealthCheckFailed. For Helm, the equivalent is the HelmRelease resource’s status.conditions.

Production discipline

• Alert on Health Status: Degraded, not on Sync Status: OutOfSync. Degraded is the signal that the application is broken; OutOfSync is the signal that the manifest has drifted.
• Probe configuration is part of the manifest, not an afterthought. A Deployment without a readiness probe is a Deployment whose health cannot be inferred by the controller.
• Wait for the rollout, do not sync the next change. A sync cadence faster than the rollout cadence produces a stack of half-applied changes.

Cross-course references

• Linux for Production Sysadmins - Parts XII (RepoSecurity) and XXXIV (ConfigMgmt) cover the equivalent gap between service-running and service-healthy in systemd unit files.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers the gap between playbook-applied and application-healthy, which Ansible’s wait_for and uri modules exist to close.

Quiz