What Healthy means — the workload is functioning correctly

Healthy is the second of the two orthogonal questions every GitOps controller answers on every reconciliation tick. The question is behavioural: is the workload the manifests describe actually doing what it is supposed to do? The answer is computed by probes - built-in resource checks, custom Lua assessments, and the indirect signals the controller can read from live objects.

Healthy is a workload-layer property. Synced is a manifest-layer property. They are computed from different data sources, by different code paths, and they can disagree.

How a controller computes Health

Argo CD ships a built-in health check library that knows about Deployment, StatefulSet, DaemonSet, Service, Ingress, PersistentVolumeClaim, Job, CronJob, and a long list of other resource kinds. For each kind, the health check reads the live object’s status fields and assigns one of: Healthy, Progressing, Degraded, Suspended, Missing, Unknown.

flowchart LR
    L["Live cluster objects"] -->|"status fields"| HC["Health check library"]
    HC -->|"kind + phase"| R["Result: Healthy / Progressing / Degraded"]
    P["Probes: readiness, liveness"] -->|"indirect signal"| HC
    HC -->|"aggregated"| App["Application Health"]

A Deployment with status.availableReplicas equal to spec.replicas, with all pods Ready, is Healthy. A Deployment with status.availableReplicas less than spec.replicas is Progressing. A Deployment with no Ready pods for too long is Degraded. A Job with status.conditions[].type: Complete and status.conditions[].status: "True" is Healthy. The controller does not need to know what the workload does; it needs to know the status fields that indicate the workload is functioning.

Flux follows the same model. A Kustomization reports Ready: True when the resources it manages pass the same kind of status-field checks. The semantics are the same; the vocabulary differs.

What Healthy is not

Healthy is not “the workload is doing the right thing”. Healthy is “the controller’s health check library reports the workload is in a state it recognises”. The difference matters because most real failures live in the gap between recognised-functioning and actually-functioning.

• Healthy does not mean “the workload serves correct responses”. A HTTP server can pass readiness probes and still return 500s on the endpoints that matter.
• Healthy does not mean “the workload is reachable”. A Service with endpoints that point at a healthy pod on a node with broken CNI is Healthy by every check the controller runs.
• Healthy does not mean “the workload’s dependencies are healthy”. An application whose database connection is broken can be Healthy if the application reports Ready before the first query fails.
• Healthy does not mean “the workload is correctly configured”. A misconfigured Secret that is syntactically valid produces a Healthy workload that fails at first use.

Health is what the controller can verify. Correctness is what only the workload can verify.

The Health block in the operator CLI

The Health status surfaces in the same two places as the Sync status: the CLI and the Application CR.

argocd app get "$APP" --show-operation

The output includes a Health Status line - Healthy, Progressing, Degraded, Suspended, or Missing - and per-resource Health blocks when the application contains multiple resources. The aggregated Health is the worst Health across all resources, with one exception: a Progressing resource does not drag a Healthy sibling down.

The cluster surface is the same Application CR. The Health status is in status.health.status. Flux’s equivalent is the Ready condition on the Kustomization.

kubectl get application "$APP" -n argocd -o yaml

When the CLI shows Healthy and the CR shows Degraded, the CR is authoritative. The CLI is a projection; the CR is the controller’s own bookkeeping. When the CR’s status.health is empty, the controller has not yet computed a Health for the application - it is in the middle of its first reconciliation.

Production discipline

The production framing of Health has three rules that prevent operators from treating Healthy as a guarantee:

1. Healthy is necessary, not sufficient. A red Health is an incident. A green Health is a precondition, not a conclusion.
2. The health library is a library, not a custom probe. For workloads that need domain-specific health checks - database connectivity, downstream API reachability, queue depth - write a Kubernetes probe or a custom resource. Do not expect the GitOps controller to know.
3. Health lag is real. The controller reports Health from the last observation. A workload that just started failing can show Healthy for seconds to minutes while the next reconciliation tick runs.

Cross-course references

• Linux for Production Sysadmins - Part XXII (ProcessSupervision) covers the difference between “the process is running” and “the process is serving requests”, which is the same recognised-versus-actually distinction.
• Ansible for Production Sysadmins - Part XLV (Verification) covers the difference between “the task reports changed” and “the system is in the desired state”.
• Kubernetes for Production Sysadmins - Parts XV-XVII (Liveness) cover probe design, which is the workload-layer equivalent of the GitOps controller’s health library.

Quiz