Progressing and Degraded states — the intermediate states

Progressing and Degraded are the two intermediate workload states. Both sit between Healthy and broken, but they demand different operator responses. Progressing says “the workload is moving toward Healthy, watch and wait”. Degraded says “the workload has stopped moving toward Healthy, intervene now”. The difference is the difference between in-flight and stuck.

The boundary between Progressing and Degraded is the progress deadline. A Deployment rolling out new pods is Progressing. A Deployment whose rollout has not completed within the deadline - typically ten minutes by default - is Degraded. The controller flips the state based on time, not on the underlying cause.

What Progressing means

A workload is Progressing when it is on a path toward Healthy and the controller’s health library recognises the path. The path is described by status fields that indicate change: status.observedGeneration is newer than metadata.generation, status.availableReplicas is less than spec.replicas, a Job’s status.active > 0, a StatefulSet’s status.updatedReplicas < spec.replicas.

Progressing is a healthy state during change. A rolling update that takes two minutes to swap pods is Progressing for two minutes, then Healthy. A Job that runs for an hour is Progressing for an hour, then Healthy or Degraded depending on the result. Progressing is the controller saying “I see the workload is changing in the direction you want; I will let it run”.

stateDiagram-v2
    [*] --> Healthy
    Healthy --> Progressing: "spec change"
    Progressing --> Healthy: "rollout complete"
    Progressing --> Degraded: "deadline exceeded"
    Progressing --> Progressing: "still rolling"
    Degraded --> Progressing: "retry / fix"
    Degraded --> Healthy: "manual intervention"

The trap with Progressing is treating it as alarming. Most Progressing states are routine - a horizontal pod autoscaler just added a pod, a Deployment rolled a new image, a CronJob triggered. The on-call engineer who pages on every Progressing state will be paged constantly and will eventually learn to ignore the pages, which means missing the Progressing state that is actually stuck.

What Degraded means

A workload is Degraded when it has stopped moving toward Healthy and the controller’s health library recognises the stuck state. The most common Degraded signal is ProgressDeadlineExceeded on a Deployment: the rollout did not complete within the configured deadline and Kubernetes marked the Deployment as failed.

Degraded is a stable signal. Once a workload is Degraded, it stays Degraded until something intervenes - a new spec, a manual rollout restart, a fix to the cause. Degraded is the controller saying “I see the workload is stuck; the system is not on a path to Healthy without help”.

The operator response to Degraded is investigation, not waiting. The cluster will not fix itself. The cause must be found: a failing readiness probe, a bad image, a missing ConfigMap, a misconfigured Service. The investigation can start from the controller’s health detail and the cluster’s events, but it ends at the workload.

Reading Progressing and Degraded in the operator CLI

The aggregated Health status shows the worst state across all resources, but the per-resource detail shows which resource is Progressing or Degraded. This matters: a Deployment can be Degraded while a sibling ConfigMap is Healthy. The aggregated Health reports Degraded; the detail tells the operator which resource to look at.

argocd app get "$APP" --show-operation

The output lists each resource with its kind, name, Sync status, and Health status. A Degraded Deployment in a sea of Healthy siblings is visible in the per-resource table. The aggregated Health at the top is Degraded because Degraded is the worst state, but the detail prevents the operator from investigating the wrong resource.

kubectl get application "$APP" -n argocd -o yaml

The same detail is in the CR’s status.health.status field. Flux’s equivalent is the per-resource status.conditions[].type block on the Kustomization.

When the per-resource detail shows Progressing and the aggregated Health shows Progressing, the system is in the middle of a rollout. When the per-resource detail shows Degraded and the aggregated Health shows Degraded, the system is stuck. When the detail shows Progressing and the aggregated shows Degraded, the system has both a stuck resource and an in-flight resource - rare but possible during partial rollouts.

Production discipline

The production framing of Progressing and Degraded has three rules:

1. Progressing is watch-and-wait; Degraded is intervene-now. A page on Progressing for less than the progress deadline is usually a false alarm. A page on Degraded is always actionable.
2. Tune the progress deadline to the workload. Stateless services can have a short deadline - one minute. Stateful services with migrations need longer. The deadline is the upper bound on “the system is allowed to be in flight”.
3. Investigate Degraded at the workload layer. Degraded means the cluster state is not converging on the desired state. GitOps cannot fix it; the cluster and the workload are the source.

Cross-course references

• Linux for Production Sysadmins - Part XXII (ProcessSupervision) covers the systemd activating state, which is the process-level equivalent of Progressing, and failed, which is the equivalent of Degraded.
• Kubernetes for Production Sysadmins - Part XVII (Probes) covers probe timeouts, which are the per-pod equivalent of the progress deadline.
• Observability for Production Sysadmins - Part VII (Alerting) covers the difference between “rate-of-change” alerts (Progressing) and “stuck” alerts (Degraded).

Quiz