Readiness failures cascading — the probe and the rollout

A readiness failure in a rollout is the new Pods not becoming Ready. The probe and the new image are the diagnostic. The remediation is to fix the probe or rollback. The discipline is the same 11-step methodology: read the rollout status, inspect the new Pods, identify the cause, apply the fix.

The readiness phase of a rollout

A rolling update phases through three stages:

1. Create. The controller creates the new Pods.
2. Ready. The new Pods become Ready (the readiness probe passes).
3. Terminate. The old Pods are terminated.

flowchart TD
    A[Create new Pods] --> B{New Pods Ready?}
    B -->|Yes| C[Terminate old Pods]
    B -->|No| D[Stuck at Ready]
    D --> E[Readiness probe failing]

The rollout is stuck at the Ready phase if the new Pods do not become Ready. The controller is waiting for the new Pods to be Ready before terminating the old Pods.

The diagnostic

A real kubectl describe deployment for a rollout stuck at the Ready phase:

Name:                   billing
Namespace:              prod
Selector:               app=billing
Replicas:               6 desired | 4 updated | 8 total | 4 available | 4 unavailable
StrategyType:           RollingUpdate
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    True    NewReplicaSetAvailable
NewReplicaSet:   billing-7d8f (4/6 replicas created)
Events:
  Type    Reason              Age   From                   Message
  ----    ------              ----  ----                   -------
  Normal  ScalingReplicaSet   6m    deployment-controller  Scaled up replica set billing-7d8f to 4

The diagnostic is:

• 4 unavailable — 4 new Pods are not Ready.
• The new ReplicaSet has 4 replicas created but only 0 are Ready.

The Pod state:

kubectl get pods -n prod -l app=billing -o wide

The output:

NAME                       READY   STATUS    RESTARTS   AGE   NODE
billing-7d8f-abcde         0/1     Running   0          6m    node-01
billing-7d8f-def01         0/1     Running   0          6m    node-02
billing-7d8f-ghi02         1/1     Running   0          6m    node-03
billing-7d8f-jkl03         1/1     Running   0          6m    node-04
billing-7d8e-mno04         1/1     Running   0          30m   node-05
billing-7d8e-pqr05         1/1     Running   0          30m   node-06

The new Pods are Running but not Ready. The old Pods are Ready. The rollout is stuck at the Ready phase.

The probe

The probe is the diagnostic. The new Pods are not Ready because the readiness probe is failing.

kubectl describe pod billing-7d8f-abcde -n prod

The events show:

Type     Reason          Age   From               Message
----     ------          ----  ----               -------
Warning  Unhealthy       4m    kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

The 503 is the application’s view: the application is returning 503 on /healthz. The probe is doing its job — removing the Pod from the Service because the application is not ready.

The remediation

The remediation depends on the cause:

• Probe misconfiguration. The probe’s path or port is wrong. Fix the probe’s configuration.
• Application failure. The application’s /healthz is returning an error. Fix the application.
• Slow startup. The application takes longer than the probe’s failureThreshold to start. Increase the threshold or add a startup probe.

The remediation is to either roll back or fix the underlying issue.

# Option 1: Rollback the rollout
kubectl rollout undo deployment/billing -n prod

# Option 2: Fix the probe
kubectl set probe deployment/billing -n prod \
  --readiness=http-get=/health:8080

# Option 3: Fix the application
kubectl set image deployment/billing -n prod \
  billing=registry.example.com/billing:1.2.4

Common causes

• New image has a bug. The new image is broken; the application is not starting.
• New image has a slow startup. The new image takes longer to start than the probe’s threshold.
• Probe misconfiguration. The probe’s path or port is wrong.
• External dependency. The application’s /healthz checks the database, and the database is unreachable.

Production discipline

A readiness failure in a rollout is the probe doing its job. The discipline is to read the events, identify the cause, apply the remediation. The rollout either continues or is rolled back.

• Read the rollout status first. The status is the canonical answer.
• Inspect the new Pods. The Pods’ state is the cluster’s hypothesis.
• Roll back if you cannot fix. The rollback is the restore.

Quiz