Rollout stuck — the progression block

A stuck Deployment rollout is the controller waiting for the new ReplicaSet to converge. The rollout status, the rollout history, and the Pod state are the diagnostic. The discipline is to identify the progression block, fix the underlying issue, and let the rollout continue — or roll it back.

The Deployment rollout state machine

A Deployment rollout is a state machine. The state machine moves the Deployment from the previous version to the new version through a rolling update:

stateDiagram-v2
    [*] --> Progressing
    Progressing --> Available: new ReplicaSet is Ready
    Progressing --> Degraded: progress deadline exceeded
    Degraded --> Progressing: fix and re-deploy
    Available --> [*]

The state machine has two states: Progressing (the controller is working on the rollout) and Available (the rollout is complete and the Deployment is available). The controller emits the state in the status.conditions field.

The rollout status

The rollout status is the canonical diagnostic:

kubectl rollout status deployment/billing -n prod

A real rollout status for a stuck rollout:

Waiting for deployment rollout to finish: 1 out of 6 new replicas updated...

The status tells the operator:

• The Deployment has 6 desired replicas.
• Only 1 new replica is updated.
• The rollout is stuck.

The remediation is in the rollout history and the Pod state.

The rollout history

The rollout history shows the previous ReplicaSets:

kubectl rollout history deployment/billing -n prod

A real rollout history:

REVISION  CHANGE-CAUSE
1         <none>
2         <none>
3         Initial deploy
4         Update to 1.2.3

The history shows that the controller has 4 revisions. The current rollout is revision 4. The previous rollout (revision 3) is the one that worked.

kubectl rollout undo deployment/billing -n prod --to-revision=3

The remediation is to roll back to the previous revision.

The Progression block

The progression block is the reason the rollout is stuck. The block is one of:

1. Pod-level block. The new Pod is failing to start (Pending, CrashLoopBackOff, OOMKilled). See Part CXIX.
2. Deployment-level block. The Deployment’s strategy is Recreate and the old Pods are not terminating.
3. Quorum-level block. The Deployment’s maxUnavailable is at 0 and the new Pod is not Ready; the controller is waiting for the new Pod to be Ready before continuing.
4. Progress deadline block. The Deployment’s progressDeadlineSeconds has been exceeded.

flowchart TD
    A[Stuck rollout] --> B{Pod-level?}
    B -->|Yes| C[Pod failing]
    B -->|No| D{Deployment-level?}
    D -->|Yes| E[Recreate strategy stuck]
    D -->|No| F{Quorum-level?}
    F -->|Yes| G[maxUnavailable=0, new Pod not Ready]
    F -->|No| H[Progress deadline exceeded]

The diagnostic is the rollout status, the rollout history, and the Pod state.

The canonical diagnostic command

# 1. Rollout status
kubectl rollout status deployment/billing -n prod

# 2. Rollout history
kubectl rollout history deployment/billing -n prod

# 3. Pod state
kubectl get pods -n prod -l app=billing -o wide

# 4. Inspect the failing Pod
kubectl describe pod billing-7d8f-abcde -n prod

# 5. Read the events
kubectl get events -n prod --sort-by=.lastTimestamp \
  --field-selector involvedObject.name=billing-7d8f-abcde

# 6. Read the previous logs
kubectl logs -n prod billing-7d8f-abcde -c billing --previous

The command sequence is the 11-step methodology applied to the Deployment.

The remediation

The remediation depends on the cause:

• Pod-level block. Fix the Pod (see Part CXIX).
• Deployment-level block. Force the rollout to continue with kubectl rollout restart or roll back with kubectl rollout undo.
• Quorum-level block. Increase maxUnavailable or fix the new Pod’s readiness.
• Progress deadline block. Either fix the rollout or extend the deadline.

Production discipline

A stuck rollout is the controller’s hypothesis. The discipline is to read the rollout status, identify the progression block, fix the underlying issue. The rollout either continues or is rolled back; the cluster does not sit in a half-converged state.

• Fix the underlying issue. The remediation is the underlying issue, not the rollout.
• Roll back if you cannot fix. The rollback is the restore.

Quiz