Bad images and rollout deadlock — the recovery path

A bad image is a Deployment rolling out a broken application. The recovery is either to roll back or to roll forward. A deadlock is a Deployment that cannot roll forward or back. The discipline is to identify the recovery path before the deadlock.

The bad image

A bad image is a Deployment rolling out a new version that fails to start, fails the readiness probe, or otherwise cannot serve traffic. The common causes:

• Missing library. The image is missing a dependency.
• Bad configuration. The image’s config is wrong.
• Bug. The image has a bug.

The rollout is stuck at the new ReplicaSet. The old ReplicaSet is still running, but the new ReplicaSet is failing.

The recovery path

The recovery path is:

1. Roll back. Revert the Deployment to the previous version. The Deployment continues to serve traffic on the old version.
2. Roll forward. Fix the image and re-deploy. The Deployment converges to the new version.
3. Pause and investigate. Pause the rollout with kubectl rollout pause, investigate, then resume or roll back.

# Roll back
kubectl rollout undo deployment/billing -n prod

# Roll forward (after fixing the image)
kubectl set image deployment/billing -n prod \
  billing=registry.example.com/billing:1.2.4

# Pause
kubectl rollout pause deployment/billing -n prod

# Resume
kubectl rollout resume deployment/billing -n prod

The recovery path is mechanical. The discipline is to choose the path before the deadlock.

The rollout deadlock

A rollout deadlock is a Deployment that cannot roll forward or back. The causes:

• The previous version is also broken. The Deployment has only one version; rolling back would not help.
• The new image is required for the cluster. The cluster cannot run without the new image (e.g., a security patch).
• The configuration is corrupt. The Deployment’s configuration is corrupt; the controller cannot converge.

flowchart TD
    A[Rollout stuck] --> B{Roll back?}
    B -->|Yes| C[Continue]
    B -->|No| D{Roll forward?}
    D -->|Yes| C
    D -->|No| E[Deadlock]
    E --> F[Investigate]
    F --> G[Manual intervention]

The deadlock is the failure mode of the recovery path. The discipline is to identify the deadlock before it happens.

The diagnostic

The diagnostic for a bad image is the standard 11-step methodology:

# 1. Rollout status
kubectl rollout status deployment/billing -n prod

# 2. Rollout history
kubectl rollout history deployment/billing -n prod

# 3. Pod state
kubectl get pods -n prod -l app=billing -o wide

# 4. Inspect the failing Pod
kubectl describe pod billing-7d8f-abcde -n prod

# 5. Read the previous logs
kubectl logs -n prod billing-7d8f-abcde -c billing --previous

# 6. Read the events
kubectl get events -n prod --sort-by=.lastTimestamp \
  --field-selector involvedObject.name=billing-7d8f-abcde

The diagnostic is the same as the previous lessons. The 11-step methodology is the discipline.

The manual intervention

The manual intervention is the recovery path when the automatic recovery is not possible. The manual intervention is:

1. Identify the cause. The events and the logs tell the operator.
2. Fix the underlying issue. The image’s bug, the configuration, the database.
3. Re-deploy. kubectl set image or kubectl apply.
4. Validate. The rollout completes.

The manual intervention is the on-call engineer’s job. The discipline is to follow the 11-step methodology, not to apply a known fix.

Production discipline

A bad image is a deployment that is not ready to be production. The discipline is to test in staging first, then roll out to production. The recovery path is either roll back or roll forward. The deadlock is the failure mode of the recovery path.

• Choose the recovery path before the deadlock. The rollback or the roll forward is the path.
• Mechanical recovery first. kubectl rollout undo or kubectl set image is the primary remediation.
• Manual intervention as a last resort. The manual fix is the on-call engineer’s job; the discipline is the 11-step methodology.

Quiz