Rollout status, pause, and resume — orchestrating long changes

Rolling out a change is not a fire-and-forget operation; production rollouts need monitoring, staging, and the ability to abort. This lesson covers the kubectl rollout verbs, the pause/resume pattern, and the production discipline around coordinating rollouts.

kubectl rollout status

kubectl rollout status deployment/web

Output (live):

deployment "web" successfully rolled out

Output (in progress):

Waiting for deployment "web" rollout to finish: 2 out of 5 new replicas updated...
Waiting for deployment "web" rollout to finish: 3 out of 5 new replicas updated...
Waiting for deployment "web" rollout to finish: 4 out of 5 new replicas updated...
Waiting for deployment "web" rollout to finish: 5 out of 5 new replicas updated...
Waiting for deployment "web" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "web" rollout to finish: 2 old replicas are pending termination...
deployment "web" successfully rolled out

Output (failed):

error: deployment "web" exceeded its progress deadline

The status command polls the Deployment’s status until it reaches a terminal state (success or failure).

# With timeout
kubectl rollout status deployment/web --timeout=60s
# After 60s, if not done, returns error

# Watch status changes only
kubectl rollout status deployment/web -w

kubectl rollout pause and resume

# Pause
kubectl rollout pause deployment/web

# Make changes (don't trigger rollouts)
kubectl set image deployment/web web=nginx:1.27.3
kubectl set env deployment/web FEATURE_FLAG=enabled

# Resume (Deployment rolls forward with accumulated changes)
kubectl rollout resume deployment/web

When a Deployment is paused, the controller does not create new ReplicaSets. Edits to the Deployment’s spec are saved but do not trigger rollouts. When rollout resume is called, the Deployment controller sees the cumulative diff and rolls forward.

stateDiagram-v2
    [*] --> Active
    Active --> Paused: rollout pause
    Paused --> Paused: edit (no rollout)
    Paused --> Active: rollout resume
    Active --> Active: edit (triggers rollout)

Use cases:

• Staged changes: make multiple related changes; pause; verify in staging; resume.
• Coordinated rollouts: pause one Deployment while updating another; resume both.
• Rollout planning: pause to plan; resume when ready.

Coordinated rollouts

For multi-service changes, the pause/resume pattern enables coordination:

# Pause both Deployments
kubectl rollout pause deployment/web
kubectl rollout pause deployment/api

# Make changes
kubectl set image deployment/web web=nginx:1.27.3
kubectl set image deployment/api api=v2

# Verify changes
kubectl get deployment/web -o yaml | grep image
kubectl get deployment/api -o yaml | grep image

# Resume when ready
kubectl rollout resume deployment/web
kubectl rollout resume deployment/api

# Monitor
kubectl rollout status deployment/web
kubectl rollout status deployment/api

This pattern is the cluster-level equivalent of an Ansible rolling update: pause both, change both, resume both, monitor both.

Detecting stuck rollouts

A rollout that won’t progress:

# Status with timeout
kubectl rollout status deployment/web --timeout=120s
# error: deployment "web" exceeded its progress deadline

The progressDeadlineSeconds (default 600s) is the maximum time the Deployment waits for progress. If exceeded, the Deployment is marked failed; new ReplicaSet may be stuck at non-zero replicas.

# Check Deployment conditions
kubectl get deployment/web -o jsonpath='{.status.conditions}' | jq

Output:

[
  {
    "type": "Progressing",
    "status": "False",
    "reason": "ProgressDeadlineExceeded"
  },
  {
    "type": "Available",
    "status": "True"
  }
]

Progressing=False with Available=True means the rollout failed but the existing Pods are still serving traffic (the old ReplicaSet is still running).

Production discipline

Pre-flight checks before a rollout:

# Verify cluster has surge capacity
kubectl describe nodes | grep -A 5 "Allocated resources"
# Check that the total request utilization is < 100% - maxSurge

# Verify the new image exists
docker manifest inspect nginx:1.27.3

# Verify PDB allows the rollout
kubectl get pdb -n team-a-prod

# Verify no recent failures
kubectl get events --field-selector reason=Failed -n team-a-prod | tail

Monitor the rollout:

# In one terminal
kubectl rollout status deployment/web

# In another terminal
kubectl get pods -l app=web -w

Rollback procedure if stuck:

# Wait for progress deadline (or check manually)
kubectl rollout status deployment/web --timeout=600s

# If stuck, investigate
kubectl describe deployment/web
kubectl get pods -l app=web -o wide

# Rollback
kubectl rollout undo deployment/web

# Verify
kubectl rollout status deployment/web

The discipline:

• Use kubectl rollout status in CI/CD pipelines. It’s exit-code aware; gate production rollouts on success.
• Use pause/resume for staged rollouts. Make all related changes, pause, verify, resume.
• Monitor ProgressDeadlineExceeded. Failed rollouts don’t stop traffic but indicate broken changes.

Coordination with CI/CD

In a CI/CD pipeline, the rollout is part of the deployment step:

- name: Deploy to staging
  run: |
    kubectl apply -f deployment.yaml
    kubectl rollout status deployment/web --timeout=300s

- name: Smoke test
  run: ./scripts/smoke-test.sh

- name: Deploy to production
  if: success()
  run: |
    kubectl apply -f deployment.yaml
    kubectl rollout status deployment/web --timeout=600s

The CI/CD gates the production rollout on the staging rollout succeeding and the smoke test passing. If the production rollout fails, the CI/CD raises an alert; the operator can then investigate and undo.

Cross-course references

• The Ansible course part XXXV-Ansible-Scripting covers rolling restarts and pause/resume; kubectl rollout is the cluster-level equivalent.
• The Docker course part XXXIV-Docker-Production covers blue-green deploys; rolling updates are a simpler alternative.
• The Observability course part LXXXV-Kubernetes-Observability covers rollout metrics; kube_deployment_status_* metrics drive rollout alerts.

Quiz