medium riskservice affecting~30 min

Runbook: Perform a Deployment Rollout

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the change ticket is open, peer-reviewed, and the change window current
· Confirm a current etcd snapshot exists from the last 24 hours: ls -lh /var/backups/etcd/
· Confirm cluster capacity can absorb replicas + maxSurge extra Pods: kubectl describe node | grep -E "Allocatable|Allocated resources"
· Confirm the new image digest is reachable from the cluster: crictl pull registry.internal/app@sha256:<digest> on a node
· Confirm the PDB allows the rollout to proceed: kubectl get pdb -n <ns> reports no Disallowed pods
· Confirm the previous revision is known and recorded: kubectl rollout history deployment/<name> -n <ns>
· Confirm there are no Warning events on the current Deployment: kubectl get events -n <ns> --field-selector involvedObject.name=<name>

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Render the manifest change in Git and capture the diff: kubectl diff -f deployment.yaml
2Pre-pull the image onto every node that will host new Pods: crictl pull registry.internal/app@sha256:<digest>
3Apply the manifest change: kubectl apply -f deployment.yaml --record
4Watch the rollout: kubectl rollout status deployment/<name> -n <ns> --timeout=10m
5Inspect the new ReplicaSet: kubectl get rs -n <ns> -l app=<name> and kubectl describe rs <rs>
6Confirm new Pods become Ready before old Pods are removed: kubectl get pod -n <ns> -l app=<name>
7If the rollout stalls, pause it: kubectl rollout pause deployment/<name> -n <ns> and inspect before deciding to resume
8If resuming, kubectl rollout resume deployment/<name> -n <ns> and continue watching
9When complete, confirm the rollout revision: kubectl rollout history deployment/<name> -n <ns>
10Validate with the post-rollout checks: requests, error rate, latency in the dashboard for at least 10 minutes

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl rollout status deployment/<name> -n <ns> reports successfully rolled out
✓kubectl get deploy/<name> -n <ns> shows the new RollingUpdateStrategy revision and Available replicas equal to desired
✓kubectl describe deploy/<name> -n <ns> reports Progressing=True, Available=True, no ProgressDeadlineExceeded
✓kubectl get rs -n <ns> -l app=<name> shows the new ReplicaSet with DESIRED == CURRENT == READY and the old ReplicaSet scaled to 0
✓kubectl get pods -n <ns> -l app=<name> shows every Pod Running and Ready with the new image
✓Dashboard p95 latency, error rate and request volume match the pre-rollout baseline (within SLO)
✓kubectl get events -n <ns> --sort-by=.lastTimestamp | grep -i warning | tail reports no new warnings since the rollout

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Pause the rollout first if it is still progressing: kubectl rollout pause deployment/<name> -n <ns>
↶Roll back to the previous revision: kubectl rollout undo deployment/<name> -n <ns> (defaults to the previous revision)
↶Roll back to a specific revision: kubectl rollout undo deployment/<name> -n <ns> --to-revision=<n>
↶Verify the rollback completes: kubectl rollout status deployment/<name> -n <ns>
↶If the new image is fundamentally broken (e.g. crash on start), kubectl rollout undo reuses the cached image on nodes; no re-pull is required
↶Record the failing revision so it is not re-applied: kubectl annotate deployment/<name> -n <ns> kubernetes.io/change-cause=""revert r<bad-rev> -- <reason>"
↶If the rollback also fails, treat as a Deployment-broken incident: drain the Deployment from the Service (kubectl scale deploy/<name> --replicas=0) and re-apply a known-good manifest

6 · Escalation

When the runbook isn't enough, contact:

· Rollout stalls at the same step for more than 5 minutes: do not wait for progressDeadlineSeconds; investigate the new ReplicaSet first
· New Pods reach Running but never Ready: readiness probe misconfiguration in the new image; pause and roll back
· Old Pods are removed before new Pods are Ready with maxUnavailable: 0: surge exceeded node capacity; do not roll forward, scale down the old ReplicaSet manually only with explicit approval
· PDB reports Disallowed pods during the rollout: the rollout itself is violating the PDB; this is a manifest bug — fix the surge settings or the PDB, do not bypass either
· Rollout succeeds but the application errors: rollback regardless of Ready; the manifest is not the application

A rolling update is not a single operation. The Deployment controller runs a reconcile loop that walks the rollout in steps bounded by maxSurge and maxUnavailable, gated by readiness. The runbook operates the loop, not the result.

1. What the controller is doing

flowchart LR
    A["old RS: N/N ready"] -->|create maxSurge new| B["old: N/N<br/>new: 0/S"]
    B -->|readiness passes| C["old: N/N<br/>new: S/S ready"]
    C -->|scale down old| D["old: N-S/N<br/>new: S/S"]
    D -->|repeat| E["old: 0/0<br/>new: N/N ready"]

Every step depends on the readiness probe passing on the new ReplicaSet. Without that, the controller throttles, and after progressDeadlineSeconds the Deployment is marked ProgressDeadlineExceeded.

2. Inspect before changing

Read-only / SafeInspect before changing


kubectl get deploy/<name> -n <ns> -o yaml | grep -E 'replicas|strategy|maxSurge|maxUnavailable|minReadySeconds|progressDeadlineSeconds'
kubectl get rs -n <ns> -l app=<name> -o wide
kubectl rollout history deploy/<name> -n <ns>

# PDB context - the rollout must not violate it
kubectl get pdb -n <ns> -o yaml | head -40

# Available capacity for the surge
kubectl describe nodes | grep -E 'Allocatable|Allocated resources' | head -40

3. Apply and watch

Read-only / SafeApply and watch


kubectl diff -f deployment.yaml | tee /tmp/rollout-diff.yaml

# Apply with change-cause recorded in rollout history
kubectl apply -f deployment.yaml --record
kubectl annotate deploy/<name> -n <ns> kubernetes.io/change-cause="image bump to <digest>" --overwrite

# Watch the rollout, do not background it
kubectl rollout status deploy/<name> -n <ns> --timeout=10m

4. Inspect the new ReplicaSet

Read-only / SafeInspect the new ReplicaSet


kubectl describe "$NEW_RS" -n <ns> | sed -n '/Events:/,$p'
kubectl get pods -n <ns> -l app=<name> -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,STATUS:.status.phase,RESTARTS:.status.containerStatuses[0].restartCount

# If a Pod is not Ready, drill in
kubectl logs -n <ns> -l app=<name> --tail=200 --previous 2>/dev/null | tail -60

5. Pause / resume for staged validation

Read-only / SafePause / resume for staged validation


kubectl rollout pause deploy/<name> -n <ns>

# Inspect at leisure
kubectl get rs -n <ns> -l app=<name>
kubectl get pods -n <ns> -l app=<name>
curl -fsS http://<service>.prod.svc:8080/version

# Resume when validation passes
kubectl rollout resume deploy/<name> -n <ns>
kubectl rollout status deploy/<name> -n <ns> --timeout=10m

6. Detect a stuck rollout

A stuck rollout looks like this:

deployment "web" successfully rolled out   # FALSE - never printed
kubectl get deploy web -n prod
# NAME  READY  UP-TO-DATE  AVAILABLE
# web   4/6    3           3

Three things to check, in order:

Read-only / SafeDetect a stuck rollout


# 1. Are the new Pods Ready?
kubectl get pods -n <ns> -l app=<name> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'

# 2. What is the new RS reporting?
kubectl describe rs -n <ns> <new-rs> | sed -n '/Conditions:/,/Events:/p'

# 3. Is PDB blocking?
kubectl get pdb -n <ns> -o jsonpath='{.items[*].status}' | jq

If the new Pods are not Ready, the readiness probe is wrong (or the application is). If they are Ready but the rollout is stuck, capacity or the PDB is wrong. The runbook does not advance past this point without resolving the cause.

7. Roll back when needed

Read-only / SafeRoll back when needed


kubectl rollout pause deploy/<name> -n <ns> || true

# Roll back to the previous revision
kubectl rollout undo deploy/<name> -n <ns>

# Or to a specific revision
kubectl rollout undo deploy/<name> -n <ns> --to-revision=<n>

# Watch
kubectl rollout status deploy/<name> -n <ns> --timeout=10m
kubectl get rs -n <ns> -l app=<name>

A rollback uses the cached image on every node that previously pulled it. If the cache was evicted (or the node is new), the rollback will stall at the image-pull step. Plan for image pre-pull before any canary on a brand-new fleet.

Common pitfalls

Symptom	Cause	Action
`progressDeadlineSeconds` exceeded	Rollout genuinely stuck	Inspect new RS and Pods; do not retry
New Pods Ready but Service still serves old Pods	EndpointSlice update lag	`kubectl get endpointslices -n <ns>`; usually transient
Rollout succeeds but `kubectl rollout history` shows nothing	`--record` was not used and annotations stripped	Re-record the change-cause on the Deployment
PDB reports `Disallowed pods` during rollout	surge settings exceed the PDB budget	Lower `maxSurge` or scale the PDB; never bypass
Rollout “succeeds” with 0/N ready	The ReplicaSet exists but no Pod can be scheduled	Check node capacity and Pod spec (resources, affinity, taints)

A Deployment rollout is a deliberately safe operation when its parameters match the workload. When something is wrong, the runbook sees it first, not the SLO.