high riskservice affecting~25 min

Runbook: Roll Back a Failed Deployment

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the failure is real: error rate or latency from the production dashboard, not from one Pod
· Confirm the failure is correlated with the new revision: kubectl rollout history deploy/<name> -n <ns> shows the recent deploy and the rollout time matches the regression
· Confirm the previous revision is known and cached: kubectl get rs -n <ns> -l app=<name> shows the prior ReplicaSet with DESIRED == 0
· Confirm kubectl rollout undo will not violate a PDB: kubectl get pdb -n <ns> reports no Disallowed pods
· Confirm the cluster has capacity to roll back: kubectl describe node | grep -E "Allocatable|Allocated resources"
· Confirm a recent etcd snapshot exists in case rollback cascades further
· Confirm stakeholders are notified that a rollback is in progress

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Decide whether to pause first: if the rollout is still progressing and traffic is mixed, kubectl rollout pause deploy/<name> -n <ns>
2Issue the rollback: kubectl rollout undo deploy/<name> -n <ns> for the previous revision, or --to-revision=<n> for a specific one
3Watch the rollback converge: kubectl rollout status deploy/<name> -n <ns> --timeout=10m
4Confirm the previous ReplicaSet scaled back up and the bad one scaled to 0: kubectl get rs -n <ns> -l app=<name>
5Confirm Pods of the previous ReplicaSet become Ready: kubectl get pods -n <ns> -l app=<name>
6Smoke test the workload: curl -fsS https://<public-host>/healthz and the version endpoint
7Validate dashboards return to baseline: error rate, latency, request volume for at least 10 minutes
8Mark the bad revision in the rollout history: kubectl annotate deploy/<name> -n <ns> kubernetes.io/change-cause=""revert r<n> -- <reason>" --overwrite
9Capture the timeline and root-cause hypothesis in the change ticket
10If the application reads ConfigMap or Secret, decide whether to revert those too: only do so if they were part of the change
11Schedule the post-incident review

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl rollout status deploy/<name> -n <ns> reports successfully rolled out
✓kubectl get deploy/<name> -n <ns> reports Ready replicas equal to desired and the previous revision
✓kubectl get rs -n <ns> -l app=<name> shows the prior ReplicaSet with DESIRED == CURRENT == READY
✓kubectl describe deploy/<name> -n <ns> reports Progressing=True, Available=True
✓kubectl get endpoints -n <ns> shows the previous revision Pod IPs
✓The public smoke test returns 200 with the previous version marker
✓Dashboard p95 latency and error rate return to pre-rollout baseline within SLO
✓kubectl rollout history deploy/<name> -n <ns> records the rollback as a new revision with the documented change-cause

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the rollback itself fails to converge, pause: kubectl rollout pause deploy/<name> -n <ns>
↶If the previous revision cannot scale up because the cached image is evicted, the rollback will stall on ImagePullBackOff; pre-pull on at least one node with crictl pull <image> and resume
↶If ConfigMap or Secret was part of the change, revert those as well via the prior overlay commit and kubectl apply -k overlays/prod/config
↶If the rollback causes a second failure, scale the Deployment to zero to stop traffic immediately: kubectl scale deploy/<name> -n <ns> --replicas=0
↶Capture the failing chain (bad revision, bad config) and any partial-rollback state in the change ticket before further action

6 · Escalation

When the runbook isn't enough, contact:

· Previous ReplicaSet does not exist in kubectl get rs: revision history was pruned; this is a manifest bug — restore from a known-good manifest, not from history
· Rollback succeeds but error rate does not recover: the failure is not the Deployment — escalate to application ownership with the evidence
· PDB blocks the rollback: surge settings combined with the PDB make the cluster unschedulable in either direction; raise the PDB or lower the replicas temporarily with explicit approval
· Rollback requires re-pulling the previous image and the registry is unavailable: see the kubernetes-rb-investigate-imagepullbackoff runbook; if the image is truly gone, restore etcd from snapshot and rebuild rather than half-rolling
· Multiple rollbacks in a short window: escalate to platform ownership — the cluster is in a flapping state and the next change may not be safe

A rollback is a Deployment operation in its own right, not a “stop the bad thing” button. It scales the prior ReplicaSet up, scales the bad one down, and converges. Every step is a step the cluster has to make; it deserves the same evidence discipline as the original rollout.

1. Decide what to roll back to

kubectl rollout undo defaults to the previous revision. Use --to-revision=<n> only when the previous revision is also known-bad and an older one is known-good. List revisions first:

Read-only / SafeDecide what to roll back to


# Decide which revision is known-good. Confirm the image digest matches the version the dashboard last accepted.

2. Roll back

Read-only / SafeRoll back


kubectl rollout pause deploy/<name> -n <ns> || true

# Roll back
kubectl rollout undo deploy/<name> -n <ns>

# Watch
kubectl rollout status deploy/<name> -n <ns> --timeout=10m

# Confirm ReplicaSet state
kubectl get rs -n <ns> -l app=<name>
# Expect: prior RS scales back up, bad RS scales to 0

If the prior RS shows DESIRED > 0 but Pods do not reach Ready, the image may not be cached. Inspect:

Read-only / SafeRoll back


kubectl get pods -n <ns> -l app=<name> -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[0].state.waiting.reason}{"\n"}{end}'

# If yes, pre-pull on each node
for n in $(kubectl get nodes -o name | cut -d/ -f2); do
ssh "$n" sudo crictl pull <image>@sha256:<digest>
done

3. Validate under real traffic

A green dashboard is not enough. The application must answer correctly.

Read-only / SafeValidate under real traffic


curl -fsS https://<public-host>/healthz
curl -fsS https://<public-host>/version

# In-cluster
kubectl -n <ns> exec deploy/<name> -c web -- /app/version-check || true

# Cross-check the version in logs
kubectl logs -n <ns> -l app=<name> --tail=200 | grep -E 'version|started' | head

4. Confirm the bad revision cannot be re-applied silently

Read-only / SafeConfirm the bad revision cannot be re-applied silently


kubectl annotate deploy/<name> -n <ns> kubernetes.io/change-cause="revert r<n> -- <reason>" --overwrite

# Revert the Git commit that introduced the bad revision
git revert <sha>
git push origin main

# Open a follow-up issue for the root cause, link it to the change ticket

5. Decide on ConfigMap and Secret

If the change included configuration objects:

Read-only / SafeDecide on ConfigMap and Secret


# Which ConfigMaps does the Deployment reference?
kubectl get deploy/<name> -n <ns> -o jsonpath='{.spec.template.spec.containers[*].envFrom[*].configMapRef.name}{" "}{.spec.template.spec.volumes[?(@.configMap)].configMap.name}'
# Which Secrets does it reference?
kubectl get deploy/<name> -n <ns> -o jsonpath='{.spec.template.spec.containers[*].envFrom[*].secretRef.name}{" "}{.spec.template.spec.volumes[?(@.secret)].secret.secretName}'

# Apply the prior config overlay
kubectl apply -k overlays/prod/config
# Re-roll to pick up new mount content if necessary
kubectl rollout restart deploy/<name> -n <ns>

Common pitfalls

Symptom	Cause	Action
Rollback stalls with `ProgressDeadlineExceeded`	The previous image is not cached and the registry is down	Pre-pull on each node or escalate to DR
Old ReplicaSet does not exist	Revision history was pruned	Restore from Git, not from the cluster
Rollback succeeds but error rate does not improve	Failure is in config or downstream, not the image	Inspect ConfigMap, Secret and downstream services
Rollback triggers PDB `Disallowed pods`	Surge exceeds the PDB budget	Pause, raise PDB temporarily with approval, resume
Rollback appears to succeed but Service still serves bad Pods	EndpointSlice update lag	`kubectl get endpointslices`; usually transient

A rollback is a code change in reverse. Validate it the same way the forward change was validated, capture the same evidence, and write up the same post-change review. If the rollback is rushed, the next incident inherits the same uncertainty.