Rollback — kubectl rollout undo and revision-aware recovery

Rolling back is the safety net for failed rollouts. This lesson covers kubectl rollout undo, the --to-revision flag, what rollback does and does not do, and the production discipline around rollback readiness.

kubectl rollout undo

# Roll back to the previous revision
kubectl rollout undo deployment/web

# Roll back to a specific revision
kubectl rollout undo deployment/web --to-revision=3

kubectl rollout undo rolls the Deployment back to a previous revision. The result is a new revision (the rollback is recorded as a new change).

sequenceDiagram
    participant U as User
    participant D as Deployment
    U->>D: kubectl rollout undo --to-revision=3
    D->>D: read revision 3 spec
    D->>D: create new RS with revision 3 spec (revision 5)
    D->>D: scale up new RS
    D->>D: scale down current RS

The undo:

1. Reads the spec from the target revision.
2. Creates a new ReplicaSet with that spec.
3. Rolls forward (scales new up, old down).
4. Records the change with a new revision number and change-cause “Kubectl rolled back to revision N.”

What rollback does and does not do

kubectl rollout undo does:

• Read the spec from a previous revision.
• Roll forward to that spec (new ReplicaSet, scaled appropriately).
• Record the rollback in the change-cause.

kubectl rollout undo does NOT:

• Revert the manifest in Git. The Deployment’s spec is now at the previous revision, but Git may still have the new spec. The next GitOps sync will roll forward to the Git spec.
• Revert dependent changes (e.g., ConfigMap updates, related Deployments).
• Revert external state (e.g., a database migration triggered by an init container).

Production discipline: after a rollback, also revert the manifest in Git (commit the previous spec); the next GitOps sync should not roll forward to the broken version.

Choosing the target revision

# Inspect the history
kubectl rollout history deployment/web

# REVISION  CHANGE-CAUSE
# 1         Initial deployment
# 2         kubectl set image deployment/web web=nginx:1.27.1
# 3         kubectl set image deployment/web web=nginx:1.27.2
# 4         Update replicas to 5

The target revision:

• Latest working version: if the latest revision is broken, undo to the previous one (default kubectl rollout undo).
• Specific version: if multiple revisions are bad, undo to a known-good revision (--to-revision=2 for nginx 1.27.1).

Production discipline: track which revisions are known-good in a deployment log; incident response should target a known-good revision.

The recovery procedure

A production incident where a Deployment needs to be rolled back:

flowchart TD
    Alert[Alert: error rate spike] --> Check{Check Deployment status}
    Check --> Rollout{Rollout in progress?}
    Rollout -- yes --> Pause[Pause rollout]
    Rollout -- no --> Undo[Roll back]
    Pause --> Undo
    Undo --> Verify[Verify rollback]
    Verify -->|Rollback successful| Done[Investigate root cause]
    Verify -->|Rollback failed| Escalate[Escalate to incident response]
    Done --> Reflect[Reflect change in Git]

The steps:

1. Pause the rollout (if still in progress): kubectl rollout pause deployment/web.
2. Roll back: kubectl rollout undo deployment/web or --to-revision=N.
3. Verify the rollback: kubectl rollout status deployment/web, check the new Pods are at the previous version.
4. Reflect in Git: commit the previous spec to the manifest repository.
5. Investigate the root cause: why did the new version fail?
6. Document the incident: what broke, how it was detected, how it was fixed.

Production discipline

Test rollback in staging:

# Deploy v1
kubectl apply -f deployment-v1.yaml

# Deploy v2
kubectl apply -f deployment-v2.yaml

# Verify v2 is running
kubectl get pods -l app=web -o jsonpath='{.items[0].spec.containers[0].image}'

# Roll back
kubectl rollout undo deployment/web

# Verify v1 is running
kubectl get pods -l app=web -o jsonpath='{.items[0].spec.containers[0].image}'

A team that has never tested rollback will struggle to rollback during an incident. Production discipline: test the rollback procedure in every release cycle.

Have a documented rollback runbook:

# Rollback procedure: web Deployment

## When to roll back
- Error rate > 1% for 5+ minutes
- p99 latency > 500ms for 5+ minutes
- Crash loop or readiness probe failures

## Steps
1. Verify in #incident channel
2. Pause rollout (if in progress): kubectl rollout pause deployment/web
3. Roll back to known-good revision: kubectl rollout undo deployment/web --to-revision=N
4. Verify: kubectl rollout status deployment/web
5. Update Git to previous spec
6. Notify the channel

## Escalation
- If rollback fails: @oncall-sre
- If cluster is degraded: @incident-commander

The runbook is the difference between a quick rollback and a prolonged incident.

• In GitOps clusters, revert Git, not just live state. The next sync will roll forward if Git is unchanged.
• Track known-good revisions. A deployment log records which revisions are working and which are not.

Multiple rollbacks

# The revision number of the broken version, from
# `kubectl rollout history deployment/web`:
BROKEN_REVISION=4

# Undo the undo (back to the broken version)
kubectl rollout undo deployment/web --to-revision="$BROKEN_REVISION"

# This is intentionally a separate operation
# Usually a bad idea

kubectl rollout undo records the rollback as a new revision. The history is a forward log; rolling forward again requires another revision.

Production discipline: don’t roll back to a known-broken version. The history shows which revisions are broken; target a known-good one.

Cross-course references

• The Linux course part XXXVII-Linux-Resources covers process restart patterns; Deployment rollback is the cluster-level equivalent.
• The Ansible course part XXXV-Ansible-Scripting covers rolling restart discipline; rollout undo is the cluster-level equivalent.
• The Terraform course part XXVIII-Terraform-Disaster-Recovery covers recovery from destructive operations; rollout undo is the cluster-level equivalent.

Quiz