Rollback, undo, and history — the recovery toolkit

The rollback toolkit is kubectl rollout undo, kubectl rollout history, kubectl rollout pause/resume, kubectl rollout restart. The discipline is to use the toolkit in the right order: investigate, undo, validate, document.

The rollout subcommands

The kubectl rollout subcommands are the canonical recovery toolkit:

# Status: show the current rollout status
kubectl rollout status deployment/billing -n prod

# History: show the rollout history
kubectl rollout history deployment/billing -n prod

# Undo: roll back to the previous version
kubectl rollout undo deployment/billing -n prod

# Undo to a specific revision
kubectl rollout undo deployment/billing -n prod --to-revision=3

# Pause: pause the rollout
kubectl rollout pause deployment/billing -n prod

# Resume: resume the rollout
kubectl rollout resume deployment/billing -n prod

# Restart: restart the Deployment (rolling restart)
kubectl rollout restart deployment/billing -n prod

The toolkit is the on-call engineer’s primary recovery path.

The rollback flow

The rollback flow is:

flowchart TD
    A[Operator detects bad rollout] --> B[Investigate]
    B --> C[Read rollout status]
    C --> D[Read rollout history]
    D --> E[Choose rollback revision]
    E --> F[kubectl rollout undo]
    F --> G[Validate rollout]
    G --> H[Document incident]

The flow is the 11-step methodology applied to the Deployment. The operator investigates first, then chooses the rollback revision, then applies the undo, then validates.

The rollout history

The rollout history is the source of truth for the rollback:

kubectl rollout history deployment/billing -n prod

A real rollout history:

REVISION  CHANGE-CAUSE
1         <none>
2         <none>
3         Initial deploy
4         Update to 1.2.3
5         Rollback to 1.2.2

Each revision is a snapshot of the Deployment’s spec at the time of the rollout. The revision is the artifact that the rollback recovers.

The rollback to a specific revision

The rollback to a specific revision is the canonical recovery:

kubectl rollout undo deployment/billing -n prod --to-revision=3

The command tells the controller to revert the Deployment’s spec to revision 3. The controller then rolls out the new (rollback) ReplicaSet as a normal rolling update.

sequenceDiagram
    participant Operator
    participant Deployment
    participant OldRS as Old ReplicaSet
    participant NewRS as New ReplicaSet
    Operator->>Deployment: kubectl rollout undo --to-revision=3
    Deployment->>NewRS: Create new ReplicaSet with revision 3's spec
    NewRS->>OldRS: Rolling update
    OldRS-->>Deployment: Decommissioned

The rollback is a normal rolling update. The customer-facing impact is the same as a regular rollout: the new Pods become Ready, the old Pods are terminated.

The pause/resume

The pause/resume is the toolkit for staged rollouts:

# Pause the rollout
kubectl rollout pause deployment/billing -n prod

# Apply a change
kubectl set image deployment/billing -n prod \
  billing=registry.example.com/billing:1.2.4

# Resume the rollout
kubectl rollout resume deployment/billing -n prod

The pause is useful for staged rollouts where the operator wants to apply multiple changes before resuming. The pause prevents the controller from rolling out the changes immediately.

The restart

The restart is a rolling restart of the Deployment:

kubectl rollout restart deployment/billing -n prod

The restart is useful for picking up new ConfigMap or Secret changes that the Pods do not pick up automatically. The restart is a rolling update that creates a new ReplicaSet with the same spec but a new pod-template-hash.

Common mistakes

• Rollback without investigation. The operator applies kubectl rollout undo before reading the events. The rollback is a guess; the cluster is still in the same state.
• Rollback to the wrong revision. The operator applies --to-revision=N where N is the broken version. The rollback is a no-op.
• Rollback without validation. The operator applies the rollback and walks away. The cluster is not validated; the rollout is not complete.

Production discipline

The rollback toolkit is the on-call engineer’s primary recovery path. The discipline is to investigate first, then undo, then validate, then document. The toolkit is the mechanical recovery; the investigation is the discipline.

• Investigate first. Read the rollout status, the rollout history, the Pods’ state, the events.
• Choose the rollback revision. The rollout history shows the revisions.
• Apply kubectl rollout undo —to-revision=N. The rollback is a normal rolling update.
• Validate the rollout. The rollout is complete when all Pods are Ready.

Quiz