Lab 6: Roll back a bad Deployment

Objective

By the end of this lab you will have broken one Deployment three different ways, rolled it back four times, and produced a table showing which of those rollbacks helped, which one made the incident worse, and how many seconds each one took.

The artefact that matters is the decision note in Task 6. kubectl rollout undo is a single command that always reports success, and this lab is built so that you run it once when it is right and once when it is wrong, and can tell the two situations apart from evidence you collected beforehand.

Architecture

Everything lives inside one namespace. Nothing touches a node, the control plane, or any existing workload.

kubeadm cluster (1.34.x)
  cp-1        control plane
  worker-1    schedulable
  worker-2    schedulable   (one worker is enough)

namespace rollbacklab
  configmap/web-content   the page nginx serves, mounted as a volume
  deployment/web          3 replicas, maxSurge 1, maxUnavailable 0,
                          progressDeadlineSeconds 120
  service/web             ClusterIP in front of the Deployment
  pod/poller              busybox, curls the Service once a second, forever

workstation
  ~/k8s-rollbacklab/manifests/   what you applied
  ~/k8s-rollbacklab/evidence/    what the cluster said at each stage

The poller is the point. Every other object in this lab can be read after the fact; what the Service was serving during the ninety seconds a rollout was failing cannot, so it is recorded continuously from the start.

Requirements

A kubeadm cluster on Kubernetes 1.34.x with at least one schedulable worker. B-nested is sufficient; A-physical behaves identically.
kubectl 1.34.x, with a context allowed to create and delete a namespace.
jq, used to read the ReplicaSet revision annotations into a table. If you have no jq, Troubleshooting gives a kubectl describe fallback.
Outbound access from the nodes to docker.io for nginx:1.27.2, nginx:1.27.3 and busybox:1.36. Under 200 MB in total.
Roughly 150 MiB of memory and 0.1 CPU of headroom. The Pods request 10m CPU and 32Mi each.
Two terminals. One follows the poller’s log stream throughout; the other runs everything else. The poller output is the only continuous record of service behaviour and it is gone once the namespace is deleted.
No out-of-band access requirement. The lab reconfigures no networking, no SSH and no firewall, and nothing in it can lock you out of a node.

Scenario

A deploy went out twenty minutes ago. Error rate is up. Somebody in the channel has already typed kubectl rollout undo into their terminal and is waiting for a nod.

The question they have not asked is which of three quite different situations this is. If the rollout failed to progress, the old Pods are still serving and nothing is down — the undo is cleanup, not recovery, and it can wait until you have read the failing Pod. If the rollout completed and the new version is bad, the undo is the whole recovery and every second counts. And if what changed was not the Deployment at all, the undo will roll the image back to whatever happens to sit one position earlier in a log that your own previous rollback has already reordered — and it will report success while doing it.

This lab builds all three on a cluster where being wrong costs nothing.

Tasks

Task 1: Capture the starting state and deploy revision 1

Record what the cluster looked like before you touched it, and start the recorder that will run for the rest of the lab.

# Substitute your own value if rollbacklab is taken on this cluster:
NS=rollbacklab
WORKDIR="$HOME/k8s-rollbacklab"

mkdir -p "$WORKDIR/manifests" "$WORKDIR/evidence"
cd "$WORKDIR"

kubectl version -o yaml > evidence/00-versions.yaml
kubectl get nodes -o wide > evidence/00-nodes.txt
kubectl get namespace "$NS" > evidence/00-namespace-before.txt 2>&1

kubectl create namespace "$NS"

If evidence/00-namespace-before.txt says Error from server (NotFound), Cleanup may delete the namespace outright. If it says anything else, the namespace predates the lab and deleting it in Cleanup would destroy work that is not yours.

manifests/01-content-v1.yaml — the page, kept in its own file because Task 4 needs to restore exactly this and nothing else:

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-content
  namespace: rollbacklab
data:
  index.html: |
    content-v1 OK

manifests/02-web.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: rollbacklab
  annotations:
    kubernetes.io/change-cause: "r1: initial deployment, nginx 1.27.2, content-v1"
spec:
  replicas: 3
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 120
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: nginx:1.27.2
          ports:
            - containerPort: 80
          volumeMounts:
            - name: content
              mountPath: /usr/share/nginx/html
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 2
            periodSeconds: 3
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
      volumes:
        - name: content
          configMap:
            name: web-content
---
apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: rollbacklab
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80

progressDeadlineSeconds: 120 is a deliberate deviation from the default of 600. You want to watch a rollout give up inside this lab session rather than in ten minutes. maxUnavailable: 0 is the zero-downtime posture, and Task 3 shows what it buys.

Configuration changeworkstation— Creates the content ConfigMap, the Deployment and the Service

$ kubectl apply -f manifests/01-content-v1.yaml -f manifests/02-web.yaml

kubectl -n "$NS" rollout status deployment/web --timeout=120s

Now start the recorder in a second terminal and leave it there:

NS=rollbacklab

kubectl -n "$NS" run poller --image=busybox:1.36 --restart=Never -- \
  sh -c 'while true; do echo "$(date +%T) $(wget -q -T 2 -O - http://web/ 2>/dev/null || echo REQUEST-FAILED)"; sleep 1; done'

kubectl -n "$NS" logs -f poller

Back in the first terminal, paste the evidence helper. Every task calls it, and it collects the four things this lab argues you must read before acting:

snap() {
  tag="$1"
  {
    echo "=== rollout history ==="
    kubectl -n "$NS" rollout history deployment/web
    echo "=== deployment conditions (type status reason) ==="
    kubectl -n "$NS" get deployment web -o json |
      jq -r '.status.conditions[] | [.type, .status, .reason] | @tsv'
    echo "=== replicasets (name revision desired ready image) ==="
    kubectl -n "$NS" get rs -o json |
      jq -r '.items[] | [.metadata.name,
                         (.metadata.annotations["deployment.kubernetes.io/revision"] // "-"),
                         ((.spec.replicas // 0)|tostring),
                         ((.status.readyReplicas // 0)|tostring),
                         .spec.template.spec.containers[0].image] | @tsv'
    echo "=== pods ==="
    kubectl -n "$NS" get pods -l app=web
    echo "=== served content, last 5 samples ==="
    kubectl -n "$NS" logs poller --tail=5
  } > "evidence/$tag.txt" 2>&1
  echo "wrote evidence/$tag.txt"
}

snap 01-baseline
cat evidence/01-baseline.txt

This is a shell function, not a file. If you open a new terminal, paste it again, and set NS there too.

Task 2: Find out what is actually a revision

The rollout history is not a log the API server keeps. It is a view over the ReplicaSets that still exist, each carrying a deployment.kubernetes.io/revision annotation. Everything surprising about rollback follows from that.

Make three changes and watch which ones produce a revision. Annotate before each spec change, because the controller copies the Deployment’s annotations onto the new ReplicaSet at the moment it creates it — annotate afterwards and you are relying on a later reconcile to catch up.

# Change A: scale out. Not part of the Pod template.
kubectl -n "$NS" scale deployment/web --replicas=4
kubectl -n "$NS" rollout status deployment/web --timeout=120s
snap 02a-after-scale

# Change B: a new image. This is a Pod template change.
kubectl -n "$NS" annotate deployment/web \
  kubernetes.io/change-cause="r2: upgrade nginx to 1.27.3" --overwrite
kubectl -n "$NS" set image deployment/web web=nginx:1.27.3
kubectl -n "$NS" rollout status deployment/web --timeout=120s
snap 02b-after-image

kubectl -n "$NS" rollout history deployment/web

Read-only / Safeworkstation— The history after a scale-out and an image change

$ kubectl -n rollbacklab rollout history deployment/web

deployment.apps/web
REVISION  CHANGE-CAUSE
1         r1: initial deployment, nginx 1.27.2, content-v1
2         r2: upgrade nginx to 1.27.3

Illustrative output

Read your own numbers rather than these; every later command in this lab takes a revision number as a variable precisely because yours may differ.

The scale-out produced no revision. spec.replicas lives on the Deployment, not in spec.template, so the controller adjusted the existing ReplicaSet’s replica count and created nothing. That has a consequence you will feel in an incident: if the change that broke production was a scale-out, there is no revision to roll back to, and kubectl rollout undo will silently roll back your image instead.

The image change produced revision 2, and a second ReplicaSet. Look at the ReplicaSet section of evidence/02b-after-image.txt: two ReplicaSets, one at 4 desired and one at 0, with revisions 1 and 2 and different images. That table is the history. Nothing else stores it.

Task 3: The loud failure — a rollout that never lands

Roll out an image tag that does not exist.

kubectl -n "$NS" annotate deployment/web \
  kubernetes.io/change-cause="r3: nginx 1.27.99 (tag does not exist)" --overwrite
kubectl -n "$NS" set image deployment/web web=nginx:1.27.99

time kubectl -n "$NS" rollout status deployment/web --timeout=180s
echo "rollout status exit code: $?"

Watch the second terminal while you wait. Then collect:

snap 03-failed-rollout
cat evidence/03-failed-rollout.txt

Four observations, and the order matters.

The rollout status command failed — non-zero exit, after roughly the 120 seconds you configured as the progress deadline. In a pipeline this is the gate: kubectl rollout status is exit-code aware, which is the entire reason to put it in CI.

The Deployment conditions show Progressing False ProgressDeadlineExceeded alongside Available True. Read those two together, because they are the whole answer: the rollout has given up, and the service is fine.

The ReplicaSet table explains why. The old ReplicaSet is still at 4 desired and 4 ready. The new one is at 1 desired and 0 ready — one surge Pod, exactly maxSurge: 1, stuck in ImagePullBackOff. maxUnavailable: 0 forbade the controller from removing a single working Pod until a replacement was ready, and no replacement ever became ready, so it removed none.

The poller log shows an unbroken run of content-v1 OK. Not one REQUEST-FAILED. Count them to be sure:

kubectl -n "$NS" logs poller | grep -c REQUEST-FAILED || true

A failed rollout is not an outage, and treating it as one costs you the evidence

Progressing=False, Available=True means the new version never took over and the old one never left. Nothing is down. The correct first move is to read the new Pod — kubectl -n rollbacklab describe pod on the pending one names the registry error in its events — because that Pod is the only copy of the diagnosis you have.

Roll back first and the broken ReplicaSet scales to zero, its Pod is deleted, and the events expire. You will have restored a service that was never interrupted, at the cost of the reason it failed.

The urgency is set by maxUnavailable, not by the failure. Re-run this task with maxUnavailable: 50% and the controller terminates two working Pods to make room for replacements that never arrive: same broken image, same conditions, and half your capacity gone until somebody rolls back. That one-line strategy setting is the difference between an incident and a ticket.

Now roll back, and time it:

Service impact possibleworkstation— Rolls the Deployment forward to the previous revision's spec

$ time kubectl -n rollbacklab rollout undo deployment/web

kubectl -n "$NS" rollout status deployment/web --timeout=120s
snap 04-after-undo
kubectl -n "$NS" rollout history deployment/web

Two things to write down. The Pods are back on nginx:1.27.3, and the history does not show revision 2 any more — it shows revision 1, revision 3, and a new revision 4 carrying the 1.27.3 spec, with a change-cause the controller wrote for you. The undo rolled forward to an old spec. It did not rewind.

Record the timing in evidence/timing.txt as you go:

{
  echo "stage                       wall-clock"
  echo "forward rollout (r1 to r2)  ____ s   # from Task 2"
  echo "failed rollout (r3)          ____ s   # from Task 3"
  echo "undo after loud failure      ____ s   # from Task 3"
} > evidence/timing.txt

Task 4: The silent failure — a change that is not in the history

Change the page. Nothing else.

manifests/03-content-v2.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-content
  namespace: rollbacklab
data:
  index.html: |
    content-v2 MAINTENANCE

Service impact possibleworkstation— Replaces the mounted page content; no Deployment field changes

$ kubectl apply -f manifests/03-content-v2.yaml

Watch the second terminal. Within seconds to a minute the poller starts printing content-v2 MAINTENANCE. Every replica changed what it serves and not one Pod restarted.

snap 05-configmap-changed
kubectl -n "$NS" rollout history deployment/web
kubectl -n "$NS" get pods -l app=web

The history is unchanged. The Pod ages are unchanged. restartCount is 0. The Deployment’s Progressing condition still says the last rollout succeeded, because it did. From the Deployment’s point of view nothing has happened, and the service is serving the wrong thing.

Now do what the channel wants, and watch it go wrong:

Service impact possibleworkstation— The reflex rollback, against an incident the Deployment history does not contain

$ kubectl -n rollbacklab rollout undo deployment/web

kubectl -n "$NS" rollout status deployment/web --timeout=180s
echo "rollout status exit code: $?"
snap 06-after-bad-undo
cat evidence/06-after-bad-undo.txt

The undo command itself reported success — it edited the Deployment, which always works. What it rolled to is the revision one position back, and after Task 3’s rollback that position is revision 3: nginx:1.27.99, the tag that does not exist. The rollout stalls at the progress deadline again, the poller still prints content-v2 MAINTENANCE, and you now have two problems where you had one.

Recover properly. Read your own history, verify the target’s Pod template before you use it, then roll to it explicitly:

kubectl -n "$NS" rollout history deployment/web

# Substitute the revision number your own history shows for nginx:1.27.3:
GOOD_REV=4

kubectl -n "$NS" rollout history deployment/web --revision="$GOOD_REV" |
  grep -i image

kubectl -n "$NS" rollout undo deployment/web --to-revision="$GOOD_REV"
kubectl -n "$NS" rollout status deployment/web --timeout=120s

The image is right and the page is still wrong, because the page was never the Deployment’s to restore. Fix the thing that actually changed:

Service impact possibleworkstation— Restores the ConfigMap captured in Task 1 — the only object that changed

$ kubectl apply -f manifests/01-content-v1.yaml

snap 07-recovered
kubectl -n "$NS" logs poller --tail=10

Three operations to undo one edit, two of them only needed because the first response was a rollback instead of a question. The question was one command: kubectl rollout history deployment/web, which would have shown that nothing had rolled out.

Task 5: The revision you wanted has been deleted

revisionHistoryLimit is a retention setting. Lower it and watch rollback targets disappear.

kubectl -n "$NS" patch deployment/web \
  -p '{"spec":{"revisionHistoryLimit":2}}'

kubectl -n "$NS" rollout history deployment/web

The patch changed no Pod template field, so it created no revision — but the controller immediately garbage-collects old ReplicaSets down to the limit. Roll forward twice more to push the oldest out:

kubectl -n "$NS" annotate deployment/web \
  kubernetes.io/change-cause="r-a: build marker a" --overwrite
kubectl -n "$NS" set env deployment/web BUILD=a
kubectl -n "$NS" rollout status deployment/web --timeout=120s

kubectl -n "$NS" annotate deployment/web \
  kubernetes.io/change-cause="r-b: build marker b" --overwrite
kubectl -n "$NS" set env deployment/web BUILD=b
kubectl -n "$NS" rollout status deployment/web --timeout=120s

snap 08-history-trimmed
kubectl -n "$NS" rollout history deployment/web
kubectl -n "$NS" get rs

Revision 1 — the original nginx:1.27.2 deployment — is gone from the history, and so is its ReplicaSet. Try to reach it:

Read-only / Safeworkstation— Attempts a rollback to a revision the controller has already deleted

$ kubectl -n rollbacklab rollout undo deployment/web --to-revision=1

error: unable to find specified revision 1 in history

Illustrative output

Task 6: Write the decision note

This is the deliverable. Create evidence/decision.md and answer, from your own evidence files rather than from this page:

Failure 1  bad image tag, rollout never progressed
  Conditions observed:
  Was the service affected?
  Was rollback the right first action?
  What did rolling back destroy?

Failure 2  ConfigMap edit, no rollout at all
  What in the Deployment showed the change?
  What did the bare `undo` roll back to, and why that revision?
  How many operations did recovery take, and how many were avoidable?

Failure 3  rollback target garbage-collected
  What was the configured limit?
  How many deploys of history did it represent?

The one command that would have separated failure 1 from failure 2 before
any action was taken:

Then add the timing table. If you cannot fill a line from your own files, re-run that stage.

Validation

Run these against your evidence rather than against the cluster; the point is that the files support the conclusions.

cd "$HOME/k8s-rollbacklab"

grep -c . evidence/decision.md
grep -A 4 "deployment conditions" evidence/03-failed-rollout.txt
grep -A 6 "replicasets" evidence/03-failed-rollout.txt
grep -A 4 "rollout history" evidence/05-configmap-changed.txt
grep -c REQUEST-FAILED evidence/03-failed-rollout.txt || true
cat evidence/timing.txt

The lab succeeded when all of the following hold:

evidence/03-failed-rollout.txt shows Progressing False ProgressDeadlineExceeded and Available True, an old ReplicaSet with ready replicas, and a new ReplicaSet at 1 desired and 0 ready.
The poller samples in evidence/03-failed-rollout.txt are all content-v1 OK, with no REQUEST-FAILED.
evidence/02a-after-scale.txt and evidence/01-baseline.txt show the same revision count; evidence/02b-after-image.txt shows one more.
evidence/05-configmap-changed.txt shows served content of content-v2 MAINTENANCE with a rollout history identical to evidence/04-after-undo.txt.
evidence/06-after-bad-undo.txt shows the Deployment back on the non-existent image after a command that reported success.
evidence/07-recovered.txt shows nginx:1.27.3 and content-v1 OK.
evidence/08-history-trimmed.txt lists at most three revisions and no ReplicaSet carrying nginx:1.27.2.
evidence/decision.md names one command, not two, as the discriminator.

Expected Outcome

~/k8s-rollbacklab/
├── manifests/
│   ├── 01-content-v1.yaml
│   ├── 02-web.yaml
│   └── 03-content-v2.yaml
└── evidence/
    ├── 00-versions.yaml, 00-nodes.txt, 00-namespace-before.txt
    ├── 01-baseline.txt
    ├── 02a-after-scale.txt, 02b-after-image.txt
    ├── 03-failed-rollout.txt, 04-after-undo.txt
    ├── 05-configmap-changed.txt, 06-after-bad-undo.txt, 07-recovered.txt
    ├── 08-history-trimmed.txt
    ├── timing.txt
    └── decision.md

A set of files in which each rollback decision is supported by the Deployment conditions that preceded it, and a timing table in which the cost of a rollback is a measured number rather than an assumption. That pairing — a condition and the action it justifies — is what turns rollout undo from a reflex into a decision.

Production notes

Map this onto a real change window as follows.

Before the change. Record the current revision number and the image it carries, in the change ticket, not only in the cluster: kubectl rollout history deployment/web plus --revision=N. This is your rollback target, and Task 5 is why it must live outside the cluster. Confirm revisionHistoryLimit is large enough to still contain it after the deploys queued behind yours.

During the change. kubectl rollout status --timeout= is the gate, and its exit code is the signal — a pipeline that runs kubectl apply and moves on has not deployed anything, it has submitted a request. Set progressDeadlineSeconds to something a human will still be watching for; the 600-second default outlasts most attention spans and most change windows.

When it goes wrong. Read Progressing and Available together before acting. Available=True means the old version is still serving and you have time to capture the failing Pod’s events and describe output — which the rollback will destroy. Available=False means capacity is already gone and recovery comes first.

Rolling back. Always --to-revision=N against a number you have read. Then reconcile the source of truth: in a GitOps cluster the live rollback is undone by the next sync unless the manifest is reverted too, and in any cluster the next kubectl apply from CI rolls the broken version straight back out.

Holding. Holding is a first-class option with an owner and an end time. For a stalled rollout with maxUnavailable: 0, holding costs nothing but one surge Pod and preserves every piece of evidence. Write it down as “hold until HH:MM, owner NAME, roll back if error rate exceeds X” — an unowned hold is just a stalled incident.

Troubleshooting

snap says “command not found”. It is a shell function from Task 1 and it lives only in the shell you pasted it into. Paste it again, and set NS in that shell too.

No jq. Replace the ReplicaSet block with kubectl -n rollbacklab describe rs | grep -E '^(Name:|Annotations:|Replicas:|Image:)', or read the annotation for one ReplicaSet at a time with kubectl -n rollbacklab describe rs RSNAME. Set RSNAME from kubectl -n rollbacklab get rs.

The CHANGE-CAUSE column is empty for a revision. The annotation must be on the Deployment when the controller creates the new ReplicaSet. If you annotated after the spec change, wait a few seconds for the next reconcile and re-check; if it stays empty, put the annotation and the spec change in one manifest and apply them together.

The revision numbers in my history do not match the illustrative output. Expected, and the reason every command here takes a variable. The controller renumbers ReplicaSets on reuse, so an extra apply anywhere shifts the sequence. Use your own numbers.

The poller shows REQUEST-FAILED during Task 3. With maxUnavailable: 0 it should not. Check that manifests/02-web.yaml applied with that value — kubectl -n rollbacklab get deployment web -o jsonpath='{.spec.strategy}' — and that the Service selector still matches app: web.

The ConfigMap change never reaches the Pods in Task 4. Confirm the volume is a plain configMap mount and not a subPath; subPath mounts never update. Then give it the full kubelet sync period before concluding anything.

rollout status returns immediately with success during a failed rollout. You ran it before the controller observed the new generation. Re-run it, or check .status.observedGeneration against .metadata.generation.

Pods stay Pending with Insufficient cpu or Insufficient memory. The cluster has no room for the surge Pod. Scale the Deployment down to 2 replicas and repeat; the lab’s behaviour is unchanged.

Cleanup

The lab created exactly one namespace and one working directory. Both go, and both are verified rather than assumed.

Step 1. Confirm what you are about to remove, and preserve the poller log:

NS=rollbacklab

kubectl -n "$NS" get all
cat "$HOME/k8s-rollbacklab/evidence/00-namespace-before.txt"
kubectl -n "$NS" logs poller > "$HOME/k8s-rollbacklab/evidence/poller-full.log"

Step 2. Delete the namespace and wait for it to actually go:

Destructiveworkstation— Removes the lab namespace and every object in it

$ kubectl delete namespace rollbacklab --wait=true

Step 3. Verify the cluster is as you found it:

NS=rollbacklab

kubectl get namespace "$NS" || echo "namespace gone, as expected"
diff <(kubectl get nodes -o wide) "$HOME/k8s-rollbacklab/evidence/00-nodes.txt" \
  && echo "node inventory unchanged"

Step 4. Keep the deliverables, then remove the working directory.

ls -la "$HOME/k8s-rollbacklab"

mkdir -p "$HOME/k8s-lab-deliverables/rollback"
cp -a "$HOME/k8s-rollbacklab/evidence" "$HOME/k8s-rollbacklab/manifests" \
      "$HOME/k8s-lab-deliverables/rollback/"

rm -rf "$HOME/k8s-rollbacklab"

What You Learned

The revision history is the surviving ReplicaSets. Each one carries a deployment.kubernetes.io/revision annotation, and that annotation is the only place the number lives.
Only Pod template changes create revisions. You scaled from 3 to 4 and the history did not move, which means a scale-out cannot be rolled back and a bare undo after one will roll back your image instead.
Progressing=False with Available=True is a failed rollout, not an outage. The poller proved it: 120 seconds of a stalled rollout and not one failed request, because maxUnavailable: 0 refused to remove a working Pod.
maxUnavailable sets the urgency, not the failure. The same broken image is a ticket at 0 and an incident at 50%.
A mounted ConfigMap changes behaviour with no revision and no restart. The Deployment history contained no record of the change that broke the page — and subPath and env consumers would not have changed at all.
Bare undo is a position in a mutable log. Rolling back renumbers the ReplicaSet you rolled back to, which promotes the broken revision to “previous”. Your second undo restored the failure. Pass --to-revision=N.
revisionHistoryLimit is your rollback horizon. At 2 you could no longer reach revision 1, and the command failed rather than doing something approximate.

Why this matters

Rollback is the one production skill everybody believes they have. The command is short, it is in every runbook, and it almost always exits 0 — which is exactly the problem, because exiting 0 tells you the Deployment was edited, not that the incident is over.

The three failures in this lab are the three that actually happen. A rollout that will not progress is the common one and the least dangerous, and the standard mistake is to treat it as an emergency: the old Pods are serving, the broken Pod holds the only copy of the diagnosis, and rolling back deletes it, so the same bad tag ships again next week. A change that never entered the history is the one that wastes an hour, because every tool the team reaches for describes the Deployment, and the Deployment is innocent. And the garbage collected revision is the one that turns a five-minute recovery into an escalation, discovered at the worst possible moment because nobody has ever looked at revisionHistoryLimit.

Underneath all three is the same misconception: that revision history is a durable archive of known-good states. It is a handful of ReplicaSets that a controller renumbers on reuse and deletes on a retention setting. Once you have watched your own rollback promote a broken revision to “previous”, you stop typing bare undo — and that single habit, --to-revision against a number you read first, is most of the difference between a rollback that ends an incident and one that extends it.

Objectives

Prerequisites