Lab 5: Drive a Deployment rolling update

Objective

By the end of this lab you will have driven four rolling updates and measured every one of them, and you will be able to answer the question a change board actually asks: “how much capacity does this need, how long will it take, and what does the user see while it happens?”

The measurement is the point. kubectl rollout status printing successfully rolled out is a statement about ReplicaSet counts, not about whether anybody could reach the service. In Task 5 you will produce a rollout that reports success while a client inside the cluster records failures for twenty seconds, and the difference between those two views is the whole lesson.

Architecture

One namespace, one Deployment, one Service, and a client Pod that measures from inside the cluster rather than from your laptop.

flowchart LR
    OP["operator<br/>kubectl set image"] --> DEP["Deployment web<br/>replicas 6"]
    DEP --> RSOLD["ReplicaSet<br/>old revision"]
    DEP --> RSNEW["ReplicaSet<br/>new revision"]
    RSOLD --> POLD["Pods · old"]
    RSNEW --> PNEW["Pods · new"]
    POLD -->|"only if Ready"| EPS["EndpointSlice"]
    PNEW -->|"only if Ready"| EPS
    SVC["Service web:80"] --> EPS
    PROBE["Pod probe<br/>wget in a loop"] --> SVC

The arrow labelled only if Ready is the one this lab is about. It is the same condition the Deployment controller waits on before it scales the old ReplicaSet down, and it is the same condition the EndpointSlice controller uses to decide whether traffic may reach a Pod. One signal, two consumers, and every availability property of a rolling update follows from it.

kubeadm cluster (1.34.x)
  k8s-cp-1     control plane
  k8s-w-1      worker
  k8s-w-2      worker

namespace rollout
  web          Deployment, 6 replicas
  web          Service, ClusterIP :80
  probe        Pod, runs a fixed number of requests and prints a failure count

workstation
  ~/k8s-rollout/manifests/   the four Deployment variants
  ~/k8s-rollout/evidence/    sampled CSVs, probe output, conditions

Requirements

• A kubeadm cluster on Kubernetes 1.34.x with two schedulable workers. One worker works, but the surge behaviour is much easier to read when the Pods are spread.
• kubectl 1.34.x with permission to create and delete a namespace.
• Two terminals. One samples Deployment status once a second while the other triggers the rollout. The evidence this lab produces does not exist after the fact — kubectl get a minute later shows a finished rollout and nothing about its shape.
• Cluster headroom for 8 nginx Pods at 32Mi/50m each: roughly 300 MiB of memory and 0.5 CPU of allocatable capacity beyond what is already running. Six replicas plus a surge of two is the peak, and if the cluster cannot fit the surge the rollout stalls for a reason that has nothing to do with the lab.
• Outbound access from the nodes to docker.io for nginx:1.27.2 and busybox:1.36.
• No out-of-band access requirement. Everything runs through kubectl against one namespace; nothing touches a node, the control plane or the network fabric.

Scenario

The change is trivial: bump an image tag. It has been approved, it is in the window, and the entire risk conversation was somebody saying “it is a rolling update, there is no downtime.”

That sentence is true for a particular set of settings and false for the defaults. A Deployment with maxUnavailable: 25% is designed to run below capacity during the change. A Deployment whose Pods report Ready before they can serve will drop requests and report a clean rollout. And a Deployment whose surge does not fit on the cluster will not roll at all, which is the safest of the three outcomes and the one that gets escalated fastest.

This lab is where you find out which of those you have, on a cluster where the answer costs nothing.

Tasks

Task 1 — Workspace, namespace, baseline

NS=rollout

mkdir -p "$HOME/k8s-rollout/manifests" "$HOME/k8s-rollout/evidence"
cd "$HOME/k8s-rollout"

kubectl get namespace "$NS" > evidence/00-namespace-before.txt 2>&1
cat evidence/00-namespace-before.txt

kubectl create namespace "$NS"
kubectl version -o yaml > evidence/00-versions.yaml
kubectl get nodes -o wide > evidence/00-nodes.txt
kubectl get nodes -o custom-columns='NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,UNSCHEDULABLE:.spec.unschedulable' \
  > evidence/00-nodes-stable.txt
kubectl describe nodes | grep -A6 'Allocated resources' > evidence/00-allocated.txt

cat evidence/00-allocated.txt

Read 00-allocated.txt before going further. It is the answer to “does the surge fit”, and it is the check nobody runs until a rollout has already stalled.

Task 2 — Deploy version 1 and read its identity

manifests/01-web-v1.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: rollout
  annotations:
    kubernetes.io/change-cause: "v1 baseline: nginx 1.27.2, surge 25%, unavailable 25%"
spec:
  replicas: 6
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 120
  minReadySeconds: 0
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: app
          image: nginx:1.27.2
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 2
            failureThreshold: 2
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              memory: 64Mi

cd "$HOME/k8s-rollout"

kubectl apply -f manifests/01-web-v1.yaml
kubectl -n rollout rollout status deployment/web --timeout=180s

kubectl -n rollout expose deployment web --port=80 --name=web
kubectl -n rollout get svc web

Now read what the controller built underneath the Deployment:

kubectl -n rollout get rs -o custom-columns='NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas,REVISION:.metadata.annotations.deployment\.kubernetes\.io/revision,HASH:.metadata.labels.pod-template-hash'
kubectl -n rollout get pods --show-labels | head -3

One ReplicaSet, revision 1, and a pod-template-hash that appears both in the ReplicaSet’s name and as a label on every Pod it owns. That hash is computed from the Pod template, and it is the mechanism that keeps two revisions’ selectors from colliding: the Deployment’s selector matches app=web, which would match both generations, so the controller adds the hash to each ReplicaSet’s own selector.

Write down your prediction before Task 3, in evidence/prediction.md:

• 25% of 6 is 1.5. maxSurge rounds up — so up to 2 extra Pods, peak 8.
• maxUnavailable rounds down — so at most 1 unavailable, floor 5.
• Therefore during the rollout, total Pods stay in [5, 8] and available Pods never drop below 5.

Predict it, then measure it. The rounding rule is the part people get wrong, and it is asymmetric on purpose: Kubernetes rounds in the direction that preserves capacity in both cases.

Task 3 — Measure one rolling update

Terminal A — start the sampler first and leave it running:

cd "$HOME/k8s-rollout"

echo "t,desired,total,ready,available,updated" > evidence/03-rollout.csv
for i in $(seq 1 90); do
  kubectl -n rollout get deployment web -o jsonpath='{.spec.replicas}{","}{.status.replicas}{","}{.status.readyReplicas}{","}{.status.availableReplicas}{","}{.status.updatedReplicas}{"\n"}' \
    | sed "s/^/$(date +%s),/" >> evidence/03-rollout.csv
  sleep 1
done
echo "sampling finished"

Terminal B — once the sampler is running, trigger the change:

$ kubectl -n rollout set image deployment/web app=nginx:1.27.3

kubectl -n rollout annotate deployment/web \
  kubernetes.io/change-cause="v2: nginx 1.27.3" --overwrite
kubectl -n rollout rollout status deployment/web --timeout=180s

When the sampler in Terminal A finishes, derive the two numbers that matter:

cd "$HOME/k8s-rollout"

awk -F, 'NR>1 {
  total=$3+0; avail=$5+0;
  if (total>peak) peak=total;
  if (!seen || avail<floor) { floor=avail; seen=1 }
} END {
  printf "peak total pods: %d   (predicted limit 8)\n", peak;
  printf "minimum available: %d (predicted floor 5)\n", floor;
}' evidence/03-rollout.csv

An empty field in the CSV is a real zero: the API server omits status.readyReplicas and status.availableReplicas entirely when they are zero rather than writing 0, and jsonpath prints nothing for a missing field. The +0 in the awk expression is what turns that back into a number, and it is worth knowing because the same omission catches out anything that parses Deployment status.

Confirm what actually changed:

cd "$HOME/k8s-rollout"

kubectl -n rollout get rs -o custom-columns='NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas,REVISION:.metadata.annotations.deployment\.kubernetes\.io/revision' \
  | tee evidence/03-replicasets.txt
kubectl -n rollout rollout history deployment/web | tee evidence/03-history.txt

The old ReplicaSet is still there, scaled to zero. It is retained because revisionHistoryLimit is 10, and it holds the complete Pod template of the previous revision — which is what makes an undo possible without the original manifest.

Task 4 — Measure the same change from a client

Status fields describe the controller’s view. A client describes the user’s.

cd "$HOME/k8s-rollout"

cat > evidence/probe-cmd.txt <<'PROBE'
i=0; f=0
while [ $i -lt 90 ]; do
  if ! wget -q -T 2 -O /dev/null http://web/; then f=$((f+1)); fi
  i=$((i+1))
  sleep 1
done
echo "requests=$i failures=$f"
PROBE

cat evidence/probe-cmd.txt

Terminal A — start the client, which runs 90 requests over roughly 90 seconds and then prints its tally:

kubectl -n rollout delete pod probe --ignore-not-found
kubectl -n rollout run probe --image=busybox:1.36 --restart=Never -- \
  sh -c 'i=0; f=0; while [ $i -lt 90 ]; do if ! wget -q -T 2 -O /dev/null http://web/; then f=$((f+1)); fi; i=$((i+1)); sleep 1; done; echo "requests=$i failures=$f"'

kubectl -n rollout wait pod/probe --for=condition=Ready --timeout=60s

Terminal B — roll back to the previous image, which is just another rollout in the opposite direction:

kubectl -n rollout set image deployment/web app=nginx:1.27.2
kubectl -n rollout rollout status deployment/web --timeout=180s

Terminal A — when the client Pod completes, read its tally:

cd "$HOME/k8s-rollout"

kubectl -n rollout wait pod/probe --for=jsonpath='{.status.phase}'=Succeeded --timeout=180s
kubectl -n rollout logs probe | tee evidence/04-probe-25pct.txt

Now repeat with the capacity-preserving setting and compare:

cd "$HOME/k8s-rollout"

kubectl -n rollout patch deployment web --type=merge -p \
  '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":0}}}}'
kubectl -n rollout get deployment web -o jsonpath='{.spec.strategy.rollingUpdate}{"\n"}'

kubectl -n rollout delete pod probe --ignore-not-found
kubectl -n rollout run probe --image=busybox:1.36 --restart=Never -- \
  sh -c 'i=0; f=0; while [ $i -lt 90 ]; do if ! wget -q -T 2 -O /dev/null http://web/; then f=$((f+1)); fi; i=$((i+1)); sleep 1; done; echo "requests=$i failures=$f"'
kubectl -n rollout wait pod/probe --for=condition=Ready --timeout=60s

Then, in Terminal B, roll forward again and let the client finish:

kubectl -n rollout set image deployment/web app=nginx:1.27.3
kubectl -n rollout rollout status deployment/web --timeout=180s

cd "$HOME/k8s-rollout"

kubectl -n rollout wait pod/probe --for=jsonpath='{.status.phase}'=Succeeded --timeout=180s
kubectl -n rollout logs probe | tee evidence/04-probe-0pct.txt

cat evidence/04-probe-25pct.txt evidence/04-probe-0pct.txt

On a healthy cluster with a correct readiness probe both runs may report zero failures, because six replicas behind one Service means the ClusterIP still has five ready backends even at maxUnavailable: 25%. That is the honest result and it is worth stating: at this replica count, on this workload, the setting bought you headroom rather than availability.

The number that changed is the one in evidence/03-rollout.csv — the floor. At maxUnavailable: 0 the Deployment never runs below six available Pods, which means the next failure, the one you did not plan for, has the full fleet to absorb it. That is what the setting actually buys, and it costs surge capacity you must have spare.

Task 5 — Readiness is the only gate

Everything so far assumed the readiness probe tells the truth. Now break that assumption honestly: keep the same image, and make the container take twenty seconds to start serving.

manifests/02-slowstart-noprobe.yaml — the same Deployment with a slow start and no readiness probe:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: rollout
  annotations:
    kubernetes.io/change-cause: "v3: 20s slow start, readiness probe removed"
spec:
  replicas: 6
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 120
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: app
          image: nginx:1.27.2
          command:
            - sh
            - -c
            - sleep 20; exec nginx -g 'daemon off;'
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              memory: 64Mi

Terminal A — start the client. Terminal B — apply the manifest and time the rollout:

cd "$HOME/k8s-rollout"

time (kubectl apply -f manifests/02-slowstart-noprobe.yaml \
  && kubectl -n rollout rollout status deployment/web --timeout=180s)

time wraps both commands deliberately. kubectl apply returns as soon as the API server has stored the object, which is a fraction of a second and tells you nothing; the number you want is how long the fleet took to be replaced, and that is what rollout status waits for.

cd "$HOME/k8s-rollout"

kubectl -n rollout wait pod/probe --for=jsonpath='{.status.phase}'=Succeeded --timeout=180s
kubectl -n rollout logs probe | tee evidence/05-probe-noprobe.txt

The rollout reported success in well under twenty seconds, because with no readiness probe a container is Ready the moment the kubelet has started it. The Deployment controller scaled the old ReplicaSet away against a signal that meant “the process exists”, and the EndpointSlice controller added those Pods as Service backends on the same signal. The failure count in evidence/05-probe-noprobe.txt is what the users got.

Now restore the probe with the slow start still in place. manifests/03-slowstart-probe.yaml — identical apart from maxUnavailable: 0 and the readinessProbe block:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: rollout
  annotations:
    kubernetes.io/change-cause: "v4: 20s slow start, readiness probe restored, unavailable 0"
spec:
  replicas: 6
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 300
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: app
          image: nginx:1.27.2
          command:
            - sh
            - -c
            - sleep 20; exec nginx -g 'daemon off;'
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 2
            failureThreshold: 2
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              memory: 64Mi

The progressDeadlineSeconds is raised to 300 here on purpose. Three batches at roughly twenty-five seconds each is comfortably inside 120, but a deadline that is merely “probably enough” is how a slow registry turns a working rollout into a failed one.

cd "$HOME/k8s-rollout"

kubectl -n rollout delete pod probe --ignore-not-found
kubectl -n rollout run probe --image=busybox:1.36 --restart=Never -- \
  sh -c 'i=0; f=0; while [ $i -lt 120 ]; do if ! wget -q -T 2 -O /dev/null http://web/; then f=$((f+1)); fi; i=$((i+1)); sleep 1; done; echo "requests=$i failures=$f"'
kubectl -n rollout wait pod/probe --for=condition=Ready --timeout=60s

cd "$HOME/k8s-rollout"

time (kubectl apply -f manifests/03-slowstart-probe.yaml \
  && kubectl -n rollout rollout status deployment/web --timeout=300s)

kubectl -n rollout wait pod/probe --for=jsonpath='{.status.phase}'=Succeeded --timeout=300s
kubectl -n rollout logs probe | tee evidence/05-probe-withprobe.txt

cat evidence/05-probe-noprobe.txt evidence/05-probe-withprobe.txt

The rollout took several times longer and the failure count went to zero. Both of those are the probe doing its job: it held each new Pod out of the EndpointSlice until it could serve, which forced the Deployment controller to wait before removing the old Pod that was serving in its place.

Task 6 — The strategy the API server will not accept

The pathological setting is worth trying once so you know what happens.

cd "$HOME/k8s-rollout"

kubectl -n rollout patch deployment web --type=merge -p \
  '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":0}}}}' \
  2>&1 | tee evidence/06-invalid-strategy.txt

Record exactly what your cluster returns. With both values at zero the Deployment could not make progress in either direction — it cannot create a new Pod without exceeding the replica count, and cannot remove an old one without going below it — so the combination is rejected at validation rather than discovered later at the progress deadline.

That is the useful shape of the fact: this class of error is caught by the API server at admission time, which is one more reason to run --dry-run=server on a manifest before applying it. A setting the API server accepts is not necessarily safe, but a setting it rejects never reaches a cluster.

Put the strategy back:

kubectl -n rollout patch deployment web --type=merge -p \
  '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":0}}}}'
kubectl -n rollout get deployment web -o jsonpath='{.spec.strategy.rollingUpdate}{"\n"}'

Task 7 — Stall a rollout, and decide what to do about it

Now the failure that actually happens on a Tuesday: the tag is wrong.

cd "$HOME/k8s-rollout"

kubectl -n rollout patch deployment web --type=merge -p \
  '{"spec":{"progressDeadlineSeconds":60}}'

kubectl -n rollout set image deployment/web app=nginx:1.27.2-typo
kubectl -n rollout annotate deployment/web \
  kubernetes.io/change-cause="v4: image tag typo, deliberate" --overwrite

$ kubectl -n rollout rollout status deployment/web --timeout=120s ; echo "exit=$?"

Do not fix it yet. Collect the evidence first, because this is the state you will be reading under pressure:

cd "$HOME/k8s-rollout"

kubectl -n rollout get deployment web \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\n"}{end}' \
  | tee evidence/07-conditions.txt

kubectl -n rollout get rs -o custom-columns='NAME:.metadata.name,DESIRED:.spec.replicas,CURRENT:.status.replicas,READY:.status.readyReplicas' \
  | tee -a evidence/07-conditions.txt

kubectl -n rollout get pods -o wide | tee evidence/07-pods.txt
kubectl -n rollout get endpointslices -l kubernetes.io/service-name=web \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\t"}{.conditions.ready}{"\n"}{end}' \
  | tee evidence/07-endpoints.txt

Read evidence/07-conditions.txt carefully. You should see Progressing with status False and reason ProgressDeadlineExceeded, and — on the same object, at the same moment — Available with status True.

Those two lines are the whole diagnosis. The rollout failed. The service did not. The old ReplicaSet still has its Pods, they are still Ready, they are still in the EndpointSlice, and every request is still being served. What has happened is that a change did not land, which is a completely different incident from an outage and carries a completely different urgency.

Now abort, deliberately, and watch it:

cd "$HOME/k8s-rollout"

kubectl -n rollout rollout history deployment/web | tee -a evidence/03-history.txt
kubectl -n rollout rollout undo deployment/web
kubectl -n rollout rollout status deployment/web --timeout=180s

kubectl -n rollout get deployment web -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n rollout rollout history deployment/web | tail -5

Note what the history shows: the undo did not delete the failed revision, it added a new one whose template happens to match an earlier revision. A Deployment’s history is an append-only log of changes, and an undo is another change. That is why rollout history after an undo has more rows than before, and why the revision numbers never go backwards.

Task 8 — Write the change record

The deliverable. In evidence/change-record.md, and in fewer than fifteen lines:

• Surge capacity required. Peak Pods from Task 3, times the Pod’s requests, compared against evidence/00-allocated.txt.
• Expected duration. Batches × per-batch readiness time, from the timings in Tasks 3 and 5. Give a range, not a point.
• What the user sees. The two client failure counts, and which setting produced which.
• Abort criterion. A condition, not a feeling: “if rollout status has not succeeded within progressDeadlineSeconds, abort” is a criterion; “if it looks wrong” is not.
• Owner. Who decides between hold, forward and abort, and by when.

Validation

cd "$HOME/k8s-rollout"

ls -1 evidence/
head -3 evidence/03-rollout.csv
wc -l evidence/03-rollout.csv
cat evidence/04-probe-25pct.txt evidence/04-probe-0pct.txt
cat evidence/05-probe-noprobe.txt evidence/05-probe-withprobe.txt
cat evidence/07-conditions.txt
kubectl -n rollout get deployment web -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

The lab succeeded when all of the following hold:

• evidence/03-rollout.csv has more than one distinct value in its total column — a file where every row is 6 means the sampler started after the rollout finished, and the measurement has to be repeated.
• The peak total from the awk analysis is at most 8, and the minimum available is at least 5, matching evidence/prediction.md. If the peak is 6, the rollout completed between two samples; shorten sleep or add replicas.
• evidence/05-probe-noprobe.txt reports a non-zero failure count and evidence/05-probe-withprobe.txt reports zero, from the same slow-starting workload.
• The apply in Task 5 with no probe completed in noticeably less wall-clock time than the one with the probe, and you can say why the faster one is the worse outcome.
• evidence/06-invalid-strategy.txt contains the API server’s rejection, quoted.
• evidence/07-conditions.txt shows Progressing False ProgressDeadlineExceeded and Available True together, and evidence/07-endpoints.txt shows six ready endpoints during the stall.
• After the undo, the Deployment’s image is the last good tag and rollout history has more revisions than before the undo, not fewer.
• evidence/change-record.md names a numeric abort criterion and a person.

Expected Outcome

~/k8s-rollout/
├── manifests/
│   ├── 01-web-v1.yaml
│   ├── 02-slowstart-noprobe.yaml
│   └── 03-slowstart-probe.yaml
└── evidence/
    ├── 00-namespace-before.txt, 00-versions.yaml
    ├── 00-nodes.txt, 00-nodes-stable.txt, 00-allocated.txt
    ├── prediction.md
    ├── 03-rollout.csv, 03-replicasets.txt, 03-history.txt
    ├── 04-probe-25pct.txt, 04-probe-0pct.txt
    ├── 05-probe-noprobe.txt, 05-probe-withprobe.txt
    ├── 06-invalid-strategy.txt
    ├── 07-conditions.txt, 07-pods.txt, 07-endpoints.txt
    └── change-record.md

In the cluster: one namespace with a six-replica Deployment on a known-good image, a Service in front of it, several scaled-to-zero ReplicaSets holding the revision history, and no probe Pod.

Troubleshooting

The rollout finishes before the sampler records anything interesting. Six nginx Pods with a two-second readiness period roll fast. Raise replicas to 10, lower periodSeconds on the probe, or start the sampler and wait a few seconds before triggering the change in the other terminal.

kubectl rollout status returns immediately with success and nothing changed. kubectl set image with the image the Deployment already has is a no-op: the Pod template is unchanged, so no new ReplicaSet is created and there is no rollout. Check kubectl rollout history for a new revision.

The rollout stalls at Task 3 with new Pods Pending. The surge does not fit. kubectl -n rollout describe pod on a Pending Pod names the reason under Events — Insufficient cpu or Insufficient memory. Lower replicas, or set maxSurge: 1, and note that you have just reproduced the most common real cause of a stalled rollout.

The client Pod reports failures even when nothing is rolling. Cluster DNS or the Service is the problem, not the rollout. Test with kubectl -n rollout run dnscheck --rm -it --image=busybox:1.36 --restart=Never -- nslookup web before blaming the Deployment.

kubectl -n rollout logs probe is empty. The Pod has not finished. It runs 90 or 120 requests one second apart and prints its tally only at the end; kubectl -n rollout get pod probe shows Running until then.

The stalled rollout in Task 7 never stalls — the Pods pull an image successfully. A registry mirror or a pull-through cache resolved the typo’d tag to something. Use a tag that is unambiguously absent, and confirm from the Pod’s events that the reason is ErrImagePull or ImagePullBackOff.

rollout undo says there is no rollout history. revisionHistoryLimit was reached, or the Deployment was recreated rather than patched. The old ReplicaSets are the history; kubectl -n rollout get rs shows what is left.

Cleanup

The lab created one namespace, several ReplicaSets inside it, and one working directory.

Step 1. Confirm what is about to go, and that the namespace was yours:

cd "$HOME/k8s-rollout"

kubectl -n rollout get all
cat evidence/00-namespace-before.txt

Step 2. Delete the namespace:

$ kubectl delete namespace rollout --wait=true

Step 3. Verify the cluster is as you found it. This matters more than usual here: the lab ran eight Pods at peak, and a namespace stuck Terminating leaves them scheduled.

cd "$HOME/k8s-rollout"

kubectl get namespace rollout || echo "namespace gone, as expected"
kubectl get pods -A | grep -q '^rollout ' \
  && echo "lab pods still present" || echo "no lab pods remain"
diff <(kubectl get nodes -o custom-columns='NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,UNSCHEDULABLE:.spec.unschedulable') \
     evidence/00-nodes-stable.txt \
  && echo "node inventory unchanged"

Step 4. Keep the deliverables, then remove the working directory.

ls -la "$HOME/k8s-rollout"

mkdir -p "$HOME/k8s-lab-deliverables/rolling-update"
cp -a "$HOME/k8s-rollout/evidence" "$HOME/k8s-rollout/manifests" \
      "$HOME/k8s-lab-deliverables/rolling-update/"

rm -rf "$HOME/k8s-rollout"

Production notes

This lab is a rehearsal for a change window, and it produces the three artefacts a change window needs.

The capacity number is a precondition, not a footnote. A maxSurge: 25% Deployment at 40 replicas needs room for 10 more Pods at the moment of the change. On a cluster running at 90% allocation, that rollout does not fail loudly — it goes Pending and sits there until the progress deadline, at which point the Deployment reports failure and the service is still on the old version. Checking allocatable capacity before a rollout is a thirty-second command, and Task 1 is where it belongs.

The duration estimate is batches times readiness, and readiness is not startup. A workload that takes 60 seconds to become ready, rolled at maxSurge: 1 across 20 replicas, is a twenty-minute change even though the image bump itself is instant. Teams routinely book a ten-minute window for this and then start improvising at minute eleven.

The abort criterion belongs in the pipeline, not in a person’s judgement. kubectl rollout status --timeout=Ns returns a non-zero exit code, which is exactly what a CI step needs. Gating a deploy job on it, with an automatic rollout undo on failure, converts the most common bad outcome — a half-rolled Deployment nobody noticed — into a failed build.

Two honest limits of this lab. progressDeadlineSeconds: 60 is far too short for production; it is used here so the stall is observable inside a lab session, and the default of 600 exists because real images sometimes take minutes to pull. And six replicas of nginx behind one Service is a forgiving workload: sessions, connection draining, database migrations and anything stateful all make a rolling update harder in ways this Deployment cannot show you.

What You Learned

• The envelope is arithmetic you can predict. maxSurge rounds up, maxUnavailable rounds down, and the peak and floor follow. You wrote the prediction down and then measured it.
• Deployment status is a lagging summary. The API server omits zero-valued status fields entirely, and a one-second sampler can miss a short dip that a client sees.
• Readiness is the only gate. The Deployment controller and the EndpointSlice controller consume the same signal, which is why a missing readiness probe breaks capacity planning and traffic routing at the same time.
• maxUnavailable: 0 without a readiness probe is a promise kept against a meaningless signal — you paid for surge capacity and received nothing.
• A rollout can fail while the service stays up. Progressing False with Available True is a change that did not land, not an outage, and the two deserve different pagers.
• Hold is a real option with an owner and an end time. Roll forward, roll back and hold are three decisions; the failure is not choosing.
• An undo is a roll forward. Revision numbers only increase, and the history after an undo is longer than before it.