Lab 4: Debug the Pod lifecycle

Objective

By the end of this lab you will have built seven Pods that each stop at a different point in the lifecycle, identified every one of them from the API alone, and measured a termination grace period against the wall clock on a container that ignores SIGTERM.

The artefact that matters is not the Pods. It is the note you write in Task 8: for each failure, the one field that would have told you fastest. Two of these failures produce no logs at all, and one produces perfect logs while the Pod serves no traffic. Reaching for kubectl logs first is the habit this lab is designed to break.

Architecture

Everything the lab creates lives inside a single namespace. Nothing touches a node, the control plane, or any existing workload.

kubeadm cluster (1.34.x)
  cp-1        control plane
  worker-1    schedulable
  worker-2    schedulable   (one worker is enough; two makes the events clearer)

namespace podlab
  unschedulable   phase Pending     nodeSelector no node carries
  bad-image       phase Pending     image tag that does not exist
  crashloop       phase Running     exits 1, restartPolicy: Always
  crash-once      phase Failed      exits 1, restartPolicy: Never
  notready        phase Running     healthy nginx, readiness probe on a 404
  liveness-trap   phase Running     healthy nginx, liveness probe on a 404
  stubborn        termination       ignores SIGTERM, grace 20s
  polite          termination       exits on SIGTERM, grace 20s
  draining        termination       preStop sleep 10, grace 30s
  overrun         termination       preStop sleep 25, grace 10s

workstation
  ~/k8s-podlab/manifests/   the Pods
  ~/k8s-podlab/evidence/    what each one told you

Requirements

• A kubeadm cluster on Kubernetes 1.34.x with at least one schedulable worker node. B-nested is sufficient; A-physical works identically.
• kubectl 1.34.x, with a context that has permission to create and delete a namespace. The lab creates nothing outside that namespace.
• Outbound access from the nodes to docker.io for busybox:1.36 and nginx:1.27.2. Both are small; the whole lab pulls under 100 MB.
• Roughly 200 MiB of memory and 0.2 CPU of headroom across the cluster. The Pods are deliberately tiny.
• Two terminals for Task 7. One follows a log stream while the other deletes the Pod, because the evidence disappears with the Pod.
• No out-of-band access requirement. The lab reconfigures no networking, no SSH and no firewall; nothing in it can lock you out of a node.

Scenario

It is 03:00 and the page says the checkout service is down. kubectl get pods shows one Pod that is not in a state you like. You have perhaps ninety seconds before somebody asks you what is wrong.

The reflex is kubectl logs. For a Pod whose image never pulled there are no logs, because no container was ever created. For a Pod that the scheduler never placed there is not even a container status. For a Pod failing its readiness probe the logs are clean, the process is healthy, and the Service is still routing to nobody.

The field order — phase, then container state, then conditions, then events, then previous logs — is what turns those ninety seconds into an answer. This lab builds each failure so that you can practise the order on a cluster where being wrong costs nothing.

Tasks

Task 1: Capture the starting state and build the workspace

Record what the cluster looked like before you touched it. This is the same discipline a production change requires, and here it also gives you the node names the events will refer to.

# Substitute your own value if podlab is taken on this cluster:
NS=podlab
WORKDIR="$HOME/k8s-podlab"

mkdir -p "$WORKDIR/manifests" "$WORKDIR/evidence"
cd "$WORKDIR"

kubectl version -o yaml > evidence/00-versions.yaml
kubectl get nodes -o wide > evidence/00-nodes.txt
kubectl get namespace "$NS" > evidence/00-namespace-before.txt 2>&1

kubectl create namespace "$NS"

$ kubectl get nodes -o wide

NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP
cp-1       Ready    control-plane   14d   v1.34.0   192.0.2.11
worker-1   Ready    none            14d   v1.34.0   192.0.2.12
worker-2   Ready    none            14d   v1.34.0   192.0.2.13

The 00-namespace-before.txt capture is the one people skip. If the file says Error from server (NotFound), Cleanup can delete the namespace outright. If it says anything else, the namespace was already there and deleting it in Cleanup would destroy somebody else’s work.

Now write the evidence helper. Every task calls it; it collects the four fields in the order this lab argues for, so the file you end up with is a record of the method as well as the answer.

capture() {
  pod="$1"
  {
    echo "=== phase ==="
    kubectl -n "$NS" get pod "$pod" -o jsonpath='{.status.phase}{"\n"}'
    echo "=== container statuses (name ready restarts state) ==="
    kubectl -n "$NS" get pod "$pod" \
      -o jsonpath='{range .status.containerStatuses[*]}{.name}{"  "}{.ready}{"  "}{.restartCount}{"  "}{.state}{"\n"}{end}'
    echo "=== conditions ==="
    kubectl -n "$NS" get pod "$pod" \
      -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"  "}{.reason}{"\n"}{end}'
    echo "=== events ==="
    kubectl -n "$NS" get events --field-selector "involvedObject.name=$pod" \
      --sort-by=.lastTimestamp
  } > "evidence/$pod.txt" 2>&1
  echo "wrote evidence/$pod.txt"
}

Paste that into the shell you will run the rest of the lab in. It is a shell function, not a file — if you open a new terminal, paste it again.

Task 2: Pending, because the scheduler never chose a node

manifests/01-unschedulable.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: unschedulable
  namespace: podlab
spec:
  nodeSelector:
    lab.runbook.academy/tier: "nonexistent"
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      resources:
        requests:
          cpu: 10m
          memory: 16Mi

$ kubectl apply -f manifests/01-unschedulable.yaml

Wait ten seconds, then collect:

capture unschedulable
cat evidence/unschedulable.txt

Three observations, in the order the helper wrote them.

The phase is Pending and will stay Pending. The container statuses section is empty — not “waiting”, empty. No container status exists because no container was ever created; the kubelet has never seen this Pod, since no kubelet owns it. That emptiness is the single strongest discriminator in the whole lab, and Task 3 is why.

The conditions show PodScheduled=False with reason Unschedulable. The events carry a FailedScheduling message that names the count of nodes and the predicate that rejected them.

$ kubectl -n podlab get events --field-selector involvedObject.name=unschedulable

LAST SEEN   TYPE      REASON             OBJECT              MESSAGE
32s         Warning   FailedScheduling   pod/unschedulable   0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) didn't match Pod's node affinity/selector.

Now prove the negative that costs people the most time at 03:00:

kubectl -n "$NS" logs unschedulable || true

The API server refuses, and the refusal is informative rather than a stack trace. There is no container to read logs from. Anyone who opened this incident with kubectl logs has just spent their first thirty seconds learning nothing.

Task 3: Pending, because the image never pulled

Same phase. Opposite cause.

manifests/02-bad-image.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: bad-image
  namespace: podlab
spec:
  containers:
    - name: app
      image: busybox:1.36-no-such-tag
      command: ["sh", "-c", "sleep 3600"]
      resources:
        requests:
          cpu: 10m
          memory: 16Mi

$ kubectl apply -f manifests/02-bad-image.yaml

Give the kubelet about a minute — the first attempt fails fast, and you want to see the transition to backoff, not just the first error.

capture bad-image
cat evidence/bad-image.txt

The phase is Pending, exactly as in Task 2. Everything else differs:

• containerStatuses exists, with ready false, restartCount 0 and a state of map[waiting:map[message:... reason:ImagePullBackOff]].
• PodScheduled=True. A node was chosen; the node is doing its job.
• The events show Pulling, then Failed with the registry’s own message, then Failed with Error: ErrImagePull, then BackOff.

ErrImagePull is the first failure; ImagePullBackOff is what the reason becomes once the kubelet starts spacing out the retries. Seeing ImagePullBackOff therefore tells you the kubelet has already tried several times — it is a statement about elapsed time, not a different fault.

Task 4: Running, and dying repeatedly

manifests/03-crashloop.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: crashloop
  namespace: podlab
spec:
  restartPolicy: Always
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c"]
      args:
        - |
          echo "app starting, build 1.4.2"
          echo "FATAL: cannot open /etc/app/config.yaml: No such file or directory" >&2
          exit 1
      resources:
        requests:
          cpu: 10m
          memory: 16Mi

$ kubectl apply -f manifests/03-crashloop.yaml

Watch it for three minutes rather than sampling it once. The interesting part is not that it fails, it is the shape of the failure over time.

kubectl -n "$NS" get pod crashloop -w

Interrupt the watch after three minutes, then collect the evidence and the logs that actually exist:

capture crashloop
kubectl -n "$NS" logs crashloop --previous > evidence/crashloop-previous.log
cat evidence/crashloop-previous.log

The phase is Running. That is not a bug in Kubernetes and it is not a lie: the Pod is bound, the kubelet owns it, and the kubelet is actively working on it. It is simply not a health signal, which is why an alert written against status.phase misses this entirely.

The real state is one level down: containerStatuses[0].ready is false, restartCount climbs, and state is map[waiting:map[reason:CrashLoopBackOff]]. The most useful field is the one next to it — lastState.terminated, carrying exitCode: 1 and reason: Error. That is the previous container instance’s death certificate.

kubectl -n "$NS" get pod crashloop \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated}{"\n"}'

Compare your -w output against the backoff series the lesson describes: 10s, 20s, 40s, 80s, 160s, capped at five minutes. Your gaps between restarts should approximate it. This matters operationally: a Pod that has been crashing for an hour restarts only every five minutes, so a fix you deploy may sit for five minutes before it is even attempted. kubectl delete pod clears the backoff by replacing the Pod, which is why it feels like it “fixed” things.

Task 5: Failed, from exactly the same container

manifests/04-crash-once.yaml is byte-for-byte the previous manifest with two changes — the name, and restartPolicy: Never:

apiVersion: v1
kind: Pod
metadata:
  name: crash-once
  namespace: podlab
spec:
  restartPolicy: Never
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c"]
      args:
        - |
          echo "app starting, build 1.4.2"
          echo "FATAL: cannot open /etc/app/config.yaml: No such file or directory" >&2
          exit 1
      resources:
        requests:
          cpu: 10m
          memory: 16Mi

$ kubectl apply -f manifests/04-crash-once.yaml

sleep 20
capture crash-once
kubectl -n "$NS" get pods crashloop crash-once

The phase is Failed. The container state is terminated, not waiting, with exitCode: 1 and reason: Error. restartCount is 0 and will stay 0. There is no --previous instance, because there is only ever one instance — the current logs are the crash.

Two Pods, identical images, identical commands, identical exit codes, and two different phases. The phase is a function of restartPolicy, not of the application. Write that down: it is the reason a fleet-wide query for status.phase!=Running finds every failed Job and none of the crash-looping Deployments, and the reason the opposite query has the opposite blind spot.

Task 6: Running, healthy, and serving nobody

manifests/05-notready.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: notready
  namespace: podlab
  labels:
    app: notready
spec:
  containers:
    - name: web
      image: nginx:1.27.2
      ports:
        - containerPort: 80
      readinessProbe:
        httpGet:
          path: /healthz
          port: 80
        initialDelaySeconds: 2
        periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: notready
  namespace: podlab
spec:
  selector:
    app: notready
  ports:
    - port: 80
      targetPort: 80

The stock nginx image serves no /healthz, so the probe gets a 404. An httpGet probe counts 200-399 as success; 404 is a failure.

$ kubectl apply -f manifests/05-notready.yaml

sleep 30
capture notready
kubectl -n "$NS" get endpointslices -l kubernetes.io/service-name=notready -o wide

The phase is Running. containerStatuses[0].state is running — the process is genuinely up, and kubectl exec into it would find nginx serving. But ready is false, ContainersReady=False, Ready=False, and restartCount is 0 and stays 0. A failing readiness probe never restarts anything.

The consequence is in the EndpointSlice: the Pod’s address is either absent or carries ready: false, so the Service has no ready endpoint and traffic to it fails. Prove it from inside the cluster rather than believing the object:

kubectl -n "$NS" run probe-check --rm -it --restart=Never \
  --image=busybox:1.36 -- \
  wget -q -T 3 -O - http://notready.podlab.svc.cluster.local/ || true

Now the same mistake wired to the other probe. manifests/06-liveness-trap.yaml is the same nginx with readinessProbe replaced by livenessProbe:

apiVersion: v1
kind: Pod
metadata:
  name: liveness-trap
  namespace: podlab
spec:
  containers:
    - name: web
      image: nginx:1.27.2
      livenessProbe:
        httpGet:
          path: /healthz
          port: 80
        initialDelaySeconds: 5
        periodSeconds: 5
        failureThreshold: 2

$ kubectl apply -f manifests/06-liveness-trap.yaml

sleep 90
capture liveness-trap
kubectl -n "$NS" get pod liveness-trap

restartCount is climbing, and the events carry Liveness probe failed: HTTP probe failed with statuscode: 404 followed by Killing. The process was never unhealthy. The probe was wrong, and the probe killed it — repeatedly, and eventually into CrashLoopBackOff, at which point the Pod looks exactly like Task 4 and the logs show a perfectly normal nginx start every time.

Task 7: Measure the termination sequence

Four Pods, four grace budgets. manifests/07-termination.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: stubborn
  namespace: podlab
spec:
  terminationGracePeriodSeconds: 20
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "trap '' TERM; echo ignoring SIGTERM; while true; do sleep 1; done"]
---
apiVersion: v1
kind: Pod
metadata:
  name: polite
  namespace: podlab
spec:
  terminationGracePeriodSeconds: 20
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "trap 'echo SIGTERM received, draining; exit 0' TERM; echo ready; while true; do sleep 1; done"]
---
apiVersion: v1
kind: Pod
metadata:
  name: draining
  namespace: podlab
spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "trap 'echo SIGTERM received, draining; exit 0' TERM; echo ready; while true; do sleep 1; done"]
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 10"]
---
apiVersion: v1
kind: Pod
metadata:
  name: overrun
  namespace: podlab
spec:
  terminationGracePeriodSeconds: 10
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "trap 'echo SIGTERM received, draining; exit 0' TERM; echo ready; while true; do sleep 1; done"]
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 25"]

$ kubectl apply -f manifests/07-termination.yaml

Wait for all four to reach Running. Then, in a second terminal, follow the polite Pod’s log stream — this is the only chance you get to see the signal arrive, because the evidence is deleted along with the Pod:

kubectl -n podlab logs -f polite

Back in the first terminal, time each delete. time is the whole measurement; the deletion blocks until the Pod is gone.

$ time kubectl -n podlab delete pod polite

{
  echo "pod        grace  preStop  wall-clock"
  for p in polite stubborn draining overrun; do
    start=$(date +%s)
    kubectl -n "$NS" delete pod "$p" > /dev/null
    echo "$p $(( $(date +%s) - start ))s"
  done
} | tee evidence/termination.txt

What you should see, and why:

• polite goes in about one to two seconds. The trap fires between sleep iterations, the shell exits 0, and the kubelet stops waiting. The follow stream in the second terminal printed SIGTERM received, draining — that is your proof the signal was delivered and handled, not merely that the Pod vanished.
• stubborn takes about the full 20 seconds. SIGTERM was delivered and ignored; the kubelet waited out terminationGracePeriodSeconds and then sent SIGKILL, which cannot be ignored. Twenty seconds is not a delay, it is the budget you configured being spent.
• draining takes about 10 seconds despite a 30-second budget. The preStop hook ran first, slept 10, and only then did SIGTERM go to the container, which exited immediately. This is the shape of a real drain: hold the container alive and serving while the endpoint is withdrawn, then shut down.
• overrun finishes at roughly its 10-second grace period, not at the hook’s 25 seconds. The hook never completed. The grace period is the total budget: preStop runs inside it, not before it.

Task 8: Write down the triage order

This is the deliverable. Create evidence/triage.md and fill in, from your own evidence files, the one field that identified each failure fastest:

unschedulable   status.containerStatuses absent            -> placement problem
bad-image       state.waiting.reason = ImagePullBackOff    -> registry/manifest problem
crashloop       lastState.terminated.exitCode + --previous -> application problem
crash-once      status.phase = Failed                      -> terminal, restartPolicy: Never
notready        containerStatuses[0].ready = false, rc 0    -> probe or dependency problem
liveness-trap   event "Liveness probe failed" before Killing -> probe configuration problem

Then answer, in a sentence each: which two of these produce no useful kubectl logs output at all, and which one produces clean logs while the Service serves nothing? If you cannot answer from your own files rather than from this page, redo the capture for that Pod.

Validation

Run these against your own evidence rather than against the cluster; the point is that the files support the conclusions.

cd "$HOME/k8s-podlab"

grep -c . evidence/triage.md
grep -l "Unschedulable" evidence/unschedulable.txt
grep -l "ImagePullBackOff" evidence/bad-image.txt
grep -l "CrashLoopBackOff" evidence/crashloop.txt
grep -l "Failed" evidence/crash-once.txt
cat evidence/termination.txt

The lab succeeded when all of the following hold:

• evidence/unschedulable.txt has an empty container-statuses section and PodScheduled=False.
• evidence/bad-image.txt has a non-empty container-statuses section and PodScheduled=True — the same phase as above, the opposite diagnosis.
• evidence/crashloop.txt shows phase Running with a non-zero restartCount, and evidence/crashloop-previous.log contains the FATAL line.
• evidence/crash-once.txt shows phase Failed with restartCount 0.
• evidence/notready.txt shows a running container with ready false and restartCount 0, and the EndpointSlice had no ready address.
• evidence/liveness-trap.txt shows a rising restartCount and a Liveness probe failed event.
• evidence/termination.txt shows polite at roughly 1-2s, stubborn at roughly its 20s grace period, draining at roughly 10s, and overrun cut off at roughly 10s rather than 25s.

Expected Outcome

~/k8s-podlab/
├── manifests/
│   ├── 01-unschedulable.yaml
│   ├── 02-bad-image.yaml
│   ├── 03-crashloop.yaml
│   ├── 04-crash-once.yaml
│   ├── 05-notready.yaml
│   ├── 06-liveness-trap.yaml
│   └── 07-termination.yaml
└── evidence/
    ├── 00-versions.yaml, 00-nodes.txt, 00-namespace-before.txt
    ├── unschedulable.txt, bad-image.txt
    ├── crashloop.txt, crashloop-previous.log, crash-once.txt
    ├── notready.txt, liveness-trap.txt
    ├── termination.txt
    └── triage.md

A set of files in which each of six failure modes is distinguished by a named field, and a termination table in which a configured number and a measured number agree. That pairing — a setting and the wall-clock consequence of it — is what makes terminationGracePeriodSeconds a decision rather than a default nobody has ever tested.

Troubleshooting

capture says “command not found”. It is a shell function defined in Task 1, and it lives only in the shell you pasted it into. Paste it again, and check that NS is still set in this shell too.

bad-image shows ErrImagePull and never becomes ImagePullBackOff. You looked too early. The reason changes only once the kubelet starts spacing the retries. Wait a minute and re-capture.

crashloop shows restartCount: 0 and phase Failed. You applied 04-crash-once.yaml under the wrong name, or the manifest lost its restartPolicy: Always. The two manifests differ in exactly those two lines; diff them.

kubectl logs crashloop --previous says “previous terminated container not found”. The Pod has not restarted yet, so there is no previous instance. Wait for restartCount to reach at least 1.

notready reports Ready=True. Something is serving /healthz — you used an image other than stock nginx:1.27.2, or a proxy is answering. Confirm with kubectl -n podlab exec notready -- wget -q -S -O /dev/null http://127.0.0.1/healthz, which should report a 404.

The EndpointSlice for notready does not exist at all. The Service selector and the Pod labels must both be app: notready. A Service with a selector that matches nothing produces no EndpointSlice rather than an empty one — a distinct failure covered in the Services labs.

stubborn is deleted instantly. Something passed --force --grace-period=0, or your shell history helpfully supplied it. That flag skips the grace period entirely and asks the API server to drop the record; it is the one deletion that genuinely does not wait, and it is why it should never be a reflex.

The termination loop reports 0s for everything. date +%s has one-second resolution and the Pods were already terminating from an earlier attempt. Re-apply 07-termination.yaml, wait for all four to be Running, and rerun.

Cleanup

The lab created exactly one namespace and one working directory. Both go, and both are verified rather than assumed.

Step 1. Confirm what you are about to remove, and that it is only lab objects:

NS=podlab

kubectl -n "$NS" get all
cat "$HOME/k8s-podlab/evidence/00-namespace-before.txt"

Step 2. Delete the namespace and wait for it to actually go:

$ kubectl delete namespace podlab --wait=true

Step 3. Verify the cluster is as you found it:

NS=podlab

kubectl get namespace "$NS" || echo "namespace gone, as expected"
kubectl get nodes -o wide
diff <(kubectl get nodes -o wide) "$HOME/k8s-podlab/evidence/00-nodes.txt" \
  && echo "node inventory unchanged"

Step 4. Keep the deliverables, then remove the working directory.

ls -la "$HOME/k8s-podlab"

mkdir -p "$HOME/k8s-lab-deliverables/pod-lifecycle"
cp -a "$HOME/k8s-podlab/evidence" "$HOME/k8s-podlab/manifests" \
      "$HOME/k8s-lab-deliverables/pod-lifecycle/"

rm -rf "$HOME/k8s-podlab"

What You Learned

• Two Pods can share a phase and share nothing else. Pending with no container status is a placement problem; Pending with ImagePullBackOff is a registry problem. One command separates them and it is not kubectl logs.
• Running is a statement about the kubelet’s intent, not about health. A Pod in CrashLoopBackOff is Running, and an alert written against status.phase will never see it.
• The phase is a function of restartPolicy. You ran the same failing container twice and got Running and Failed. Neither phase told you anything about the application.
• --previous is where the crash is. The current logs belong to a container that has not failed yet.
• Readiness removes a Pod from service; liveness destroys it. The same wrong path produced an invisible outage in one case and a restart loop with clean logs in the other.
• terminationGracePeriodSeconds is a budget that preStop spends first. You measured 20 seconds of ignored SIGTERM, a 10-second hook inside a 30-second budget, and a 25-second hook cut off by a 10-second budget.
• The evidence order is the skill. Phase, container state, conditions, events, previous logs — and the file you wrote in Task 8 is the version of that order you will actually remember at 03:00.