Lab 9: Diagnose an OOMKilled container

Objective

By the end of this lab you will have produced a cgroup OOMKill on demand, produced an exit code 137 that is not an OOMKill, and written down the one field that tells them apart. You will also have sampled memory.current against memory.max in the seconds before a kill, so the event has an approach and not only a corpse.

The deliverable is the discriminator table in Task 8. Exit code 137 has at least two causes, a Pod can report Running while a container inside it dies every ninety seconds, and a node-pressure eviction leaves a completely different set of objects behind than an OOMKill does. All three arrive in the channel as “the pod keeps dying”.

Architecture

Everything lives in one namespace. Nothing touches a node or the control plane.

kubeadm cluster (1.34.x)
  cp-1        control plane
  worker-1    schedulable
  worker-2    schedulable

namespace oomlab
  oom-victim      memory limit 100Mi, allocates 250M   -> OOMKilled, restarts
  oom-survivor    memory limit 256Mi, tmpfs emptyDir    -> lives, watched
  fake-137        ignores SIGTERM, failing liveness    -> exit 137, not OOM
  running-liar    2 containers; the sidecar OOMKills   -> Pod stays Running

workstation
  ~/k8s-oomlab/manifests/   the Pods
  ~/k8s-oomlab/evidence/    what each one told you

Requirements

• A kubeadm cluster on Kubernetes 1.34.x with at least one schedulable worker. B-nested is sufficient.
• kubectl 1.34.x, with a context allowed to create and delete a namespace, and to read /api/v1/nodes/NODE/proxy/configz for Task 1. If your RBAC forbids the node proxy, Task 1 has a fallback.
• cgroups v2 on the nodes, for the memory.current reads in Task 3.
• Outbound access from the nodes to docker.io for busybox:1.36 and polinux/stress. The second is the image upstream’s own memory-limit task uses; it is a few megabytes.
• Roughly 600 MiB of memory headroom across the cluster. Every Pod here has a memory limit, and the largest is 256Mi.
• No metrics-server required. Every number comes from the API or from a cgroup file inside a container.
• No out-of-band access requirement. Nothing reconfigures networking, SSH or the firewall, and no task deliberately pressures a node.

Scenario

The alert says CrashLoopBackOff on the checkout service. You run kubectl logs, and get a clean startup banner from a container that started four seconds ago. You run it again a minute later and get the same banner.

Nothing in the logs is wrong, because the process that failed did not fail — it was executed. SIGKILL is not deliverable to a handler, so there is no shutdown path, no flush, no final line. Whatever the application was going to say about its own death, it did not get the chance to say.

The evidence lives on the Pod object instead, in a field most people have never read: the previous container instance’s termination record. This lab builds four situations that all arrive as “the pod keeps dying” and separates them using that field.

Tasks

Task 1: Baseline, and your node’s real eviction thresholds

# Substitute your own value if oomlab is taken on this cluster:
NS=oomlab
WORKDIR="$HOME/k8s-oomlab"

mkdir -p "$WORKDIR/manifests" "$WORKDIR/evidence"
cd "$WORKDIR"

kubectl version -o yaml > evidence/00-versions.yaml
kubectl get nodes -o wide > evidence/00-nodes.txt
kubectl get namespace "$NS" > evidence/00-namespace-before.txt 2>&1

kubectl create namespace "$NS"

If evidence/00-namespace-before.txt says Error from server (NotFound), Cleanup may delete the namespace. Anything else means it predates the lab.

Now read the thresholds that decide when your kubelet starts evicting. Most people quote the documented default; read yours:

# Substitute a schedulable worker from evidence/00-nodes.txt:
NODE=worker-1

kubectl get --raw "/api/v1/nodes/$NODE/proxy/configz" \
  | tee evidence/01-kubeletconfig.json \
  | tr ',' '\n' | grep -iE 'eviction|memory' \
  | tee evidence/01-eviction-thresholds.txt

kubectl get node "$NODE" \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}' \
  | tee evidence/01-node-conditions.txt

You are looking for evictionHard and its memory.available entry, and for MemoryPressure=False in the conditions. Write both into evidence/thresholds.md with the date. That threshold is the line between the failure this lab produces — a container killed inside its own cgroup — and the one it deliberately does not: the kubelet reclaiming the node by evicting Pods.

Task 2: Produce the OOMKill and read the evidence chain

manifests/01-oom-victim.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: oom-victim
  namespace: oomlab
spec:
  containers:
    - name: app
      image: polinux/stress
      command: ["stress"]
      args: ["--vm", "1", "--vm-bytes", "250M", "--vm-hang", "1"]
      resources:
        requests:
          cpu: 50m
          memory: 100Mi
        limits:
          cpu: 200m
          memory: 100Mi

The container asks the kernel for 250 MB inside a 100Mi ceiling. There is no configuration that lets that succeed.

$ kubectl apply -f manifests/01-oom-victim.yaml

Watch it for two minutes rather than sampling once. The shape over time is the diagnosis:

kubectl -n "$NS" get pod oom-victim -w

Interrupt after two minutes and collect the chain, in the order a triage should read it:

capture() {
  pod="$1"; tag="${2:-$1}"
  {
    echo "=== phase ==="
    kubectl -n "$NS" get pod "$pod" -o jsonpath='{.status.phase}{"\n"}'
    echo "=== container: name ready restarts ==="
    kubectl -n "$NS" get pod "$pod" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"  "}{.ready}{"  "}{.restartCount}{"\n"}{end}'
    echo "=== current state ==="
    kubectl -n "$NS" get pod "$pod" -o jsonpath='{range .status.containerStatuses[*]}{.name}{": "}{.state}{"\n"}{end}'
    echo "=== lastState.terminated ==="
    kubectl -n "$NS" get pod "$pod" -o jsonpath='{range .status.containerStatuses[*]}{.name}{": "}{.lastState.terminated}{"\n"}{end}'
    echo "=== events ==="
    kubectl -n "$NS" get events --field-selector "involvedObject.name=$pod" --sort-by=.lastTimestamp
  } > "evidence/$tag.txt" 2>&1
  echo "wrote evidence/$tag.txt"
}

capture oom-victim
cat evidence/oom-victim.txt

Paste that function into the shell you will use for the rest of the lab; it is a shell function, not a file.

Four observations.

The phase is Running. It is not a lie and it is not a bug: the Pod is bound, a kubelet owns it, and that kubelet is actively working on it. The default restartPolicy for a Pod is Always, so as long as the kubelet intends to keep restarting the container, the Pod is Running. An alert written against status.phase will never fire for this.

The container is not ready and restartCount is climbing. Current state alternates between running and waiting with reason CrashLoopBackOff as the kubelet spaces out the retries.

lastState.terminated is the answer, and it is the field this lab exists for:

$ kubectl -n oomlab get pod oom-victim -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

map[containerID:containerd://... exitCode:137 finishedAt:... reason:OOMKilled startedAt:...]

exitCode: 137 is 128 plus signal 9, SIGKILL. reason: OOMKilled is the part that matters, and Task 5 is about why.

The events show Started, then BackOff and Failed with Back-off restarting failed container. Note what is not there: no message naming memory, no message naming the limit. The events tell you it keeps failing; only lastState tells you how.

Now spend thirty seconds proving that the reflex is worthless here:

kubectl -n "$NS" logs oom-victim | tee evidence/oom-victim-current.log
kubectl -n "$NS" logs oom-victim --previous | tee evidence/oom-victim-previous.log
wc -c evidence/oom-victim-current.log evidence/oom-victim-previous.log

Task 3: Watch the approach, not just the aftermath

An OOMKill has a run-up, and the run-up is visible from inside the container. Build one that survives so you can watch it.

manifests/02-oom-survivor.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: oom-survivor
  namespace: oomlab
spec:
  volumes:
    - name: cache
      emptyDir:
        medium: Memory
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: cache
          mountPath: /cache
      resources:
        requests:
          cpu: 50m
          memory: 256Mi
        limits:
          cpu: 200m
          memory: 256Mi

medium: Memory makes /cache a tmpfs. Nothing written there ever reaches a disk, and — this is the part that surprises people — every page of it is charged to the container’s memory cgroup.

$ kubectl apply -f manifests/02-oom-survivor.yaml

kubectl -n "$NS" wait --for=condition=Ready pod/oom-survivor --timeout=120s

trace() {
  kubectl -n "$NS" exec oom-survivor -- sh -c \
    'printf "%s max=%s current=%s\n" "$1" "$(cat /sys/fs/cgroup/memory.max)" "$(cat /sys/fs/cgroup/memory.current)"' \
    sh "$1"
}

trace baseline                                        | tee    evidence/03-memory-trace.txt
kubectl -n "$NS" exec oom-survivor -- \
  dd if=/dev/zero of=/cache/block bs=1M count=100
trace after-100Mi-write                               | tee -a evidence/03-memory-trace.txt
kubectl -n "$NS" exec oom-survivor -- rm -f /cache/block
trace after-delete                                    | tee -a evidence/03-memory-trace.txt

cat evidence/03-memory-trace.txt

memory.max is the byte form of your limit: 256Mi is 268435456, constant throughout. memory.current starts at a few mebibytes, jumps by roughly 100Mi after the write, and falls back when the file is removed.

The application allocated nothing. Its heap did not move. A hundred mebibytes of its memory limit went to a file, and a heap profiler taken at that moment would have reported a container comfortably inside budget while it sat 100Mi closer to being killed.

Compute the headroom — max minus the peak current — and write it into the trace file. That number, tracked over hours rather than three samples, is what a right-sizing exercise is actually about: not “did it die”, but “how close did it get, and how often”.

Task 4: An exit code 137 that is not an OOMKill

manifests/03-fake-137.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: fake-137
  namespace: oomlab
spec:
  terminationGracePeriodSeconds: 5
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "trap '' TERM; echo started, ignoring SIGTERM; while true; do sleep 1; done"]
      livenessProbe:
        exec:
          command: ["false"]
        initialDelaySeconds: 10
        periodSeconds: 10
        failureThreshold: 1
      resources:
        requests:
          cpu: 10m
          memory: 32Mi
        limits:
          cpu: 100m
          memory: 64Mi

This container uses almost no memory. Its liveness probe always fails, so the kubelet kills it; it ignores SIGTERM, so after the five-second grace period the kubelet sends SIGKILL.

$ kubectl apply -f manifests/03-fake-137.yaml

sleep 90
capture fake-137

echo "--- fake-137 ---"; grep -A 1 "lastState" evidence/fake-137.txt
echo "--- oom-victim ---"; grep -A 1 "lastState" evidence/oom-victim.txt

Both terminated records carry exitCode: 137. Both containers restart. Both Pods report Running. The reason fields differ, and that is the whole discriminator: the OOMKilled container says OOMKilled, and this one does not.

The events differ too, and they corroborate: fake-137 carries Liveness probe failed immediately before Killing, on a container whose own output is a perfectly ordinary startup line.

Task 5: The Pod that reports Running while a container dies

manifests/04-running-liar.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: running-liar
  namespace: oomlab
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "while true; do echo app healthy; sleep 30; done"]
      resources:
        requests: {cpu: 10m, memory: 32Mi}
        limits:   {cpu: 100m, memory: 64Mi}
    - name: exporter
      image: polinux/stress
      command: ["stress"]
      args: ["--vm", "1", "--vm-bytes", "200M", "--vm-hang", "1"]
      resources:
        requests: {cpu: 10m, memory: 64Mi}
        limits:   {cpu: 100m, memory: 64Mi}

$ kubectl apply -f manifests/04-running-liar.yaml

sleep 120
capture running-liar
kubectl -n "$NS" get pod running-liar
kubectl -n "$NS" logs running-liar -c app --tail=3

kubectl get pod shows STATUS: Running and READY: 1/2. The application container is genuinely healthy and logging. The sidecar is being OOMKilled roughly every ninety seconds, and its restartCount is the only number moving.

The READY column is the tell, and it is the column people read as a progress indicator rather than as a fault. A dashboard that reports Pod phase sees nothing wrong here. A dashboard that reports ready replicas sees a Deployment permanently one short and never says why.

kubectl -n "$NS" get pod running-liar \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{" ready="}{.ready}{" restarts="}{.restartCount}{"\n"}{end}'

Task 6: Tell the three mechanisms apart from the objects

An OOMKilled container and an evicted Pod leave completely different debris. Run the queries on your own cluster and record what each returns:

{
  echo "=== live Pods with restarting containers (container-level kills) ==="
  kubectl -n "$NS" get pods \
    -o jsonpath='{range .items[*]}{.metadata.name}{" phase="}{.status.phase}{" restarts="}{.status.containerStatuses[*].restartCount}{"\n"}{end}'

  echo "=== Failed Pods across the cluster (node-level evictions live here) ==="
  kubectl get pods -A --field-selector status.phase=Failed

  echo "=== eviction events across the cluster ==="
  kubectl get events -A --field-selector reason=Evicted
} 2>&1 | tee evidence/06-discriminators.txt

On a healthy lab cluster the last two are empty, and that emptiness is the point: everything this lab produced is in the first query and nothing is in the other two.

The structural differences, which is what Task 8 asks you to write down:

• A cgroup OOMKill leaves a live Pod. Phase Running, a rising restartCount, lastState.terminated.reason: OOMKilled. The Pod object is never replaced, so the evidence accumulates on it.
• A kubelet SIGKILL leaves the same shape with a different reason, and events naming a probe or a termination rather than memory.
• A node-pressure eviction leaves a dead Pod. The kubelet marks it phase Failed with reason Evicted and stops running it there; a controller creates a different Pod elsewhere. The evicted object stays until something garbage-collects it, which is why a node that had a bad night is surrounded by Failed Pods the following morning.

Task 7: Fix it, and name what the fix costs

Raise the victim’s limit until it fits, and confirm the kills stop:

$ kubectl -n oomlab delete pod oom-victim --wait=true

sed 's/memory: 100Mi/memory: 400Mi/' manifests/01-oom-victim.yaml \
  > manifests/05-oom-victim-fixed.yaml

kubectl apply -f manifests/05-oom-victim-fixed.yaml
kubectl -n "$NS" wait --for=condition=Ready pod/oom-victim --timeout=120s

sleep 90
capture oom-victim oom-victim-fixed
kubectl -n "$NS" get pod oom-victim

The sed moved the request as well as the limit, because both lines said 100Mi, so the Pod is still Guaranteed and now reserves 400Mi of scheduler budget on whichever node it lands on.

restartCount is 0 and stays 0. That is the correct fix for exactly one of the four causes the lessons name — a limit set below the workload’s real footprint — and it is indistinguishable, at this moment, from the wrong fix for the other three.

Task 8: Write the discriminator table

This is the deliverable. Create evidence/discriminators.md and fill it in from your own evidence files:

symptom                       field that identifies it            fix belongs to
container exceeded its limit  ____________________________        ____________
kubelet SIGKILLed it          ____________________________        ____________
node evicted the Pod          ____________________________        ____________
sidecar dying, app healthy    ____________________________        ____________

Exit code 137 appears in ____ of the four rows above.
kubectl logs --previous is useful in ____ of the four rows above.

My node's evictionHard memory.available threshold, read from the kubelet
on <date>: ____________

Then answer in a sentence each: which of your four Pods reported phase Running while broken, and what would you alert on instead of phase?

Validation

cd "$HOME/k8s-oomlab"

grep -c . evidence/discriminators.md
grep "reason:OOMKilled" evidence/oom-victim.txt
grep "exitCode:137" evidence/fake-137.txt
grep -c "OOMKilled" evidence/fake-137.txt || echo "0 — as expected"
head -3 evidence/03-memory-trace.txt
wc -c evidence/oom-victim-previous.log
cat evidence/06-discriminators.txt

The lab succeeded when all of the following hold:

• evidence/oom-victim.txt shows phase Running, a non-zero restartCount, and lastState.terminated carrying both exitCode:137 and reason:OOMKilled.
• evidence/fake-137.txt shows exitCode:137 and a reason that is not OOMKilled, on a container limited to 64Mi that used almost none of it.
• evidence/oom-victim-previous.log is empty or near-empty, while evidence/oom-victim.txt contains the diagnosis.
• evidence/03-memory-trace.txt shows a constant max=268435456, a current that rises by roughly 100Mi across the write, and falls again after the delete — on a container whose process allocated nothing.
• evidence/running-liar.txt shows phase Running with one container ready and one with a rising restart count.
• evidence/06-discriminators.txt shows every lab failure in the live-Pod query and none in the Failed-Pod or Evicted-event queries.
• evidence/oom-victim-fixed.txt shows restartCount 0 after 90 seconds.
• evidence/discriminators.md records a threshold you read from your kubelet.

Expected Outcome

~/k8s-oomlab/
├── manifests/
│   ├── 01-oom-victim.yaml
│   ├── 02-oom-survivor.yaml
│   ├── 03-fake-137.yaml
│   ├── 04-running-liar.yaml
│   └── 05-oom-victim-fixed.yaml
└── evidence/
    ├── 00-versions.yaml, 00-nodes.txt, 00-namespace-before.txt
    ├── 01-kubeletconfig.json, 01-eviction-thresholds.txt, 01-node-conditions.txt
    ├── oom-victim.txt, oom-victim-current.log, oom-victim-previous.log
    ├── 03-memory-trace.txt
    ├── fake-137.txt, running-liar.txt
    ├── 06-discriminators.txt
    ├── oom-victim-fixed.txt
    ├── thresholds.md
    └── discriminators.md

Two termination records that share an exit code and disagree about the reason, a memory trace with a real ceiling in it, and a table that maps each symptom to the one field that identifies it.

Production notes

Triage order for “the pod keeps dying”. Phase, then per-container ready and restartCount, then lastState.terminated.reason, then events, then --previous logs. The reason field ends the question in most cases, and every step before it takes seconds.

What to alert on. Not status.phase, which says Running for three of the four failures in this lab. Alert on a rising restartCount, on ready replicas below desired, and on the OOMKilled reason specifically — a container that has been OOMKilled once is a capacity fact, and one that has been OOMKilled fifty times is an outage nobody has noticed.

Before raising a memory limit. Get a usage trace long enough to show whether the workload plateaus. Record the peak and the new limit in the change ticket, along with the effect on the Pod’s QoS class and on how many Pods now fit per node. “Raised because it was OOMKilled” is not a justification, it is a restatement of the symptom.

Evictions. Handle these as node capacity work, not as workload work. Read the node’s MemoryPressure condition and the eviction thresholds you captured in Task 1, and look at what was not evicted — the Guaranteed workload sitting at the top of the node is usually the cause and never the victim.

Change window mapping. Raising a limit on a Deployment is a rolling update and inherits every property of one: it needs surge capacity, it is gated by readiness, and it can be rolled back. Treat it as a deploy, not as a configuration tweak, and rehearse the rollback the way Lab 6 does.

Troubleshooting

capture says “command not found”. It is a shell function from Task 2 and lives only in the shell you pasted it into. Paste it again and set NS too.

polinux/stress will not pull. The Pod sits in ImagePullBackOff with containerStatuses present and PodScheduled=True — a registry problem, not a memory one, and worth noticing that the evidence chain says so. Check outbound access from the node, or any registry mirror or pull-through cache your cluster enforces.

kubectl get --raw .../configz is Forbidden. Your context cannot use the node proxy. Read the kubelet config from the node instead (/var/lib/kubelet/config.yaml) if you have host access, and record that you could not verify it if you do not. An unverified threshold recorded as unverified is worth more than the documented default recorded as fact.

oom-victim shows reason: Error instead of OOMKilled. Wait for at least one full restart cycle; the reason is set from the cgroup counter after the container exits, so a Pod captured mid-start has nothing to report yet. Confirm restartCount is at least 1 before capturing.

memory.current reads fail with “no such file”. The node is on cgroups v1, or the container has no private cgroup namespace. Locate the file with kubectl -n oomlab exec oom-survivor -- find /sys/fs/cgroup -maxdepth 3 -name memory.max.

dd into /cache fails with “No space left on device”. The tmpfs is sized from the Pod’s memory limits, so a 100Mi write into a 256Mi Pod should fit. If yours does not, reduce count to 50 and read the trace with that number instead; the shape is what matters.

fake-137 never restarts. The liveness probe has not fired yet — it has a ten-second initial delay and a ten-second period. Wait ninety seconds. If it still has not, confirm the probe is exec with command false and failureThreshold: 1.

running-liar shows 2/2 ready. The exporter fitted inside 64Mi on your node, which can happen if the image behaves differently than expected. Lower its limit to 32Mi and re-apply; the teaching point is the READY column, not the number.

Every Pod is Pending with Insufficient memory. The four Pods request about 500Mi in total. Delete the ones from tasks you have finished before starting the next, or run on a worker with more headroom.

Cleanup

The lab created exactly one namespace and one working directory.

Step 1. Confirm what you are about to remove, and that the evidence is on disk:

NS=oomlab

kubectl -n "$NS" get pods
ls -la "$HOME/k8s-oomlab/evidence"
cat "$HOME/k8s-oomlab/evidence/00-namespace-before.txt"

Step 2. Delete the namespace and wait for it to actually go:

$ kubectl delete namespace oomlab --wait=true

Step 3. Verify the cluster is as you found it. This lab put memory pressure on nothing, so the node conditions should be unchanged:

NS=oomlab
NODE=worker-1

kubectl get namespace "$NS" || echo "namespace gone, as expected"
kubectl get node "$NODE" \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'
diff <(kubectl get nodes -o wide) "$HOME/k8s-oomlab/evidence/00-nodes.txt" \
  && echo "node inventory unchanged"

MemoryPressure should read False, exactly as evidence/01-node-conditions.txt recorded at the start.

Step 4. Keep the deliverables, then remove the working directory.

ls -la "$HOME/k8s-oomlab"

mkdir -p "$HOME/k8s-lab-deliverables/oomkilled"
cp -a "$HOME/k8s-oomlab/evidence" "$HOME/k8s-oomlab/manifests" \
      "$HOME/k8s-lab-deliverables/oomkilled/"

rm -rf "$HOME/k8s-oomlab"

What You Learned

• lastState.terminated.reason is the diagnosis. Phase said Running, events said BackOff, logs said nothing, and one field said OOMKilled.
• An OOMKilled container cannot report its own death. SIGKILL runs no handler, so --previous is near-empty — and that emptiness is itself a narrowing signal.
• Exit code 137 has more than one sender. You built two containers with the same exit code and different reasons, one of which used almost no memory.
• A Pod reports Running while a container inside it dies repeatedly. The READY column, not the STATUS column, is where a multi-container Pod tells the truth.
• The limit constrains the cgroup, not the process. Page cache and memory-backed volumes are charged there too, which is why a heap profile can clear a container the kernel just killed.
• Container OOMKill and node eviction leave different debris. A live Pod with restarts versus a Failed Pod with reason Evicted — and only one of them is fixed by editing the victim.
• Raising the limit is a real fix and a real cover-up. Whether usage plateaus is the discriminator, and it needs a trace nobody took.