low riskservice affecting~25 min

Runbook: Investigate an OOMKilled Pod

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the container exited with reason OOMKilled: kubectl get pod <name> -n <ns> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
· Capture the exit code: kubectl get pod <name> -n <ns> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}' (137 for SIGKILL)
· Capture the previous container logs: kubectl logs <name> -n <ns> --previous --tail=200
· Capture the Pod spec requests/limits: kubectl get pod <name> -n <ns> -o jsonpath='{.spec.containers[0].resources}' | jq
· Capture node memory state at the time of the kill: kubectl describe node <node> | grep -E 'Memory|MemoryPressure'
· Capture the kernel OOM events from the node journal: kubectl get pod <name> -n <ns> -o jsonpath='{.spec.nodeName}' | xargs -I{} ssh {} sudo journalctl -k --since '10 min ago' | grep -i oom

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Classify the OOM: container-level (cgroup), node-level (kernel), or node-pressure eviction
2For container-level: compare spec.containers[*].resources.limits.memory to actual peak usage; the limit is too low or the workload has a leak
3Capture peak usage from the metrics server or Prometheus: kubectl top pod <name> -n <ns> --containers shows current, not peak
4For long-running measurement, query the historical series: kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/<ns>/pods/<name> is current only; use Prometheus or in-process metrics for peak
5Decide whether the limit is genuinely too low or whether the application has a memory leak (steady climb over time)
6For node-level OOM: read the node journal for Out of memory: Killed process; the cgroup of the killed process identifies which Pod
7For node-pressure eviction: read kubectl describe node <node> | grep -A5 "Conditions" for MemoryPressure=True; the kubelet evicted the Pod under pressure
8Apply the smallest fix: raise the limit (with measurement), fix the leak, move the Pod off a pressured node, increase node memory
9For a Deployment, kubectl rollout restart deploy/<name> -n <ns> re-creates the Pod
10For a one-off Pod, kubectl delete pod <name> -n <ns> lets the controller recreate it

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get pod <name> -n <ns> reports Running and a stable restart count
✓kubectl top pod <name> -n <ns> --containers reports current usage below the new limit (with at least 20% headroom)
✓kubectl describe node <node> | grep "MemoryPressure" reports False after the new Pod is scheduled
✓kubectl logs <name> -n <ns> shows the application starting cleanly
✓No further OOMKilled events for the same workload over a representative period
✓kubectl get events -n <ns> --field-selector involvedObject.name=<name> shows no new Warning events for 30 minutes

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If raising the limit causes the Pod to be scheduled on a node that is now under pressure, revert the limit and re-evaluate; the limit was not the cause
↶If the application has a confirmed leak and the workload cannot be patched in-window, scale to zero: kubectl scale deploy/<name> -n <ns> --replicas=0
↶If the OOM was node-pressure, do not raise the limit; the right fix is more node memory or fewer Pods per node
↶Capture the pre- and post-fix memory curve to the incident record; "OOMKilled" is a class, not a single cause
↶If the new limit is accepted but the next OOM cycle is shorter, the leak is real and a code change is required

6 · Escalation

When the runbook isn't enough, contact:

· Container OOMs with healthy node memory state and a generous limit: the application has a leak; escalate to application ownership with the heap/profile evidence
· Node-level OOMs that kill unrelated Pods on the same node: a workload exceeded its cgroup and escaped into the node; do not retry the same workload without identifying the escape path
· Memory pressure eviction on a node pool: scale out the node pool or reduce Pod density; escalate to capacity planning
· OOMKilled on system Pods (kube-proxy, CNI, etc.): a node-level issue; see kubernetes-rb-troubleshoot-node-notready
· Repeated OOMKilled after limit adjustment: the application has a confirmed leak; open a ticket with the heap dump and timeline

OOMKilled is the cgroup memory.high or memory.max firing on the container. It is not the kernel OOM killer (which leaves different evidence in the node journal) and it is not the kubelet’s memory- pressure eviction (which leaves different evidence in the Pod’s status). The runbook distinguishes them.

1. Confirm the OOM source

Read-only / SafeConfirm the OOM source


kubectl get pod <name> -n <ns> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# Expect: OOMKilled

# Exit code is 137 (128 + SIGKILL 9)
kubectl get pod <name> -n <ns> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

# Node-level OOM (the kernel killed a process that escaped the cgroup)
NODE=$(kubectl get pod <name> -n <ns> -o jsonpath='{.spec.nodeName}')
ssh "$NODE" -- sudo journalctl -k --since "30 min ago" | grep -E 'Out of memory|Killed process'

# Memory pressure eviction
kubectl describe pod <name> -n <ns> | grep -E 'Evicted|reason|message' | head

If OOMKilled is set, the cgroup fired. If Evicted is the reason in the Pod status, kubelet evicted under node pressure. If the node journal shows Killed process <pid> ..., the kernel killed something on the host — possibly the same workload, possibly another.

2. Capture the memory curve

Read-only / SafeCapture the memory curve


kubectl top pod <name> -n <ns> --containers

# The metrics API reports current usage only - it keeps no history
kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/<ns>/pods/<name> | jq

# The limit the cgroup is enforcing
kubectl get pod <name> -n <ns> -o jsonpath='{.spec.containers[*].resources.limits.memory}'

3. Choose the fix

Source	Real cause	Right fix
Container OOM, peak == limit, steady-state	Limit too low for workload	Raise the limit (after measuring peak + 20% headroom)
Container OOM, peak > limit, monotonic climb	Memory leak	Escalate to application ownership; do not raise limit indefinitely
Container OOM, peak far below the limit	cgroup limit hit before Java/.NET-style working set settled	Raise the limit modestly; consider JVM heap flags
Node-level OOM (kernel), unrelated Pods killed	Workload escaped its cgroup (e.g. via sidecar or volume)	Identify the escape path; do not raise the cgroup limit
Memory pressure eviction	Node has too little memory for current Pods	Reduce density or scale out the node pool

4. Apply the fix

Read-only / SafeApply the fix


#   resources:
#     requests:
#       memory: "256Mi"
#     limits:
#       memory: "512Mi"   # was 256Mi - raise to 384Mi-512Mi based on measurement

git commit -am "raise memory limit for <name> based on measured peak + 20% headroom"
git push
kubectl rollout status deploy/<name> -n <ns> --timeout=10m

# Verify the new limit
kubectl get pod -l app=<name> -n <ns> -o jsonpath='{.items[*].spec.containers[*].resources.limits.memory}'

5. Verify under load

Read-only / SafeVerify under load


kubectl get pod -l app=<name> -n <ns> -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,READY:.status.containerStatuses[0].ready
sleep 1800
kubectl get pod -l app=<name> -n <ns> -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,READY:.status.containerStatuses[0].ready

# Confirm no OOM in the events
kubectl get events -n <ns> --field-selector reason=OOMKilling | tail
kubectl get events -n <ns> --field-selector involvedObject.name=<name> | grep -iE 'oom|killed' || echo "no OOM events"

Common pitfalls

Symptom	Cause	Action
OOM recurs at a shorter interval after raising the limit	Confirmed leak; the higher limit just delays the inevitable	Escalate to application ownership with the heap dump
OOMKilled on a Pod with no memory limit set	Node-level OOM or memory pressure eviction	Check node journal and node conditions
OOMKilled on first start, before any traffic	Application initialisation needs more memory than the limit	Raise the limit; consider startupProbe to delay readiness
OOMKilled in a JVM after raising the limit	JVM heap flags don’t match the limit	Align `-Xmx` with the limit; see `kubernetes-cxxviii-03-memory-oom`
OOMKilled on a system Pod (CNI, kube-proxy)	Node-level issue	Follow `kubernetes-rb-troubleshoot-node-notready`

OOMKilled is a memory management signal. The right fix depends on whether the workload genuinely needs more memory, the cluster genuinely has less memory available, or the application genuinely leaks. Measure, then change one thing.