Troubleshooting resource pressure — a production triage framework

When a Pod is slow or restarting, the cause is often a resource issue. The triage framework here identifies whether the issue is CPU throttling, memory OOMKill, or node-level eviction; gathers evidence at the right layer; and applies the fix without making things worse.

The three failure modes

flowchart TD
    Symptom[Symptom: Pod slow or restarting] --> Mode{Which mode?}
    Mode -- CPU throttling --> T1[Latency spike, no restart]
    Mode -- Memory OOMKill --> T2[Restart, exit code 137]
    Mode -- Node eviction --> T3[Pod terminated, Rescheduled elsewhere]
    T1 --> Fix1[Increase CPU limit / reduce usage]
    T2 --> Fix2[Increase memory limit / fix leak]
    T3 --> Fix3[Add capacity / right-size]

The mode determines the evidence to gather and the fix to apply.

Mode 1: CPU throttling

Symptoms:

• Latency spikes (p99 increases; tail latency worsens).
• Throughput drops (requests per second decreases).
• No container restarts.
• kubectl top pod shows CPU usage near or above the limit.

Evidence:

• Container is in state.running (not Waiting or Terminated).
• restartCount is stable (no restarts).
• kubectl describe pod events are normal (no OOMKill, no Failed).
• On the node: cgroup cpu.stat shows high throttled_usec and nr_throttled.

Diagnosis:

# On the node hosting the Pod
POD_UID=$(kubectl get pod web-7c8 -o jsonpath='{.metadata.uid}')
CGROUP="/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/pod${POD_UID}"

# Read CPU throttling
cat $CGROUP/container-*/cpu.stat | grep throttled
# nr_throttled 4567
# throttled_usec 987654321

throttled_pct = throttled_usec / (nr_periods * 100000) in microseconds. If >10%, the container is throttled.

Fix:

1. Increase CPU limit (if the workload genuinely needs more CPU).
2. Reduce CPU usage (profile the application; optimise hot paths).
3. Move the workload to a less contended node.
4. Scale horizontally (more replicas with smaller CPU each).

Mode 2: Memory OOMKill

Symptoms:

• Container restarts repeatedly with exit code 137.
• kubectl logs --previous may be empty (process killed before logging).
• Pod is in CrashLoopBackOff.
• kubectl describe pod shows state.terminated.reason: OOMKilled.

Evidence:

• state.terminated.exitCode: 137.
• state.terminated.reason: OOMKilled (not Error).
• On the node: cgroup memory.events shows oom_kill > 0.
• kubectl top pod shows memory usage near or above the limit before the kill.

Diagnosis:

# Verify OOMKill from cgroup
cat $CGROUP/container-*/memory.events | grep oom
# max 5
# oom 0
# oom_kill 3

# Memory usage at time of kill (if not reset)
cat $CGROUP/container-*/memory.current
# 536870912  # 512Mi — hit the limit

Fix:

1. Increase memory limit (if the workload legitimately needs more memory).
2. Fix a memory leak (profile with heap dump; review recent changes).
3. Reduce memory usage (cache eviction, smaller buffers, streaming).
4. Set memory request close to limit so the scheduler reserves enough; helps with placement.

Mode 3: Node-level eviction

Symptoms:

• Pod is evicted (not OOMKilled); kubectl describe pod shows reason: Evicted.
• The Pod is rescheduled on another node if there’s capacity.
• Other Pods on the same node may also be evicted.
• kubectl describe node shows low memory.available or nodefs.available.

Evidence:

• state.terminated.reason: Evicted (different from OOMKilled).
• The kubelet’s eviction manager logged the reason (memory.available, nodefs.available, etc.).
• Multiple Pods on the same node are evicted around the same time.

Diagnosis:

# Check node-level pressure
kubectl describe node node-3 | grep -A 5 "Conditions"
# Conditions:
#   ... MemoryPressure   True   ...
#   ... DiskPressure     True   ...

# Check evicted Pods
kubectl get pods -A --field-selector=status.reason=Evicted

Fix:

1. Add capacity (scale up nodes; Cluster Autoscaler).
2. Reduce per-Pod requests (right-size via VPA).
3. Move workloads off contended nodes (Pod topology spread, anti-affinity).
4. Tune eviction thresholds to evict earlier (smaller buffer before full pressure).

Triage decision tree

flowchart TD
    Start[Pod slow or restarting] --> Q1{Container restarting?}
    Q1 -- no --> Q2{Latency spike?}
    Q2 -- yes --> CPU[CPU throttling]
    Q2 -- no --> Other[Look elsewhere: network, disk, app bug]
    Q1 -- yes --> Q3{Exit code?}
    Q3 -- 137 --> Q4{state.terminated.reason?}
    Q4 -- OOMKilled --> OOM[Memory OOMKill]
    Q4 -- Error --> Q5{SIGKILL from kubelet?}
    Q5 -- yes --> Grace[Grace period exceeded]
    Q5 -- no --> Cgroup{cgroup oom_kill > 0?}
    Cgroup -- yes --> OOM
    Cgroup -- no --> OtherKill[Other cause: signal, runtime error]
    Q3 -- 0 --> Completed[App exited cleanly; check restartPolicy]
    Q3 -- 1-127 --> AppErr[Application error; read logs]
    Q3 -- 143 --> SIGTERM[Graceful termination; check Pod lifecycle]

The decision tree:

1. Container restarting?: if no, it’s CPU throttling or something else.
2. Latency spike?: if yes, CPU throttling.
3. Exit code 137?: if yes, OOMKill (cgroup) or kubelet SIGKILL (grace period).
4. state.terminated.reason: OOMKilled = cgroup; Error = kubelet SIGKILL.
5. memory.events.oom_kill: if > 0, confirmed cgroup OOMKill.

Production patterns

CPU throttling — increase limit safely:

# 1. Confirm throttling
kubectl exec web-7c8 -- cat /sys/fs/cgroup/cpu.stat   # may not work from inside
# Or check from the node

# 2. Profile the application (pprof, perf, etc.)

# 3. If legitimate, raise the limit
kubectl set resources deployment web -c web --limits=cpu=1000m

Memory OOMKill — fix or raise:

# 1. Confirm OOMKill from cgroup
kubectl describe pod web-7c8 | grep -A 3 OOMKilled

# 2. Profile memory usage (heap dump, /proc/PID/status)

# 3. If legitimate, raise the limit
kubectl set resources deployment web -c web --limits=memory=1Gi

# 4. Set memory request close to limit (helps with scheduling)
kubectl set resources deployment web -c web --requests=memory=768Mi

Node eviction — add capacity:

# 1. Confirm node pressure
kubectl describe node node-3 | grep -A 5 "Conditions"

# 2. Add capacity
# Via Cluster Autoscaler: it does this automatically
# Manually: terraform apply for the node pool

# 3. Or reduce requests
kubectl set resources deployment web -c web --requests=cpu=100m,memory=128Mi

What NOT to do

Common anti-patterns during incident response:

• Don’t raise all limits blindly: making the problem worse by over-allocating.
• Don’t force-delete Pods in CrashLoopBackOff: the Deployment will recreate them; fix the root cause.
• Don’t disable kubelet eviction thresholds: this hides the problem, doesn’t fix it.
• Don’t add nodes without understanding the cause: if the workload is leaking memory, more nodes just delay the crash.
• Don’t ignore memory.events on the node: it’s the ground truth for OOMKill.

Capturing evidence before the incident clears

Production discipline: capture the state before debugging resets it:

# Capture the Pod's full spec
kubectl get pod web-7c8 -o yaml > /tmp/web-7c8.yaml

# Capture events
kubectl get events --field-selector involvedObject.name=web-7c8 \
  > /tmp/web-7c8-events.txt

# Capture top metrics (snapshot)
kubectl top pod web-7c8 --containers > /tmp/web-7c8-top.txt

# If you can SSH to the node, capture cgroup state
ssh node-3 "cat /sys/fs/cgroup/.../web-7c8/cpu.stat" > /tmp/cpu.stat
ssh node-3 "cat /sys/fs/cgroup/.../web-7c8/memory.events" > /tmp/memory.events

Save these to the incident channel. The postmortem depends on them.

Cross-course references

• The Linux course part XXXVII-Linux-Resources covers cgroup resource management; the kubelet’s enforcement is the cluster-level equivalent.
• The Ansible course part XLV-Ansible-Debugging covers systematic debugging; the triage framework is the same.
• The Observability course part LXXXVII-Kubernetes-MetricsServer covers kubectl top; production resource monitoring uses these metrics.

Quiz

Production discipline

• Start with kubectl describe pod. State and events tell you the layer.
• Verify with cgroup files. The kubelet’s view is derived from the cgroup; the cgroup is ground truth.
• Capture state before debugging. Pod YAML, events, top metrics, cgroup files. The postmortem depends on them.
• Profile before raising limits. A leak or CPU bottleneck is fixed at the application, not at the limit.
• Monitor resource growth rate. Linear growth is a leak; alert before OOMKill happens.