Memory limits and OOM semantics — the memory bottleneck

A container that reaches its cgroup memory limit is killed with SIGKILL. No shutdown hook runs, no stack trace is written, and the only record is exit code 137 with the reason OOMKilled on the previous container state. kubectl logs shows the restarted attempt rather than the one that died, so the evidence an operator needs sits one flag away and is easy to walk past. This lesson covers reading that evidence and separating a limit set too low from a genuine leak and from node-level memory pressure.

The OOM semantics

The OOM (Out Of Memory) is the kernel’s response to a container that has exceeded its memory limit. The cgroup’s memory limit is hit; the kernel kills the container with SIGKILL; the kubelet records the exit reason as OOMKilled.

flowchart LR
    A[Container] --> B{Memory limit reached?}
    B -->|Yes| C[Kernel OOM killer]
    C --> D[Container killed]
    B -->|No| E[Run]

The OOM is the kernel’s memory enforcement.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
POD=checkout-5f9c7d8b6c-2xk9p
NS=production

# 1. Check the OOM
kubectl describe pod "$POD" -n "$NS" | grep -A5 "Last State"

# 2. Check the memory usage
kubectl top pod "$POD" -n "$NS"

# 3. Check the memory limits
kubectl describe pod "$POD" -n "$NS" | grep -A5 Limits

# 4. Check the kernel logs
kubectl exec -it "$POD" -n "$NS" -- dmesg | tail -50

# 5. Check the application's logs
kubectl logs -n "$NS" "$POD" --previous

The diagnostic is the OOM, the memory usage, the limits, the kernel logs, and the application’s logs.

Common failures

• Memory limit too low. The memory limit is too low. The remediation is to increase the memory limit.
• Memory leak. The application has a memory leak. The remediation is to fix the application.
• Memory-intensive workload. The workload is memory-intensive. The remediation is to optimise the workload.
• Node under memory pressure. The node is under memory pressure. The remediation is to add memory or scale the cluster.

flowchart TD
    A[OOM] --> B{Memory limit low?}
    B -->|Yes| C[Increase the memory limit]
    B -->|No| D{Memory leak?}
    D -->|Yes| E[Fix the application]
    D---|No| F{Memory-intensive workload?}
    F -->|Yes| G[Optimise the workload]
    F -->|No| H{Node under pressure?}
    H -->|Yes| I[Add memory or scale]
    H -->|No| J[Unknown]

The OOM exit code

The OOM exit code is 137 (128 + 9, SIGKILL). The kubelet records the exit reason as OOMKilled.

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

The exit code is the canonical OOM signal.

The remediation

The remediation depends on the cause:

# Substitute your own values before running:
DEPLOYMENT=checkout
NS=production
NODEPOOL=workers-general
NODEPOOL_REPLICAS=6

# Option 1: Increase the memory limit
kubectl set resources deployment "$DEPLOYMENT" -n "$NS" --limits=memory=1Gi

# Option 2: Fix the memory leak
# (application-specific)

# Option 3: Add memory
kubectl scale nodepool "$NODEPOOL" --replicas="$NODEPOOL_REPLICAS"

# Option 4: Optimise the workload
# (workload-specific)

The remediation is the memory recovery.

Production discipline

An OOM issue is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the memory layer, identify the cause, apply the remediation. The memory is the cluster’s data; the remediation is the memory recovery.

• Check the memory limits. The limits are the workload’s boundary.
• Check the memory usage. The usage is the workload’s profile.

Quiz