Throttling, OOMKill, and the resource pressure lifecycle

The resource pressure lifecycle is the progression from a healthy Pod to a degraded one to a failed one. This lesson walks through that lifecycle, explains the signals at each stage, and shows the production discipline around preventing resource exhaustion.

The resource pressure lifecycle

stateDiagram-v2
    [*] --> Healthy
    Healthy --> CPUPressure: usage > request
    CPUPressure --> Throttled: usage > limit
    Throttled --> Latency: tail latency spikes
    Latency --> Healthy: load decreases
    Throttled --> Healthy: load decreases
    Healthy --> MemPressure: usage > request
    MemPressure --> MemLimit: usage approaches limit
    MemLimit --> OOMKill: usage > limit (cgroup)
    OOMKill --> CrashLoop: repeated OOMKill
    CrashLoop --> Healthy: memory limit raised / leak fixed
    MemPressure --> NodePressure: usage > node allocatable
    NodePressure --> Eviction: kubelet evicts Pods

Three failure modes:

• CPU throttling: gradual; latency increases; container runs slower but doesn’t die.
• Memory OOMKill: immediate; container restarts; logs empty.
• Node pressure eviction: the kubelet evicts Pods to reclaim node resources (memory or disk).

CPU throttling in detail

sequenceDiagram
    participant App as Application
    participant Sched as CFS Scheduler
    participant K as Kernel

    App->>Sched: asks for CPU
    alt usage < quota
        Sched-->>App: CPU granted
    else usage > quota
        Sched->>App: throttled
        Note over Sched: until next period
    end

When a container’s CPU usage exceeds its limit, the CFS scheduler throttles the container for the remainder of the period. The container’s threads are paused.

The symptoms:

• Tail latency spikes: requests that take 10ms normally now take 100ms because the thread is throttled.
• Throughput drops: under load, the application processes fewer requests per second.
• No crash: the container continues running; logs show no errors.

Detection:

cat /sys/fs/cgroup/.../container-*/cpu.stat
# nr_throttled 4567
# throttled_usec 987654321

throttled_pct = throttled_usec / (nr_periods * period_us) in decimal. >10% is a problem.

Memory OOMKill in detail

sequenceDiagram
    participant App as Application
    participant K as Kernel
    participant CG as cgroup

    App->>CG: allocates memory
    CG->>CG: tracks memory.current
    alt memory.current < memory.max
        Note over App,CG: running
    else memory.current > memory.max
        CG->>K: OOM signal
        K->>App: SIGKILL
        App-->>App: process dies
        CG->>CG: oom_kill++
    end

When a container’s memory usage exceeds its limit, the cgroup triggers the OOM killer. The kernel sends SIGKILL to the container’s PID 1 (and other processes in the cgroup). The process is killed immediately.

The symptoms:

• Empty logs: the process is killed before it can write logs. kubectl logs --previous may be empty.
• CrashLoopBackOff: the kubelet restarts the container with exponential backoff.
• Exit code 137: 128 + SIGKILL (9). The container’s exit code matches.

Detection:

cat /sys/fs/cgroup/.../container-*/memory.events
# oom_kill 3

oom_kill count > 0 confirms cgroup OOMKill. Compare with the container’s restartCount; if they match, every restart was an OOMKill.

The OOMKill exit code 137

Exit code 137 is not unique to cgroup OOMKill. It can also be:

• kubelet SIGKILL after grace period: the application didn’t exit in time; kubelet SIGKILLs it.
• cgroup OOMKill: the memory usage exceeded the limit.

To distinguish:

• cgroup OOMKill: state.terminated.reason: OOMKilled (kubelet reports this), memory.events: oom_kill > 0 on the node.
• kubelet SIGKILL: state.terminated.reason: Error, memory.events: oom_kill = 0.

For cgroup OOMKill, the kubelet knows because it reads the cgroup counters after the container exits.

Node-level eviction

The kubelet monitors node-level resource pressure and evicts Pods to reclaim resources. The thresholds are configured via --eviction-hard and --eviction-soft flags:

--eviction-hard=memory.available<500Mi
--eviction-hard=nodefs.available<10%
--eviction-soft=memory.available<1Gi
--eviction-soft=nodefs.available<15%
--eviction-grace-period=30s

When memory.available drops below 500Mi (or the soft threshold for 30s), the kubelet starts evicting Pods. The eviction order:

1. BestEffort Pods (no requests/limits) first.
2. Burstable Pods next (by usage vs request).
3. Guaranteed Pods last.

flowchart TD
    Node[Node memory low] --> Q{Pod QoS?}
    Q -- BestEffort --> E1[Evict first]
    Q -- Burstable --> E2[Evict second]
    Q -- Guaranteed --> E3[Evict last]

Within Burstable, the kubelet evicts the Pod whose usage is furthest above its request. A Pod using 800Mi with a 100Mi request is evicted before one using 300Mi with a 100Mi request.

The evicted Pod is rescheduled elsewhere if there’s capacity. If the cluster has no capacity, the Pod stays Pending.

The pressure lifecycle in practice

A typical production scenario:

1. Healthy: Pod runs normally, 300Mi memory, 100m CPU.
2. Memory growth: application memory creeps up due to a leak; usage is now 450Mi (limit is 512Mi).
3. Memory pressure: usage approaches limit; PSI shows memory.pressure: avg10 > 5%. Slow performance.
4. OOMKill: usage exceeds 512Mi; cgroup kills the process. Container exits with 137, reason OOMKilled.
5. CrashLoopBackOff: kubelet restarts; container hits OOMKill again; backoff increases.
6. Memory limit raised: operator raises limit to 1Gi.
7. Healthy again: usage stabilises at 600Mi; no more OOMKills.

Detection points:

• Step 2-3: PSI metrics, memory.current growth rate.
• Step 4: memory.events: oom_kill > 0, state.terminated.reason: OOMKilled.
• Step 5: restartCount increasing, CrashLoopBackOff event.

Production patterns

Detect CPU throttling early:

# node-exporter exposes container_cpu_cfs_throttled_seconds_total
rate(container_cpu_cfs_throttled_seconds_total[5m]) > 0.1

A rate of > 0.1 means the container is throttled more than 10% of the time. Alert when sustained.

Detect memory pressure:

# PSI memory pressure
node_pressure_memory_waiting_seconds_total
/ node_pressure_memory_total_seconds_total
> 0.05

A ratio > 5% means significant memory pressure. Alert when sustained.

Detect OOMKill:

kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}

This metric counts Pods whose last termination reason was OOMKilled. Alert on any value > 0.

Diagnosing the lifecycle

The diagnostic workflow:

1. Pod is slow: check kubectl top pod and node-exporter cgroup metrics for CPU throttling.
2. Pod is restarting: check kubectl describe pod for exit reason; check memory.events for oom_kill.
3. Pod is being evicted: check node-exporter’s kubelet_evictions metric and the node’s memory.available.
4. Node is under pressure: check kube_node_status_allocatable and kube_node_status_capacity; consider adding capacity.

Cross-course references

• The Linux course part XXXVII-Linux-Resources covers cgroup resource management; the kubelet’s enforcement is the cluster-level equivalent.
• The Docker course part XXXIX-Docker-Resources covers container resource limits; Pod-level resource limits are the cluster-level extension.
• The Observability course part LXXXVII-Kubernetes-MetricsServer covers kubectl top; production resource monitoring uses these metrics.

Quiz

Production discipline

• Detect CPU throttling with cgroup metrics. Alert on throttled_pct > 10% sustained for 5 minutes.
• Detect memory pressure with PSI. Alert on memory.pressure: some avg10 > 10%.
• Track OOMKill with exit reasons. OOMKilled in state.terminated.reason is the cgroup kill.
• Set memory limits on every container. OOMKill at the limit is better than node-wide eviction.
• Monitor memory growth rate. A leak shows up as sustained growth, not just absolute usage.