DiskPressure and PIDPressure — the resource exhaustion

A node under DiskPressure or PIDPressure starts evicting Pods, and the Pods it picks are rarely the ones responsible. Disk fills from the image cache and container logs far more often than from any single workload, and the PID space is exhausted by one process forking without reaping while everything else on the node behaves. This lesson covers reading the node conditions and then finding the actual consumer on the host, which is where both problems live.

The resource exhaustion modes

A node has three resource exhaustion modes:

1. MemoryPressure. The node’s memory is exhausted. The kubelet evicts Pods to free memory.
2. DiskPressure. The node’s disk is exhausted. The kubelet evicts Pods to free disk.
3. PIDPressure. The node’s PID limit is exhausted. The kubelet evicts Pods to free PIDs.

flowchart TD
    A[Node] --> B{MemoryPressure?}
    B -->|Yes| C[Evict Pods]
    B -->|No| D{DiskPressure?}
    D -->|Yes| E[Evict Pods]
    D---|No| F{PIDPressure?}
    F -->|Yes| G[Evict Pods]

The exhaustion modes are the cluster’s guarding.

The diagnostic

The canonical diagnostic:

# Substitute your own value before running:
NODE=worker-03            # node name from `kubectl get nodes`

# 1. Check the node's conditions
kubectl describe node "$NODE"

# 2. Check the node's resources
kubectl top node "$NODE"

# 3. Check the node's disk usage
ssh "$NODE" "df -h"

# 4. Check the node's memory usage
ssh "$NODE" "free -h"

# 5. Check the node's PID usage
ssh "$NODE" "ps aux | wc -l"

# 6. Check the kubelet's eviction logs
ssh "$NODE" "journalctl -u kubelet | grep -i evicted"

The diagnostic is the node conditions, the resource usage, and the kubelet’s eviction logs.

DiskPressure

A DiskPressure condition is set when the node’s disk usage exceeds the kubelet’s threshold (default 85%). The kubelet evicts Pods to free disk.

Conditions:
  Type             Status  Reason                       Message
  ----             ------  ------                       -------
  DiskPressure     True    KubeletHasDiskPressure       kubelet has disk pressure

The diagnostic is the disk usage:

NODE=worker-03            # node name from `kubectl get nodes`

ssh "$NODE" "df -h"

A real disk usage:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       100G   92G  8.0G  92% /

The 92% is above the threshold. The kubelet is evicting Pods.

The remediation is to free disk:

NODE=worker-03            # node name from `kubectl get nodes`

# Find the largest directories
ssh "$NODE" "du -sh /var/lib/containerd/* | sort -h | tail -10"

# Clean up unused images
ssh "$NODE" "crictl rmi --prune"

# Clean up unused containers
ssh "$NODE" "crictl rm --prune"

PIDPressure

A PIDPressure condition is set when the node’s PID count exceeds the kernel’s limit (default kernel.pid_max = 4194304). The kubelet evicts Pods to free PIDs.

Conditions:
  Type             Status  Reason                       Message
  ----             ------  ------                       -------
  PIDPressure      True    KubeletHasPIDPressure        kubelet has insufficient PID available

The diagnostic is the PID count:

NODE=worker-03            # node name from `kubectl get nodes`

ssh "$NODE" "ps aux | wc -l"

A real PID count:

4190000

The 4190000 is above the threshold. The kubelet is evicting Pods.

The remediation is to free PIDs:

NODE=worker-03            # node name from `kubectl get nodes`

# Find the processes with the most PIDs
ssh "$NODE" "ps -eLf | awk '{print \$2}' | sort | uniq -c | sort -rn | head -10"

# Kill processes that are stuck
STUCK_PID=28417           # PID from the ps output above
ssh "$NODE" "kill -9 $STUCK_PID"

# Increase the kernel's PID limit
ssh "$NODE" "sysctl -w kernel.pid_max=8388608"

The remediation

The remediation depends on the cause:

• DiskPressure. Clean up unused images, unused containers, and unused logs.
• PIDPressure. Kill stuck processes, increase the kernel’s PID limit.
• MemoryPressure. Identify the memory-intensive Pods, scale them down, or increase the node’s memory.

Production discipline

A resource exhaustion is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the node, identify the cause, apply the remediation. The node’s resources are the cluster’s capacity; the remediation is the pressure recovery.

• Check the node conditions. The conditions are the cluster’s view.
• Clean up the disk. The cleanup is the cluster’s recovery.

Quiz