PID pressure eviction — when the kernel runs out of PIDs

PID pressure is the kubelet’s signal that the node is exhausting the PID namespace. The kubelet detects the pressure through the kernel’s PID usage; the eviction loop reclaims the PIDs by evicting Pods. This lesson walks the detection, the eviction loop, and the operational patterns.

The kernel’s PID limits

The Linux kernel’s PID namespace limits the number of PIDs available in a namespace. The limits:

• pid_max: the maximum number of PIDs in the system. Default 32768 on older kernels, 4194304 on newer kernels.
• pid_max per user: the maximum number of PIDs per user. Default 32768.
• threads-max: the maximum number of threads. Default approximately pid_max.

The kubelet’s PID pressure is based on the pid.available metric. The metric is the number of PIDs available for new processes.

cat /proc/sys/kernel/pid_max

The kernel’s pid_max is the absolute limit. Raising it lifts the node’s ceiling but does nothing about a workload that leaks processes.

The kubelet’s PID threshold

The kubelet’s PID pressure threshold:

kubelet --eviction-hard=pid.available<5%

The default threshold is 5% of the available PIDs. The kubelet considers the node to be in PID pressure when the available PIDs is below 5% of the kernel’s pid_max.

The threshold is the kubelet’s detection limit. The production rule is to set the threshold to a value that gives the operator time to react.

The PID pressure scenario

A node with pid_max = 32768 is running 100 Pods with 300 processes each. The kernel’s PIDs are exhausted; the node is in PID pressure.

flowchart TD
    A[pid_max: 32768] --> B[100 Pods with 300 processes each]
    B --> C[pid.available: 2700]
    C --> D{PID pressure?}
    D -->|Yes| E[PIDPressure=True]
    E --> F[Eviction loop]
    F --> G[Evict Pods by process count]
    G --> H[pid.available: 5400]
    H --> I{Above threshold?}
    I -->|Yes| J[Done]
    I -->|No| K[Evict more Pods]

The eviction loop evicts the Pods based on the process count. The BestEffort Pods are evicted first; the Burstable Pods are next; the Guaranteed Pods are last.

The PID pressure’s Pod selection

The kubelet’s PID pressure eviction selects the Pods based on the Pod’s process count. The Pod with the highest process count is evicted first.

The calculation:

For each Pod:
  process_count = sum of processes in the Pod's containers
  over_threshold = process_count > pod's PID threshold
  evict_first = over_threshold

The kubelet’s PID threshold is the Pod’s resources.limits.processes if set; otherwise, no threshold is applied.

The Pod’s resources.limits.processes is a Kubernetes extension that limits the Pod’s process count. The limit is enforced by the runtime’s cgroup.

The PID pressure’s diagnostic

A node in PID pressure has the PIDPressure=True condition:

# Substitute your own value before running:
NODE=node-03

kubectl describe node "$NODE" | grep -A 5 "Conditions"

Conditions:
  Type                 Status  Reason
  ----                 ------  ------
  PIDPressure          True    KubeletHasInsufficientPID

The diagnostic:

journalctl -u kubelet | grep -i "pid"
ps aux | wc -l

The kubelet logs the eviction. The fix is to:

1. Identify the workload. The kubelet’s logs show the evicted Pods.
2. Investigate the workload’s process count. The Pod’s process count is the cause.
3. Set the PID limit. The Pod’s resources.limits.processes limits the process count.
4. Add capacity. The cluster may need more nodes; the pid_max may be too low.

The PID pressure’s metrics

The kubelet’s metrics expose the PID pressure:

# The kubelet serves its metrics on the node's own address:
NODE=node-03
NODE_IP=$(kubectl get node "$NODE" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')

curl -k https://"$NODE_IP":10250/metrics | grep pid

The relevant metrics:

• kubelet_pid_evictions_total
• kubelet_process_stats_pid_count
• node_processes_total

The operator should monitor the metrics and alert on the eviction rate. A rising eviction rate is a cluster that is losing PIDs.

The PID pressure’s common causes

The PID pressure’s common causes:

• Fork bomb: a workload that is spawning processes in an infinite loop. A per-Pod PID limit contains the damage to that Pod’s cgroup.
• Process leak: a workload that is spawning processes without reaping them. The children pile up as zombies until the container’s PID 1 reaps them; an init process in the image fixes it.
• High-density deployments: a node that is running too many Pods. The fix is to add capacity.
• Container’s PID namespace sharing: a Pod with shareProcessNamespace: true shares the PID namespace; the Pod’s processes are exposed to the other Containers in the Pod.

The production rule: monitor the PID pressure; alert on the condition.

The kernel’s PID configuration

The kernel’s PID configuration:

cat /proc/sys/kernel/pid_max
cat /proc/sys/kernel/threads-max

The pid_max is the maximum number of PIDs in the system. The threads-max is the maximum number of threads.

The kubelet’s threshold is a percentage of the available PIDs. A higher pid_max is a larger PID namespace; the kubelet’s threshold is a larger absolute value.

The production rule: set the pid_max for the workload’s expected process count. A workload that uses 1000 processes should have a pid_max of at least 110% of the expected count.

The Pod’s PID limit

The Pod’s resources.limits.processes is a Kubernetes extension that limits the Pod’s process count. The limit is enforced by the kubelet’s PID cgroup.

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: registry.example.com/app:1.0.0
      resources:
        limits:
          processes: 1000

The limit is the Pod’s process count ceiling. The container is killed by the kubelet when the limit is exceeded.

The limit is a Kubernetes extension; not all distributions support it. The production rule is to verify the limit is supported before using it.

The production patterns

The PID pressure’s production patterns:

• Set the Pod’s PID limit. The Pod’s resources.limits.processes limits the process count. The limit is the safety net.
• Monitor the PID pressure. The cluster’s alerts should fire on the PIDPressure=True condition.
• Tune the kubelet’s threshold. The default is 5%; a production cluster may want to lower the threshold to give the operator more time to react.
• Audit the process count at every release. A new workload that is using more processes than expected is a workload that needs investigation.

Quiz

Production discipline

• PID pressure is the cluster’s signal. The kubelet’s PIDPressure=True is the cluster’s signal that the node is exhausting the PID namespace.
• The typical cause is a process leak. The fix is to identify the workload and address the leak.
• Set the Pod’s PID limit. The Pod’s resources.limits.processes limits the process count.
• Tune the kubelet’s threshold. The default is 5%; a production cluster may want to lower the threshold.
• Monitor the PID pressure. The cluster’s alerts should fire on the condition.
• Audit the process count at every release. A new workload that is using more processes than expected is a workload that needs investigation.
• Test the eviction in non-production. A staging cluster that mirrors production is the right place to test the eviction logic.