Pressure-driven conditions — memory, disk, PID when the kubelet detects them

The kubelet detects node pressure through three checks: memory, disk, and PID. The checks are configurable; the detection triggers the eviction loop and the condition updates. This lesson walks the checks, the thresholds, and the operational patterns for managing pressure.

The kubelet’s pressure detection

The kubelet runs three periodic checks:

flowchart TD
    A[kubelet housekeeping] --> B[Memory pressure check]
    A --> C[Disk pressure check]
    A --> D[PID pressure check]
    B -->|threshold exceeded| E[MemoryPressure=True]
    C -->|threshold exceeded| F[DiskPressure=True]
    D -->|threshold exceeded| G[PIDPressure=True]
    E --> H[Eviction loop]
    F --> H
    G --> H

The checks run every 10 seconds (the --node-status-update-frequency flag). The checks are independent; a node can have multiple conditions at once.

The memory pressure check

The kubelet’s memory pressure check uses the kernel’s memory pressure metrics. The default threshold is the kernel’s PSI (Pressure Stall Information) metric when available; on older kernels, the kubelet uses the memory available threshold.

kubelet \
  --eviction-hard=memory.available<100Mi \
  --eviction-soft=memory.available<500Mi \
  --eviction-soft-grace-period=30s

The kubelet’s --eviction-hard flag sets the hard threshold. When the memory is below the threshold, the kubelet immediately evicts Pods.

The kubelet’s --eviction-soft flag sets the soft threshold. When the memory is below the soft threshold for the grace period, the kubelet evicts Pods.

The kubelet’s --eviction-soft-grace-period flag sets the grace period for the soft threshold. The default is the kubelet’s housekeeping interval.

The disk pressure check

The kubelet’s disk pressure check uses the filesystem usage. The default threshold is 85% of the node’s filesystem.

kubelet \
  --eviction-hard=nodefs.available<10% \
  --eviction-hard=nodefs.inodesFree<5% \
  --eviction-hard=imagefs.available<15%

The kubelet distinguishes three filesystems:

• nodefs: the node’s root filesystem. The threshold is 10% free.
• nodefs.inodesFree: the node’s inode usage. The threshold is 5% free.
• imagefs: the runtime’s image filesystem. The threshold is 15% free.

The kubelet’s disk pressure is triggered when any of the three thresholds is exceeded.

The PID pressure check

The kubelet’s PID pressure check uses the kernel’s PID usage. The default threshold is 0.5 of the kernel’s pid_max.

kubelet \
  --eviction-hard=pid.available<5%

The kubelet’s PID pressure is triggered when the kernel’s PID usage exceeds the threshold. The default threshold is conservative; a workload that spawns many processes may need to lower the threshold.

The thresholds

The kubelet’s default thresholds:

The thresholds are the kubelet’s detection limits. The production rule is to set the thresholds to values that give the operator time to react.

The eviction loop

When the kubelet detects a pressure condition, the eviction loop runs. The loop:

1. Identifies the Pods to evict based on the Pod’s QoS class and the resource requests.
2. Evicts the Pods by sending the API server’s eviction API.
3. The kubelet waits for the Pods to be terminated.
4. The kubelet checks the pressure again.
5. The loop continues until the pressure is below the threshold.

The eviction loop is the kubelet’s response to pressure. The pressure condition is the cluster’s signal.

The eviction order

The kubelet evicts Pods in a specific order based on the QoS class:

flowchart TD
    A[Eviction loop] --> B[BestEffort Pods]
    B --> C[Burstable Pods]
    C --> D[Guaranteed Pods]
    D --> E[Eviction complete]

The order is:

1. BestEffort Pods: Pods with no resource requests. The least important; the first to be evicted.
2. Burstable Pods: Pods with some resource requests. The next to be evicted.
3. Guaranteed Pods: Pods with requests equal to limits. The last to be evicted.

The kubelet’s eviction order is the QoS class. The production rule: design the workload to be in the Guaranteed class when the workload is critical; design the workload to be in the BestEffort class when the workload is not critical.

The eviction’s effects

The eviction’s effects:

• The Pod is sent a SIGTERM.
• The kubelet waits for the Pod’s terminationGracePeriodSeconds (default 30s).
• The kubelet sends a SIGKILL.
• The Pod’s status is Failed with reason Evicted.
• The Pod’s controller (Deployment, StatefulSet, etc.) creates a replacement Pod.
• The replacement Pod is scheduled by the scheduler.

The total time from eviction to replacement is the Pod’s termination grace period plus the scheduler’s scheduling time. The production rule is to tune the Pod’s termination grace period for the workload’s shutdown time.

The pressure’s operational patterns

The pressure’s operational patterns:

• Monitor the pressure conditions. The cluster’s alerts should fire on the MemoryPressure=True, DiskPressure=True, and PIDPressure=True conditions.
• Tune the thresholds for the workload. A latency-sensitive workload may want a higher threshold (more aggressive eviction); a batch workload may want a lower threshold (less aggressive eviction).
• Use a separate filesystem for the image cache. The image cache is the largest directory; isolating it prevents the image cache from filling the node’s root filesystem.
• Set the Pod’s resource requests. The eviction order is based on the QoS class. The Pod’s resource requests determine the QoS class.
• Audit the kubelet’s flags at every release. A kubelet whose thresholds are misconfigured is a kubelet that is missing the pressure.

Quiz

Production discipline

• The kubelet detects pressure through three checks. The memory, disk, and PID checks are independent. Each has a threshold; each has a flag.
• Tune the thresholds for the workload. The default thresholds are conservative; a production cluster may want to lower the thresholds to give the operator more time to react.
• The eviction loop is the kubelet’s response. The loop is destructive; the preferred response is to add capacity or to reduce the workload.
• The eviction order is the QoS class. A Guaranteed Pod is the last to be evicted; a BestEffort Pod is the first.
• Monitor the eviction metrics. The kubelet_containers_evicted_total and kubelet_pods_evicted_total metrics expose the eviction rate. The operator should alert on the rate.
• Audit the thresholds at every node repave. A new node that joins the cluster with the wrong thresholds is a node that is missing the pressure. The audit catches the failure.