Memory pressure eviction — the kubelet's reclaim logic

Memory pressure eviction is the kubelet’s response to memory exhaustion. The kubelet detects the pressure through the kernel’s memory metrics; the eviction loop reclaims the memory by evicting Pods based on the QoS class. This lesson walks the detection, the eviction loop, and the operational patterns.

The kernel’s memory metrics

The kubelet reads the kernel’s memory metrics through the cgroup and the PSI interfaces. The relevant metrics:

• memory.available: the memory available for new allocations. The kubelet uses this as the primary signal.
• memory.pressure: the PSI metric for memory pressure. Available on newer kernels (4.20+).
• memory.usage: the memory used by the cgroup. Used for the Pod’s resource tracking.

The kubelet’s default threshold is memory.available< 100Mi. The threshold is the available memory; when the available memory is below 100Mi, the kubelet considers the node to be in memory pressure.

The memory pressure scenario

A node with 64Gi of memory is running 100 Pods with 8Gi each. The kubelet’s memory.available is 0. The node is in memory pressure.

flowchart TD
    A[Node memory: 64Gi] --> B[100 Pods with 8Gi each]
    B --> C[memory.available: 0]
    C --> D[MemoryPressure=True]
    D --> E[Eviction loop]
    E --> F[Evict BestEffort Pods]
    F --> G[memory.available: 4Gi]
    G --> H{Below threshold?}
    H -->|No| I[Evict Burstable Pods]
    H -->|Yes| J[Done]

The eviction loop evicts the BestEffort Pods first. If the memory is still below the threshold, the loop evicts the Burstable Pods. The loop continues until the memory is above the threshold.

The reclaim logic

The eviction loop’s reclaim logic:

flowchart TD
    A[Eviction loop] --> B[Get Pod list]
    B --> C[Sort by QoS class]
    C --> D[BestEffort first]
    D --> E[Calculate memory usage]
    E --> F[Pod over its requests?]
    F -->|Yes| G[Evict Pod]
    F -->|No| H[Skip Pod]
    G --> I[Update memory.available]
    I --> J{Above threshold?}
    J -->|No| B
    J -->|Yes| K[Done]

The loop iterates over the Pods, evicting the Pods that are over their requests. The Pod’s actual usage is compared to the request; the Pod is evicted if the usage is over the request.

The eviction is bounded by the loop’s iteration count. The kubelet’s --eviction-max-pod-grace-period flag sets the maximum grace period for the Pod’s termination.

The BestEffort eviction

The BestEffort Pods are the first to be evicted. A BestEffort Pod has no resource requests; the kubelet evicts the Pod without comparing its usage to a request.

The BestEffort eviction is destructive. The Pod’s containers are sent SIGTERM; the kubelet waits for the grace period; the Pod is terminated.

The BestEffort Pod’s state is lost. The cluster’s controller (Deployment, StatefulSet, etc.) creates a replacement Pod; the replacement Pod is scheduled by the scheduler.

The Burstable eviction

The Burstable Pods are the next to be evicted. A Burstable Pod has some resource requests; the kubelet evicts the Pod if its usage is over its requests.

The Burstable eviction is destructive. The kubelet ranks the Burstable Pods by how far each one’s usage exceeds its request and evicts the worst offender first.

The Burstable Pod’s state is lost. The cluster’s controller creates a replacement Pod; the replacement Pod is scheduled by the scheduler.

The Guaranteed eviction

The Guaranteed Pods are the last to be evicted. A Guaranteed Pod has requests equal to limits. The kubelet evicts the Pod if its usage is over its limits (the request is the same as the limit).

The Guaranteed eviction is destructive. The kubelet sets the Pod’s phase to Failed with reason Evicted and a message naming the exhausted resource; the object stays on the API server until the Pod garbage collector removes it.

The Guaranteed eviction is the last resort. The kubelet’s memory pressure is severe; the cluster’s capacity is exhausted.

The reclaim minimum

The kubelet’s --eviction-minimum-reclaim flag sets the minimum amount of memory to reclaim per eviction cycle. The default is 0.

The flag is the safety net. The kubelet evicts the Pods until the memory is below the threshold; the flag ensures the kubelet does not stop the eviction while the memory is still below the threshold.

The flag is configurable:

kubelet \
  --eviction-minimum-reclaim=memory.available=500Mi

The reclaim is in absolute values (500Mi), not percentages. The flag is the operator’s tuning for the reclaim logic.

The memory pressure’s diagnostic

A node in memory pressure has the MemoryPressure=True condition:

NODE=worker-03            # node name from `kubectl get nodes`

kubectl describe node "$NODE" | grep -A 5 "Conditions"

Conditions:
  Type                 Status  Reason
  ----                 ------  ------
  MemoryPressure       True    KubeletHasInsufficientMemory

The diagnostic:

journalctl -u kubelet | grep -i "memory"

The kubelet logs the eviction. The fix is to:

1. Identify the workload. The kubelet’s logs show the evicted Pods.
2. Investigate the workload. The Pod’s memory usage is the cause.
3. Add capacity. The cluster may need more nodes.
4. Reduce the workload’s memory usage. The workload may be leaking memory.

The memory pressure’s metrics

The kubelet’s metrics expose the memory pressure:

NODE_IP=192.0.2.31        # address of the node's kubelet

curl -k "https://$NODE_IP:10250/metrics" | grep memory

The relevant metrics:

• kubelet_memory_evictions_total
• kubelet_memory_reserved_bytes
• node_memory_MemAvailable_bytes
• node_memory_MemTotal_bytes

The operator should monitor the metrics and alert on the eviction rate. A rising eviction rate is a cluster that is losing memory.

The memory pressure’s operational patterns

The memory pressure’s operational patterns:

• Set the Pod’s resource requests. The eviction order is based on the QoS class. The Pod’s resource requests determine the QoS class.
• Set the memory limits. The limits are the Pod’s memory ceiling. The Pods that exceed the limits are OOMKilled.
• Monitor the memory usage. The metrics expose the memory usage; the operator should alert on the threshold.
• Audit the memory usage at every release. A new workload that is using more memory than expected is a workload that needs investigation.

Quiz

Production discipline

• Memory pressure is the cluster’s signal. The kubelet’s MemoryPressure=True is the cluster’s signal that the node is exhausting memory.
• The eviction order is the QoS class. A Guaranteed Pod is the last to be evicted; a BestEffort Pod is the first.
• Tune the thresholds for the workload. The defaults are conservative; a production cluster may want to lower the thresholds.
• Monitor the eviction metrics. The cluster’s alerts should fire on the eviction rate. A rising rate is a cluster that is losing memory.
• Audit the memory usage at every release. A new workload that is using more memory than expected is a workload that needs investigation.
• Test the eviction in non-production. A staging cluster that mirrors production is the right place to test the eviction logic.