Eviction order and QoS — node pressure and pod survival

The kubelet’s eviction algorithm is what protects nodes from running out of memory, disk, or PIDs. This lesson covers the algorithm, the eviction signals, and how to tune thresholds to match production workloads.

The eviction algorithm

flowchart TD
    Monitor[Kubelet monitors node resources] --> Threshold{Threshold crossed?}
    Threshold -- no --> Monitor
    Threshold -- yes --> Soft{Soft threshold?}
    Soft -- yes --> Grace[Wait grace period]
    Soft -- no --> Act[Act immediately]
    Grace --> Persistent{Pressure persists?}
    Persistent -- no --> Reset[Reset; no eviction]
    Persistent -- yes --> Act
    Act --> Sort[Sort Pods by QoS + usage]
    Sort --> Evict[Evict Pods until pressure relieved]
    Evict --> Monitor

The kubelet:

1. Monitors node resources (memory, disk, inodes, imagefs, PIDs).
2. Detects when a resource crosses an eviction threshold.
3. For soft thresholds, waits the grace period to see if pressure resolves.
4. Sorts Pods by QoS class and usage vs request.
5. Evicts Pods one at a time until pressure is relieved.
6. Repeats until resource is above threshold.

The pressure relief loop is important: the kubelet evicts the minimum number of Pods needed to bring the resource above threshold.

Eviction signals

The kubelet watches these signals:

evictionHard:
  memory.available: 500Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
  imagefs.available: 15%
  pid.available: 10%

The eviction order: QoS first

Within the eviction loop, Pods are sorted by:

1. QoS class: BestEffort first, Burstable second, Guaranteed last.
2. Within Burstable: Pods with the highest usage / request ratio are evicted first.
3. Within Guaranteed: same logic, but Guaranteed Pods are evicted last overall.

flowchart TD
    Sort[Sort Pods] --> Q1{QoS?}
    Q1 -- BestEffort --> S1[Sort by absolute usage descending]
    Q1 -- Burstable --> S2[Sort by usage/request ratio descending]
    Q1 -- Guaranteed --> S3[Sort by usage/request ratio descending]
    S1 --> Evict[Evict in order]
    S2 --> Evict
    S3 --> Evict

For a node with 30% memory pressure:

• All BestEffort Pods are evicted first (regardless of usage).
• Then Burstable Pods with the highest usage-vs-request ratio.
• If still under pressure, Guaranteed Pods with the highest ratio.

Hard vs soft thresholds

Two threshold types:

• Hard (evictionHard): when the threshold is crossed, the kubelet evicts immediately. No grace period.
• Soft (evictionSoft): when the threshold is crossed, the kubelet waits the grace period. If pressure resolves during the grace period, no eviction. If not, eviction proceeds.

evictionHard:
  memory.available: 500Mi

evictionSoft:
  memory.available: 1Gi

evictionSoftGracePeriod:
  memory.available: 30s

The pattern:

• Hard threshold: the safety net. When memory is critically low, evict immediately.
• Soft threshold: early warning. When memory is getting low, give the workload 30s to recover (e.g., a garbage collection cycle).

Tuning eviction thresholds

Production discipline:

• Memory hard threshold: 500Mi-1Gi. Lower means earlier eviction but more Pods lost; higher means later eviction but more risk of OOMKill.
• Memory soft threshold: 1-2Gi. Earlier warning with grace period.
• Disk hard threshold: 10-15%. Lower means earlier eviction when emptyDir fills.
• Imagefs hard threshold: 15-20%. The image filesystem fills up with cached image layers; eviction cleans the cache.

evictionHard:
  memory.available: 500Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
  imagefs.available: 15%
  pid.available: 10%
evictionSoft:
  memory.available: 1Gi
  nodefs.available: 15%
evictionSoftGracePeriod:
  memory.available: 30s
  nodefs.available: 30s

Eviction signals in alerts

# Node has memory pressure condition
kube_node_status_condition{condition="MemoryPressure",status="true"} == 1

# Node has disk pressure condition
kube_node_status_condition{condition="DiskPressure",status="true"} == 1

# Pods evicted in last hour
increase(kube_pod_container_status_last_terminated_reason{reason="Evicted"}[1h])

Production discipline: alert on MemoryPressure and DiskPressure conditions; track eviction rates per node.

The kubelet eviction manager

The eviction manager runs as part of the kubelet:

• Reads the kubelet config for thresholds.
• Polls node resources (memory.available, nodefs.available, etc.).
• Triggers eviction when thresholds are crossed.
• Logs each eviction with the reason and the Pod’s QoS.

journalctl -u kubelet | grep -i evict
# I0815 12:00:00.123 ... eviction manager: must evict pod(s) to reclaim memory
# I0815 12:00:01.456 ... eviction manager: pod web-7c8 (QoS: Burstable) is evicted

The kubelet logs are the ground truth for what was evicted and why.

Production patterns

Set eviction thresholds in the kubelet config:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
  memory.available: 500Mi
  nodefs.available: 10%
evictionSoft:
  memory.available: 1Gi
evictionSoftGracePeriod:
  memory.available: 30s

Use node-level alerts:

- alert: NodeMemoryPressure
  expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
  for: 5m
  labels:
    severity: warning

- alert: HighEvictionRate
  expr: increase(kube_pod_container_status_last_terminated_reason{reason="Evicted"}[1h]) > 5
  labels:
    severity: warning

Investigate evicted Pods:

kubectl get pods -A --field-selector=status.reason=Evicted
# Identify Pods that were evicted; check their QoS and usage

Cross-course references

• The Linux course part XXXVII-Linux-Resources covers cgroup OOM and node memory pressure; eviction is the cluster-level equivalent.
• The Ansible course part XXXV-Ansible-Scripting covers service priority; QoS-based eviction is the cluster-level equivalent.
• The Observability course part LXXXIV-Kubernetes-CapacityPlanning covers capacity planning; eviction is the response to capacity planning failures.

Quiz

Production discipline

• Set both hard and soft eviction thresholds. Soft gives grace periods; hard is the safety net.
• Tune thresholds to match workload peak. Memory threshold must be below the workload’s peak burst.
• Monitor MemoryPressure and DiskPressure conditions. These signal that the kubelet has begun eviction.
• Track eviction rates per node. High eviction rates indicate over-commitment or workload issues.
• Plan capacity so Guaranteed Pods are not at risk. QoS is the order, not a guarantee.