Eviction fundamentals — soft, hard, and the kubelet's loop

The kubelet’s eviction subsystem is the node’s last defence against resource pressure. When the node’s memory, disk, or PIDs are exhausted, the kubelet evicts Pods to reclaim the resources. This lesson walks the soft and hard thresholds, the eviction loop, and the operational patterns.

The soft and hard thresholds

The kubelet’s eviction thresholds are configurable. The two types:

• Hard threshold: the kubelet evicts Pods immediately when the threshold is exceeded. The threshold is a hard limit.
• Soft threshold: the kubelet evicts Pods when the threshold is exceeded for the grace period. The threshold is a soft limit; the grace period is the time the threshold is allowed to be exceeded.

kubelet \
  --eviction-hard=memory.available<100Mi \
  --eviction-soft=memory.available<500Mi \
  --eviction-soft-grace-period=30s

The hard threshold is the immediate eviction. The soft threshold is the gradual eviction. The two are complementary: the soft threshold gives the cluster time to react; the hard threshold is the safety net.

The eviction signals

The kubelet’s eviction signals:

The signals are the kubelet’s measurement of the node’s state. The signals are configurable; the cluster’s production rule is to tune the signals for the workload.

The eviction loop

The kubelet’s eviction loop runs every 10 seconds (the --eviction-pressure-transition-period flag). The loop:

sequenceDiagram
    autonumber
    participant K as kubelet
    participant API as API server

    K->>K: check eviction signals
    Note over K: thresholds exceeded?
    K->>K: identify Pods to evict
    K->>K: sort Pods by QoS class
    K->>API: evict Pod 1 (BestEffort)
    API-->>K: evict API
    K->>API: evict Pod 2 (Burstable)
    API-->>K: evict API
    K->>API: evict Pod 3 (Guaranteed)
    API-->>K: evict API
    K->>K: check signals again
    Note over K: pressure relieved?

The loop continues until the pressure is below the threshold. The eviction is not destructive; the kubelet uses the API server’s eviction API, which sends a graceful termination notice to the Pod.

The eviction API

The kubelet uses the API server’s eviction API to evict Pods. The eviction API is a POST to /api/v1/namespaces/<ns>/pods/<name>/eviction. The API server sends a SIGTERM to the Pod’s containers.

# Bearer token for a ServiceAccount allowed to create pods/eviction:
TOKEN=$(kubectl create token default)

curl -X POST https://api.example.com:6443/api/v1/namespaces/default/pods/test-1/eviction \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"apiVersion":"policy/v1","kind":"Eviction","metadata":{"name":"test-1","namespace":"default"}}'

The API server’s eviction API is the standard mechanism for evicting Pods. The kubelet uses the same API to evict Pods; the operator uses the same API to evict Pods manually.

The eviction order

The kubelet evicts Pods in a specific order based on the QoS class:

flowchart TD
    A[Pod list] --> B{BestEffort?}
    B -->|Yes| C[Evict first]
    B -->|No| D{Burstable?}
    D -->|Yes| E[Evict second]
    D -->|No| F{Guaranteed?}
    F -->|Yes| G[Evict last]

The order is:

1. BestEffort Pods: Pods with no resource requests. The first to be evicted.
2. Burstable Pods: Pods with some resource requests. The next to be evicted.
3. Guaranteed Pods: Pods with requests equal to limits. The last to be evicted.

The eviction order is the kubelet’s heuristic. The heuristic is not configurable; the kubelet’s eviction is based on the Pod’s QoS class.

The eviction’s resource calculations

The kubelet’s eviction loop calculates the resource usage of each Pod. The calculation considers:

• The Pod’s resource requests.
• The Pod’s resource limits.
• The Pod’s actual usage.

The kubelet evicts the Pod with the highest resource usage relative to its requests. A Pod that is using more than its requests is evicted before a Pod that is using less.

The calculation is approximate; the kubelet uses the metrics from the CRI to estimate the usage.

The eviction’s effects

The eviction’s effects:

• The Pod’s Status.Phase becomes Failed.
• The Pod’s Status.Reason is Evicted.
• The Pod’s controller (Deployment, StatefulSet, etc.) creates a replacement Pod.
• The replacement Pod is scheduled by the scheduler.

The eviction is destructive. The Pod’s state is lost. The Pod’s volume is preserved (the PVC is still bound).

The eviction is not graceful. The kubelet sends the eviction API call; the API server sends the SIGTERM. The Pod’s terminationGracePeriodSeconds is the grace period.

The eviction’s metrics

The kubelet’s metrics expose the eviction:

# Address of the node whose kubelet you are scraping:
NODE_IP=192.0.2.31

curl -k "https://$NODE_IP:10250/metrics" | grep evictions

The relevant metrics:

• kubelet_containers_evicted_total
• kubelet_pods_evicted_total
• kubelet_memory_evictions_total
• kubelet_disk_evictions_total
• kubelet_pid_evictions_total

The operator should monitor the metrics and alert on the eviction rate. A rising eviction rate is a cluster that is losing capacity.

The eviction’s tuning

The eviction’s tuning:

• Set the thresholds deliberately. The defaults are conservative; a production cluster may want to lower the thresholds to give the operator more time to react.
• Set the grace period deliberately. The soft threshold’s grace period is the time the cluster is allowed to be above the threshold. The default is 30s.
• Use the --eviction-minimum-reclaim flag. The flag sets the minimum amount of resource to reclaim per eviction. The default is 0.
• Use the --eviction-max-pod-grace-period flag. The flag sets the maximum grace period for the Pod’s termination. The default is 30s.

The cluster’s production rule is to tune the eviction for the workload’s resource pressure profile.

Quiz

Production discipline

• Soft thresholds are the right default. The soft threshold gives the cluster time to react. The hard threshold is the safety net.
• Tune the thresholds for the workload. The defaults are conservative; a production cluster may want to lower the thresholds.
• The eviction order is the QoS class. A Guaranteed Pod is the last to be evicted; a BestEffort Pod is the first.
• Monitor the eviction metrics. The cluster’s alerts should fire on the eviction rate. A rising rate is a cluster that is losing capacity.
• Audit the thresholds at every node repave. A new node that joins the cluster with the wrong thresholds is a node that is missing the pressure. The audit catches the failure.
• Test the eviction in non-production. A staging cluster that mirrors production is the right place to test the eviction logic.