Disk pressure eviction — imagefs, logs, and the node filesystem

Disk pressure eviction is the kubelet’s response to filesystem exhaustion. The kubelet distinguishes the node’s root filesystem (nodefs) from the runtime’s image filesystem (imagefs); the eviction is based on the worst signal. This lesson walks the detection, the eviction loop, and the operational patterns.

The two filesystems

The kubelet distinguishes two filesystems:

• nodefs: the node’s root filesystem. Default /. Contains the kubelet’s data, the runtime’s mountpoints, and the logs.
• imagefs: the runtime’s image filesystem. Default /var/lib/containerd for containerd. Contains the image cache.

The two may be on the same filesystem or on separate filesystems. The production pattern is to mount the two on separate volumes; the isolation prevents the image cache from filling the node’s root filesystem.

# /etc/fstab
UUID=abc /var/lib/containerd ext4 defaults 0 0

The kubelet’s configuration file can specify the imagefs path:

imagefs:
  - path: /var/lib/containerd

The default is /var/lib/containerd for containerd.

The disk pressure thresholds

The kubelet’s disk pressure thresholds:

The thresholds are the kubelet’s detection limits. The production rule is to set the thresholds to values that give the operator time to react.

The kubelet’s configuration:

# The `<` is part of the threshold expression, not a redirection,
# so each value has to be quoted.
kubelet \
  --eviction-hard='nodefs.available<10%' \
  --eviction-hard='imagefs.available<15%'

The thresholds are the bytes and inodes free. The percentages are converted to bytes by the kubelet.

The disk eviction loop

The kubelet’s disk eviction loop:

flowchart TD
    A[Eviction loop] --> B[Check nodefs]
    B --> C{Below threshold?}
    C -->|Yes| D[Evict Pods by disk usage]
    D --> E[Reclaim nodefs]
    C -->|No| F[Check imagefs]
    F --> G{Below threshold?}
    G -->|Yes| H[Garbage collect images]
    H --> I[Reclaim imagefs]
    G -->|No| J[Done]

The loop distinguishes the two filesystems. The nodefs eviction is based on the Pod’s disk usage; the imagefs eviction is based on the runtime’s image cache.

The nodefs eviction

The nodefs eviction is based on the Pod’s disk usage. The kubelet tracks the Pod’s disk usage through the container’s writable layer and the Pod’s volumes.

The kubelet’s eviction order for the nodefs:

1. BestEffort Pods: evicted first.
2. Burstable Pods: evicted next.
3. Guaranteed Pods: evicted last.

The eviction is the same as the memory eviction. The kubelet evicts the Pods based on the QoS class and the resource usage.

The imagefs eviction

The imagefs eviction is based on the runtime’s image cache. The kubelet’s image GC removes the least-recently-used images until the imagefs is below the threshold.

kubelet --image-gc-high-threshold=85

The image GC is the kubelet’s image cache management. The kubelet’s --image-gc-high-threshold is the upper bound; the kubelet’s --image-gc-low-threshold is the lower bound.

The image GC is independent of the eviction loop. The image GC runs every 5 minutes; the eviction loop runs every 10 seconds.

The combined eviction

The kubelet’s eviction loop combines the nodefs and imagefs pressures. The eviction is based on the worst signal:

flowchart TD
    A[Eviction loop] --> B[Read nodefs and imagefs]
    B --> C[Calculate the worst signal]
    C --> D{Below threshold?}
    D -->|Yes| E[Evict Pods]
    D -->|No| F[Done]

The worst signal is the filesystem that is the most constrained. The kubelet evicts the Pods based on the worst signal.

The log eviction

The kubelet’s log rotation is the disk pressure’s front-line defence. The kubelet rotates the container’s log file when the size exceeds the --log-file-max-size (default 100Mi). The kubelet keeps at most --log-file-max-count (default 5) log files.

A noisy container that fills the log file is rotated automatically. The kubelet’s log rotation is the safety net.

The volume eviction

The kubelet’s volume cleanup is the orphaned volume’s protection. The kubelet removes the orphaned volume directories after the Pod is evicted.

The volume cleanup is the kubelet’s last defence against orphaned volumes. A CSI driver that fails to clean up the volume is a volume that fills the node’s filesystem.

The disk pressure scenario

A node with 100Gi of nodefs is running 100 Pods with 1Gi of ephemeral-storage each. The kubelet’s nodefs.available is 0. The node is in disk pressure.

flowchart TD
    A[Node filesystem: 100Gi] --> B[100 Pods with 1Gi each]
    B --> C[nodefs.available: 0]
    C --> D[DiskPressure=True]
    D --> E[Eviction loop]
    E --> F[Evict Pods by disk usage]
    F --> G[nodefs.available: 10Gi]
    G --> H{Above threshold?}
    H -->|No| I[Evict more Pods]
    H -->|Yes| J[Done]

The eviction loop evicts the Pods based on the disk usage. The BestEffort Pods are evicted first; the Burstable Pods are next; the Guaranteed Pods are last.

The disk pressure’s diagnostic

A node in disk pressure has the DiskPressure=True condition:

# Substitute your own node name:
NODE=node-18

kubectl describe node "$NODE" | grep -A 5 "Conditions"

Conditions:
  Type                 Status  Reason
  ----                 ------  ------
  DiskPressure         True    KubeletHasNoDiskSpace

The diagnostic:

journalctl -u kubelet | grep -i "disk"
df -h / /var/lib/containerd

The kubelet logs the eviction. The fix is to:

1. Identify the workload. The kubelet’s logs show the evicted Pods.
2. Investigate the workload. The Pod’s disk usage is the cause.
3. Clean up the image cache. The runtime’s image cache may be filling the filesystem.
4. Clean up the container logs. The container logs may be filling the filesystem.
5. Add disk capacity. The cluster may need more disk.

The disk pressure’s metrics

The kubelet’s metrics expose the disk pressure:

# The kubelet's address; substitute your own node's IP:
NODE_IP=192.0.2.18

curl -k "https://$NODE_IP:10250/metrics" | grep disk

The relevant metrics:

• kubelet_disk_evictions_total
• kubelet_image_filesystem_available_bytes
• kubelet_image_filesystem_capacity_bytes
• kubelet_node_filesystem_available_bytes
• kubelet_node_filesystem_capacity_bytes

The operator should monitor the metrics and alert on the eviction rate. A rising eviction rate is a cluster that is losing disk.

Quiz

Production discipline

• Disk pressure is the cluster’s signal. The kubelet’s DiskPressure=True is the cluster’s signal that the node is exhausting disk.
• The two filesystems are distinct. The node’s root filesystem and the image filesystem are separate. The production rule is to mount them on separate volumes.
• Tune the thresholds for the workload. The defaults are conservative; a production cluster may want to lower the thresholds.
• Monitor the eviction metrics. The cluster’s alerts should fire on the eviction rate. A rising rate is a cluster that is losing disk.
• Audit the disk usage at every release. A new workload that is using more disk than expected is a workload that needs investigation.
• Test the eviction in non-production. A staging cluster that mirrors production is the right place to test the eviction logic.