DiskPressure evicts pods

Reported symptoms

At 09:40 the platform channel takes three pages in four minutes, and they read like three separate incidents.

The observability team reports that their log-shipper DaemonSet Pod on node-18 has disappeared. The DaemonSet controller recreates it; nine minutes later it is gone again. The documents team reports that thumbnail rendering has stalled and their Pods on the same node keep coming back as Failed. A third team reports that a routine rollout will not place a single Pod on node-18, and their kubectl describe pod shows the scheduler rejecting the node.

Nobody pages the mail team, whose ingest spooler is by a wide margin the largest storage consumer on that node. Its Pods are running normally.

The capacity dashboard is no help. node-18 root filesystem has been flat at 58 percent used for a week. The disk-space alert has a threshold at 85 percent and has never fired. kubectl get nodes shows node-18 as Ready, so the node-health alert has not fired either. As far as every dashboard in the estate is concerned, this node is healthy and has 173 GiB spare.

Then a fourth report arrives that does not fit the pattern at all: an image pull on node-18 failed with no space left on device. The engineer who takes that one checks df, sees 173 GiB available, and concludes the runtime is broken.

Evidence provided

$ kubectl describe node node-18 | grep -A4 'Conditions:' ; kubectl describe node node-18 | grep -A2 'Taints:'

Conditions:
Type             Status  Reason                   Message
----             ------  ------                   -------
DiskPressure     True    KubeletHasDiskPressure   kubelet has disk pressure
Taints:            node.kubernetes.io/disk-pressure:NoSchedule

$ ssh node-18 'df -h /'

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  412G  239G  173G  58% /

$ ssh node-18 'df -i /'

Filesystem        Inodes    IUsed  IFree IUse% Mounted on
/dev/nvme0n1p2  27394048 27373120  20928  100% /

$ ssh node-18 'journalctl -u kubelet --since -1h | grep -i eviction | head -5'

eviction_manager: attempting to reclaim  resourceName="inodes"
eviction_manager: must evict pod(s) to reclaim  resourceName="inodes"
eviction_manager: pods ranked for eviction  pods=[observability/log-shipper-8k2vq docs/thumbnailer-6d9f4-w7lqz ...]

$ ssh node-18 'findmnt -no SOURCE,TARGET /var/lib/containerd /var/lib/kubelet /'

/dev/nvme0n1p2 /var/lib/containerd
/dev/nvme0n1p2 /var/lib/kubelet
/dev/nvme0n1p2 /

$ ssh node-18 'du --inodes -x -d 1 /var/lib/kubelet/pods /var/lib/containerd 2>/dev/null | sort -n | tail -3'

  7614402 /var/lib/containerd
19412663 /var/lib/kubelet/pods
27027065 total

$ kubectl -n mail-ingest get deploy ingest-spooler -o jsonpath='{.spec.template.spec.containers[0].resources}'

{"limits":{"ephemeral-storage":"100Gi"},"requests":{"ephemeral-storage":"80Gi"}}

Work the evidence before reading on

Three teams were paged and none of them owns the cause. Everything on the node is reporting accurately.

1. df -h and df -i describe the same filesystem. What is each one counting, and which of the two does the kubelet act on?
2. The image-pull failure said no space left on device with 173 GiB free. What else can produce ENOSPC on a filesystem that has free blocks?
3. The spooler declares an 80Gi ephemeral-storage request and was not evicted. The log shipper declares no ephemeral-storage request and was evicted first. What quantity is the kubelet comparing when it ranks candidates?

Before continuing: the node correctly detected that it was out of a resource, and then evicted Pods that were not consuming that resource. What single sentence explains both halves at once?

Root cause

1. One filesystem, two resources, four signals

The kubelet’s disk eviction signals are nodefs.available, nodefs.inodesFree, imagefs.available and imagefs.inodesFree. The default hard threshold for nodefs.available is 10 percent free and for nodefs.inodesFree is 5 percent free. Any one of them crossing sets DiskPressure and applies the node.kubernetes.io/disk-pressure:NoSchedule taint.

Bytes were never in trouble on node-18: 42 percent of the volume was unused. Inodes were. The filesystem was made with roughly 27.4 million inodes, that number is fixed at mkfs time and cannot be raised on a live filesystem, and 20,928 of them were left. That is 0.08 percent free against a 5 percent threshold.

Everything that followed is the node behaving exactly as designed. The DiskPressure condition was correct. The taint was correct, which is why the third team’s rollout would not place. The ENOSPC on the image pull was correct too: a filesystem out of inodes cannot create a file however many free blocks it has, and the error the kernel returns is the same one it returns when blocks are exhausted. That single error string is why one engineer spent twenty minutes investigating the container runtime.

2. The ranking is a byte comparison

The second half of the puzzle is separable from the first, and it is the part that sent three teams to the wrong place.

When the kubelet has to reclaim disk it ranks candidates by how far each Pod’s local ephemeral-storage usage exceeds its ephemeral-storage request. A Pod over its request is considered before a Pod under it. A Pod that declares no ephemeral-storage request has an effective request of zero, so any usage at all puts it over.

That comparison is in bytes. It has to be - ephemeral-storage is a byte quantity.

So the spooler, holding 19.4 million files worth roughly 74 GiB against an 80Gi request, is measured as under its request and ranks last. The log-shipper DaemonSet, holding a few hundred megabytes of buffered logs and declaring no ephemeral-storage request at all, is over its request of zero and ranks first. The thumbnailer is in the same position. Neither of them was consuming the resource that was exhausted, and neither of them could have relieved it.

The node was out of inodes and ranked the candidates by bytes.

3. Nothing in the estate was watching the other number

node_filesystem_avail_bytes and node_filesystem_files_free are two separate series from the same exporter, describing two separate resources on the same mount. Every dashboard and every alert in this estate was built on the first one. The inode count had been climbing for weeks and was visible the entire time to anyone who queried the second.

Resolution

1. Cordon node-18. The eviction loop is currently recreating Pods onto a node that cannot host them, and each cycle costs another team a page. kubectl cordon node-18 stops the churn without touching the cause.
2. Hold, and give the hold an owner and a clock. The spool directories contain queued messages; deleting them is a data decision that belongs to the mail-ingest on-call, not to the platform engineer holding the pager. The question they must answer is "are these messages replayable from the upstream queue?" If there is no answer within thirty minutes, drain the remaining workloads to healthy nodes and leave node-18 cordoned and full rather than guessing.
3. Run the cheap reclaim while waiting, and record what it returned. crictl rmi --prune on the node frees unused image layers. Expect it to help and not to be sufficient; the number it returns is evidence for the postmortem either way.
4. Once the owner confirms the spool is replayable, delete the spooler Pods on node-18. Their emptyDir directories are removed with the Pod objects.
5. Delete the Failed Pods the eviction left behind, so nothing is holding a directory: kubectl delete pod --field-selector status.phase=Failed scoped to the affected namespaces.
6. Watch df -i / on the node rather than assuming. The confirmation that free inodes climbed is the step; the deletions were only the attempt.
7. Uncordon node-18 only once the inode count is back under the threshold with margin and has stayed there through a full eviction-loop interval.
8. File the workload change before closing the incident. Batching messages into fewer files, or spooling to a PersistentVolume, is what actually bounds this; everything above is cleanup.

Verification

1. df -i / on node-18 shows IUse% well clear of the 95 percent mark that the default nodefs.inodesFree<5% corresponds to, sampled twice at least a minute apart rather than once mid-cleanup.
2. The Node object agrees. kubectl describe node node-18 reports DiskPressure False, and the node.kubernetes.io/disk-pressure taint is gone. These are two separate observations: the taint is what the scheduler reads.
3. No Pod on node-18 acquires status.reason: Evicted in the following hour.
4. An image pull on node-18 succeeds. That failure was the same exhaustion in different words, so it is a real check rather than a courtesy one.
5. The free-inode alert fires when it should. Fill a scratch filesystem on a staging node until the inode threshold is crossed and confirm the alert arrives. An alert that has never fired has not been tested.
6. On a staging node, the reconfigured spooler runs for a full working day and its inode count plateaus instead of climbing. This is the only check that speaks to the cause rather than the symptom.

Prevention

• Alert on free inodes alongside free bytes. Both numbers come from the same exporter and describe the same mount:

node_filesystem_files_free{mountpoint="/"} / node_filesystem_files{mountpoint="/"} < 0.15

• Give the container runtime its own volume, so the image store and workload scratch space stop competing for one inode table. The change needs a node drain, so schedule it as maintenance rather than attempting it during an incident.
• Set ephemeral-storage requests and limits on every workload that writes to an emptyDir, and be precise about what that buys. It bounds bytes, it makes the Pod visible to the eviction ranking, and it does not bound file count.

resources:
  requests:
    ephemeral-storage: 2Gi
  limits:
    ephemeral-storage: 4Gi

• Treat “writes one file per unit of work” as a design review item for any workload that will run on shared nodes. A queue depth that is normally in the thousands and is occasionally in the millions is an inode incident waiting for a backlog.
• Record the signal, not the condition, in incident notes and alerts. DiskPressure names a category. nodefs.inodesFree names the fault.