MemoryPressure eviction

Reported symptoms

For three weeks, Pods on worker-07 have been dying a few times an hour. Their phase is Failed, their reason is Evicted, and they belong to three unrelated namespaces: a CI runner pool, a batch queue, and the platform namespace that carries the node monitoring agent.

Every investigation so far has bounced:

• Nothing was OOMKilled. No exit code 137, no container reached its own memory limit, and the application teams can show flat memory graphs across every window in which they lost a Pod.
• kubectl top node worker-07, run after each page, reports memory around 70%. That reads as ample headroom, so capacity planning closes the ticket - and it is only comfortable because the evictions have already reclaimed the memory by the time anyone looks.
• The node’s MemoryPressure condition goes True and then back to False within a couple of minutes. By the time anyone runs describe node after a page, the node looks perfectly healthy.
• The node monitoring agent is itself one of the things being evicted, so the node’s memory graph has holes precisely in the intervals under investigation.

Two other threads are running in parallel, and neither team knows about the other. The batch namespace has started hitting its ResourceQuota Pod count and has raised a ticket about being unable to create Pods; nobody has noticed the quota is being consumed by dozens of Evicted Pod objects that were never cleaned up. And three weeks ago the platform team rolled out a log shipper to every node in the cluster - which is dismissed early on, because it went everywhere and only two nodes misbehave.

Evidence provided

$ kubectl get events -A --field-selector reason=Evicted --sort-by=.lastTimestamp | tail -5

batch     4m    Warning   Evicted   pod/import-runner-4q8sd   The node was low on resource: memory. Threshold quantity: 100Mi, available: 84Mi.
ci        9m    Warning   Evicted   pod/runner-77c9f6d4b-x2plq   The node was low on resource: memory. Threshold quantity: 100Mi, available: 91Mi.
platform  9m    Warning   Evicted   pod/node-agent-l8kkr          The node was low on resource: memory. Threshold quantity: 100Mi, available: 91Mi.
batch     23m   Warning   Evicted   pod/import-runner-mm4vz       The node was low on resource: memory. Threshold quantity: 100Mi, available: 88Mi.

$ kubectl describe node worker-07 | grep -A 6 -E '^(Capacity|Allocatable)'

Capacity:
cpu:                16
memory:             65806132Ki
pods:               110
Allocatable:
cpu:                16
memory:             65703732Ki
pods:               110

$ kubectl describe node worker-07 | grep -A 5 'Allocated resources'

Allocated resources:
Resource           Requests      Limits
--------           --------      ------
cpu                6200m (38%)   19400m (121%)
memory             34Gi (54%)    58Gi (92%)

# systemd-cgtop -m --iterations=1 -n0 | head -6

Control Group                    Tasks   %CPU   Memory  Input/s Output/s
/                                 1284    412    60.9G        -        -
/kubepods.slice                   1102    355    52.1G        -        -
/system.slice                      164     44     8.4G        -        -
/system.slice/logshipper.service    38     29     5.8G        -        -
/system.slice/containerd.service    92      9     1.1G        -        -

$ kubectl get pod -n ci runner-77c9f6d4b-x2plq -o jsonpath='{.status.qosClass}'

BestEffort

$ kubectl get pods -A --field-selector status.phase=Failed -o wide | grep -c worker-07

61

Work the evidence before reading on

Nothing here contradicts anything else. Every number is correct. The node is short of memory and the scheduler thinks it is half empty, and both of those are true at the same time.

1. Compare Capacity and Allocatable for memory. What is the difference, and what is it supposed to cover?
2. The scheduler places Pods using requests against allocatable. Which memory consumers on this node does that arithmetic include, and which does it not?
3. The evicted Pods are BestEffort. What does the kubelet use QoS class for, and does that ranking say anything about which Pod caused the shortage?
4. Only two nodes in the fleet misbehave, and the agent was rolled out to all of them. What is different about those two?

Before continuing: the kubelet can only evict Pods. What happens when the thing consuming the memory is not one?

Root cause

1. Allocatable is a promise the node cannot keep

Allocatable is what the kubelet advertises to the scheduler. It is meant to be capacity minus what the operating system needs, minus what the kubelet and the container runtime need, minus the eviction threshold that keeps the node off the cliff.

On this node it is capacity minus 100Mi. kubeReserved and systemReserved are unset, so the node is telling the scheduler that everything on the host outside of Pods runs on 100 megabytes: systemd, journald, sshd, the kubelet itself, containerd, the CNI agent and the log shipper.

That was survivable for a long time because there was slack. The node was never full, so the gap between the promise and reality was absorbed by memory nobody had claimed.

2. The agent grew into the slack, on exactly two nodes

The log shipper’s working set scales with log volume. It was rolled out everywhere, which is why the rollout was dismissed as a suspect: if it were the cause, surely every node would break.

But only the two busiest log producers in the cluster ever grew it past the slack. On every other node the same agent, from the same rollout, sits well inside the unclaimed memory and is invisible. The rollout did not create the defect - the defect was the missing reservation - but it is what turned a latent mis-sizing into a daily incident, and only where the load was highest.

3. The kubelet defends the node with the only tool it has

When memory.available crosses the threshold, the kubelet evicts. That is the correct behaviour and it works: memory is reclaimed, pressure clears, the condition flips back to False.

It also cannot possibly fix anything. The memory is being held by a systemd unit; the kubelet has no authority over it and evicts Pods instead. So the node recovers, reports itself healthy, the scheduler refills the space that was just freed, the agent keeps growing, and the whole thing repeats. The flapping condition is not a symptom of instability - it is the loop working exactly as designed against a cause it cannot reach.

4. The ranking chose the worst possible victims

Eviction order is BestEffort first, ranked by absolute usage; then Burstable, ranked by how far usage exceeds request; Guaranteed last. It is a ranking of which Pods the node can most afford to lose, not a finding about which Pod is responsible.

Every Pod evicted here was BestEffort, and none was a significant consumer. Two of them were the node monitoring agent, which had no requests set - so the node lost its own instrumentation every time it got into trouble, and the graph everyone was reading showed 70% because it is an average over the minutes the agent was alive to report.

Resolution

1. Stop the bleeding without changing capacity yet. Bound the log shipper with a memory limit on its systemd unit, so its growth restarts one unit on one host instead of evicting Pods across the node. This is reversible, it is node-local, and it buys time to do the rest properly.
2. Measure what the node actually needs outside Pods, at peak rather than at rest. Everything under the system slice, plus the kubelet and the runtime, at the busiest hour of the day. The defect was assuming this number was zero; guessing a replacement repeats it.
3. Set systemReserved and kubeReserved from that measurement plus headroom, on one drained node first. Restart the kubelet and confirm allocatable has dropped by roughly the amount you reserved.
4. Watch that node through a full peak before touching a second one. This fault only appears when the agent working set is at its largest, so a quiet afternoon proves nothing at all.
5. Give the monitoring agent requests equal to limits so it becomes Guaranteed and ranks last for eviction. A node that loses its instrumentation under pressure cannot be diagnosed under pressure.
6. Work out what the reservation costs the cluster before applying it fleet-wide. Reducing allocatable everywhere at once can put a large number of Pods into Pending simultaneously, and that recovery is a purchase order.
7. Roll out node by node, or node class by node class, and re-measure for classes with different agent sets. A control plane node and a log-heavy worker do not need the same reservation.
8. Clear the accumulated Evicted Pod objects and confirm the batch namespace quota is back under its limit. That ticket was always this incident wearing a different name.

Verification

1. The repaired node survives a full peak log period with no MemoryPressure transition and no eviction event. Peak is the only condition under which this fault exists, so it is the only condition in which the absence of the fault means anything.
2. Allocatable is now visibly below capacity by approximately the measured amount. If the two are still nearly equal, the configuration was not applied - check the kubelet actually restarted with it.
3. Pod requests plus the reservations no longer exceed what the host physically has. This is the arithmetic that was wrong; confirm it is now right rather than inferring it from the absence of alerts.
4. The bound on the agent works. Push it past its limit deliberately on a scratch node and confirm the unit is restarted rather than the node starting to evict Pods.
5. The monitoring agent survives node pressure. Induce pressure on a scratch node and confirm the agent is still running and still reporting - an eviction record with no metrics behind it is what made this take three weeks.
6. The Evicted Pod objects are cleared and the namespace that hit its quota is back under it, with the quota ticket closed against this incident rather than separately.
7. An alert on Evicted events grouped by node exists and has been tested by generating one. An alert that has never fired is a plan, not a control.

Prevention

• Treat rolling out a node agent as a capacity change. Anything installed on every host consumes memory the scheduler cannot see. Size it, bound it and reserve for it before it ships, rather than discovering it through evictions on the two busiest nodes three weeks later.

• Set the reservations from measurement, on every node class. The numbers differ between a log-heavy worker, a control plane node and a GPU node, because the agent sets differ:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
systemReserved:
  cpu: 500m
  memory: 8Gi
kubeReserved:
  cpu: 500m
  memory: 2Gi

The value above is this node class after measurement, not a default to copy. Re-measure when the agent set changes.

• Bound every node-level agent. An agent without a memory limit turns its own growth into the workload’s problem. With one, it becomes a restart on one host and an alert with the agent’s name on it.

• Put the agents you need during an incident into Guaranteed QoS. A monitoring agent with no requests is BestEffort, which means it is first out of the door at exactly the moment you need it most.

• Alert on Evicted events grouped by node. It costs nothing, it is a leading indicator, and grouping by node separates a node fault from a workload fault in one glance - the distinction this incident lacked for three weeks.

• Garbage collect Evicted Pod objects. Left alone they consume quota and reappear as an unrelated ticket in a namespace that has done nothing wrong, which splits one incident into two investigations that never meet.