OOMKilled container

Reported symptoms

Three tickets are open against the platform team and none of them mentions the other two.

The data team’s ingest pipeline is restarting. Throughput is about a third of normal and their dashboard shows the drops lining up with container restarts. They are certain they changed nothing: no resource values have been touched in this quarter’s manifests.

A different team reports that a report-runner Job and a metrics sidecar were killed overnight without warning. Both were on worker-4. Neither team deployed anything.

The platform on-call sees worker-4 flapping MemoryPressure=True and recovering, three times in twelve hours. When they look at the node the numbers look fine:

$ kubectl describe node worker-4 | grep -A 6 'Allocated resources'

Allocated resources:
Resource           Requests       Limits
--------           --------       ------
cpu                4100m (27%)    11500m (76%)
memory             5.9Gi (41%)    24Gi (167%)
ephemeral-storage  0 (0%)         0 (0%)

Forty-one percent. The node is, on paper, more than half empty.

Evidence provided

Start with the workload that is restarting, because it is the only one anybody can see failing in real time.

$ kubectl describe pod -n data-prod ingest-worker-5b7f9c4d6-jr2mn

    State:          Running
    Started:      Tue, 18 Aug 2026 08:14:52 +0000
  Last State:     Terminated
    Reason:       OOMKilled
    Exit Code:    137
    Started:      Tue, 18 Aug 2026 08:02:10 +0000
    Finished:     Tue, 18 Aug 2026 08:14:49 +0000
  Restart Count:  14
  Limits:
    memory:  2Gi
  Requests:
    memory:  256Mi
QoS Class:       Burstable

OOMKilled with exit code 137 is unambiguous: the kernel sent SIGKILL because the container’s cgroup hit its memory limit. Note what the container’s own logs do and do not show.

$ kubectl logs -n data-prod ingest-worker-5b7f9c4d6-jr2mn --previous --tail=4

08:14:48 INFO  batch 41219 accepted, 8192 records
08:14:49 INFO  decoding partition 3 of 8
08:14:49 INFO  buffer allocated for partit

Now the two Pods nobody deployed.

$ kubectl get pods -A --field-selector=status.reason=Evicted -o wide

NAMESPACE      NAME                      STATUS    NODE       REASON
report-prod    report-runner-28914-tqx7z Evicted   worker-4   Evicted
platform-obs   metrics-agent-9k4pd       Evicted   worker-4   Evicted

And the number that resolves the contradiction in the node description.

$ kubectl top node worker-4

NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
worker-4   5210m        34%    13.6Gi          95%

$ kubectl get pods -n data-prod -l app=ingest-worker -o wide --no-headers | awk '{print $8}' | sort | uniq -c

      6 worker-4
    2 worker-1
    2 worker-2
    1 worker-3
    1 worker-5

The only change anyone can find is a ConfigMap edit two days ago that raised a per-batch buffer size. No resource value was touched.

Work the evidence before reading on

Two Pods were Evicted and one is being OOMKilled. These are not two words for the same event.

1. What kills a container that is OOMKilled, and what scope does it act at? What kills a Pod that is Evicted, and what scope does that act at? Which of the two consults the QoS class?
2. The node reports 41 percent of memory requested and 95 percent used. Which of those two numbers did the scheduler use when it placed these Pods, and which one does the kubelet act on?
3. Six of twelve replicas are on one node. Given a request of 256Mi, how much memory did the scheduler believe it was committing on worker-4, and how much is actually there?
4. The evicted Pods belong to teams that changed nothing. Why did the kubelet choose them rather than the workload that was consuming the memory?

Before continuing: decide whether raising the memory limit from 2Gi to 4Gi makes the incident better or worse, and say which of the three symptoms each part of your answer applies to.

Root cause

1. Two killers, two scopes, one appearance

OOMKilled and Evicted look adjacent in a dashboard and are produced by different components.

Reading them as one problem is what sent the first hour of this incident into the wrong namespace. They are two symptoms of the same underlying number, but they are not the same failure and they do not have the same fix.

2. The request is the lie

The scheduler places Pods using requests and nothing else. It never looks at limits and it never looks at current usage.

ingest-worker declares requests.memory: 256Mi. Its real working set is around 1.6Gi. So when the scheduler considered worker-4, six replicas read as 1.5Gi of commitment against 14.3Gi of allocatable memory - trivially feasible - and it packed them there. The actual footprint of those six replicas is close to 10Gi.

That is the entire explanation for the contradiction in the node description. The 41 percent figure is a sum of promises. The 95 percent figure is a sum of reality. The gap between them is exactly the amount by which this workload’s request understates it, multiplied by the number of replicas that landed here.

Once real memory approaches the eviction threshold the kubelet begins reclaiming, and it reclaims in QoS order: BestEffort Pods first, then Burstable Pods that are over their requests. The report-runner Job declared no resources at all, so it was first in line. ingest-worker is Burstable and enormously over its request, so it is a candidate too - but the kubelet only needs to reclaim enough to get back under the threshold, and two smaller Pods were enough. The workload causing the pressure survived; its neighbours paid.

3. The ConfigMap change is the trigger, not the cause

The buffer-size increase two days ago raised peak usage from comfortably under 2Gi to occasionally over it, and raised the steady-state working set from about 900Mi to about 1.6Gi.

That single change crossed two thresholds at once. Crossing the container’s own limit produced the OOMKills. Crossing the node’s real memory ceiling - which was reachable only because six replicas were co-located on a request that never described them - produced the evictions.

Both thresholds had been sitting close for months. Nothing in the cluster said so, because the number that would have said so is the difference between requested and used memory, and nothing was watching it.

Resolution

1. Separate the two failures in writing before touching anything: which Pods were OOMKilled (container scope, kernel, cgroup limit) and which were Evicted (node scope, kubelet, QoS order). Every later decision depends on not confusing them.
2. Establish the real peak working set. Sample memory at a finer interval than the metrics pipeline collects, across a full batch cycle; the fifteen-second default will not show a spike that lasts two seconds, and that spike is what the limit has to accommodate.
3. Revert the batch-size key in the ConfigMap as the hold. It is consumed as an environment variable, so it takes effect on each replica at its next restart - which in a workload restarting every few minutes is immediate enough. Give this hold an owner and a date, because a reverted batch size is a capacity decision made by accident.
4. Raise requests.memory to the measured peak plus headroom. This is the change that fixes the evictions, because it is the only one the scheduler reads.
5. Set limits.memory above the new request with enough room for a genuine outlier batch. If the workload cannot tolerate eviction at all, set requests equal to limits and take the Guaranteed QoS class deliberately.
6. Roll the Deployment and let the scheduler redistribute. Expect the replicas to spread across more nodes and expect the cluster to look fuller afterwards - it was always this full; it now says so.
7. Confirm there is capacity for the honest requests before rolling, not after. Twelve replicas at a real request may not fit on five workers, and discovering that through a wave of Pending Pods during an incident is avoidable.
8. Tell the two teams whose Pods were evicted what happened and that the cause was not theirs. Their Pods will be recreated by their controllers, but an eviction with no explanation gets remembered as platform unreliability.
9. Record the requested-versus-used gap on worker-4 as it was, so the alert threshold you set afterwards is calibrated against a real incident rather than a guess.

Verification

1. Restart counts are frozen across a full batch cycle. A count that has slowed rather than stopped means the new limit is still inside the working set.
2. The kernel agrees. The oom_kill counter in the container cgroup memory.events stays flat over the same period; this is independent of the kubelet and of the API server.
3. The request-versus-usage gap has closed. Compare the allocated-memory figure from kubectl describe node against kubectl top node on every worker and require them to be within a defensible margin. That divergence was the defect; its absence is the fix.
4. The replicas have redistributed. No node holds six of twelve. If one still does, the request did not change enough to alter the scheduling decision.
5. No Evicted Pods on worker-4 across a full daily cycle, including the overnight batch window where the previous evictions occurred.
6. MemoryPressure has not transitioned. Check node conditions rather than trusting the absence of alerts, since the condition flapped three times without paging anyone.
7. The teams whose Pods were evicted confirm it has not recurred. Silence is not evidence; ask.
8. Prove the new alert fires. Set the requested-versus-used divergence rule, then reproduce the condition on a test node and confirm it pages. An alert that has never fired is an untested alert.

Prevention

• Size the memory request from measured usage and treat a large request-to-usage ratio as a defect in its own right. A request six times below reality is not conservative; it is a false statement to the scheduler that will be believed until a busy day.
• Give every container a memory limit. Without one there is no cgroup ceiling, and the workload’s first hard boundary is node exhaustion - which is everybody’s problem rather than its own.
• Alert on terminations with reason OOMKilled and on Pods with reason Evicted as two separate rules. They have different causes, different fixes, and a habit of appearing together while pointing at different teams.
• Monitor node memory requested against node memory used as a standing pair. The divergence between them is the leading indicator for this entire class of failure and it was visible for months before this incident.
• Read an eviction list as evidence about the node, not about the Pods on it. QoS ordering means the Pods that die are usually not the Pods responsible; expecting otherwise is what cost this incident its first hour.
• Review buffer sizes, batch sizes, concurrency settings and cache limits as memory changes. The manifest does not move when they change, so nothing in the deployment pipeline treats them as resource changes, and the working set moves anyway.