Memory limits and OOM behaviour · Docker & Containers

--memory 512m writes 536870912 to the container’s cgroup file memory.max. Everything else in this lesson follows from that one sentence: the limit is enforced by the kernel, not by Docker, and when it is exceeded the kernel takes an action that Docker only learns about afterwards.

The reason this deserves thirty minutes rather than three is that the evidence of an OOM kill is almost never where people look for it. The application log is empty. docker logs shows a clean shutdown or nothing at all. The exit code is a number nobody recognises. And the process the kernel killed is frequently not the process that caused the problem.

Setting the limits

Configuration changehard limit— The minimum accepted value is 6m. Below that Docker rejects the flag.

docker run -d --name api --memory 512m myorg/api:1.0.0

Configuration changesoft target— A soft limit the kernel prefers to reclaim below when the host is under pressure. It does not cap anything.

docker run -d --name api --memory 512m --memory-reservation 256m myorg/api:1.0.0

Configuration changememory plus swap— --memory-swap is the COMBINED total, not the swap allowance. 1g here means 512m RAM plus 512m swap.

docker run -d --name api --memory 512m --memory-swap 1g myorg/api:1.0.0

The --memory-swap semantics catch people out often enough to be worth a table:

`--memory`	`--memory-swap`	Effective swap allowance
`512m`	unset	`512m` — the container may swap as much as its memory limit, giving 1 GB total
`512m`	`1g`	`512m`
`512m`	`512m`	none — swap is disabled for this container
`512m`	`-1`	unlimited, up to what the host has

Setting --memory-swap equal to --memory is the useful trick: it turns swap off for that container, so a memory leak fails fast and loudly instead of degrading into a host that is technically alive and answering nothing.

The exit that explains nothing

Here is the shape of the 03:00 page.

Read-only / Safethe symptom— A container that was fine for six hours is gone. The exit code is the only clue on this screen.

$ docker ps -a --filter name=api --format 'table {{.Names}}\t{{.Status}}'

NAMES     STATUS
api       Exited (137) 4 minutes ago

Illustrative output

137 is 128 + 9. The process was killed by signal 9, SIGKILL. SIGKILL cannot be caught, blocked, or handled — so the application had no opportunity to write a shutdown message, flush a log buffer, or run an exit hook. That is why docker logs api ends mid-sentence on an ordinary request. The absence of a log line is the evidence, not the absence of evidence.

137 does not by itself mean OOM. Anything that sends SIGKILL produces it: docker kill, a docker stop whose grace period expired, a health-check-driven orchestrator, an operator. So the next command is not optional.

Read-only / Safeask the daemon— The authoritative first check. OOMKilled is set from a containerd cgroup-OOM event.

$ docker inspect api --format 'oom={{.State.OOMKilled}} exit={{.State.ExitCode}} err={{printf "%q" .State.Error}}'

oom=true exit=137 err=""

Illustrative output

Note err="". .State.Error is empty on a normal OOM kill — there is no message, because nothing generated one. Any runbook that tells you to look for a string in .State.Error is wrong, and following it wastes the first ten minutes of the incident.

`OOMKilled: true` on a container that is still running

The daemon sets .State.OOMKilled when containerd reports a cgroup OOM event, which is a different thing from the container exiting. So both of these are real states you will meet:

Exited, OOMKilled: true — the OOM killer picked PID 1, or picked a child whose death took PID 1 with it. The obvious case.
Running, OOMKilled: true — the OOM killer picked a child process and PID 1 survived. The container is up, its health check may still pass, and one worker, one thread pool, or one background job is silently gone.

The second is much worse to diagnose, because nothing restarted and nothing alerted. The queue stops draining, or one endpoint starts timing out, and the container has been “healthy” for hours. Alert on the oom Docker event, not on container restarts:

docker events --filter event=oom --format '{{.Time}} {{.Actor.Attributes.name}}'

The flag also persists in the container’s state until the next start, so a true you find during an investigation may be from an OOM that happened yesterday. Correlate it against dmesg timestamps before concluding it is the one you are chasing.

Where the evidence actually is

Three sources, in the order you should read them.

Read-only / Safe1. the cgroup's own counters— memory.events is a running tally per cgroup. oom_kill counts processes actually killed.

CID=api
PID=$(docker inspect --format '{{.State.Pid}}' "$CID")
CG=$(awk -F: '/^0::/{print $3}' "/proc/$PID/cgroup")
cat /sys/fs/cgroup"$CG"/memory.events

low 0
high 0
max 4128
oom 3
oom_kill 3
oom_group_kill 0

Read it precisely — each field means something different:

Field	Meaning
`low`	times the cgroup was reclaimed despite being under `memory.low`
`high`	times processes were throttled into direct reclaim by `memory.high`
`max`	times usage was about to exceed `memory.max`, so the kernel reclaimed instead
`oom`	times reclaim failed and an allocation was about to fail
`oom_kill`	processes killed by any OOM killer in this cgroup
`oom_group_kill`	times the whole cgroup was killed together

max 4128 with oom_kill 0 is the healthy-but-tight case: the container repeatedly bumped its ceiling and the kernel reclaimed page cache each time. Nothing died, but it is doing reclaim work on every request. oom_kill 3 is three dead processes.

This cgroup path disappears when the container is removed. On a container that has already exited and been cleaned up, skip to the kernel log.

Read-only / Safe2. the kernel log— The kernel prints a full report for every OOM kill, including a table of every task in the cgroup and its RSS.

sudo dmesg -T | grep -iE 'oom-kill|Killed process|Memory cgroup out of memory' | tail -20
sudo journalctl -k --since '1 hour ago' | grep -i 'out of memory'

[Sun Aug  9 03:14:02 2026] python invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[Sun Aug  9 03:14:02 2026] memory: usage 524288kB, limit 524288kB, failcnt 0
[Sun Aug  9 03:14:02 2026] Memory cgroup stats for /system.slice/docker-9f2c1a....scope:
[Sun Aug  9 03:14:02 2026] Tasks state (memory values in pages):
[Sun Aug  9 03:14:02 2026] [  pid  ]   uid  tgid total_vm      rss ... name
[Sun Aug  9 03:14:02 2026] [  31840]     0 31840    12043      812 ... tini
[Sun Aug  9 03:14:02 2026] [  31871]  1001 31871   198442    94210 ... gunicorn
[Sun Aug  9 03:14:02 2026] [  31904]  1001 31904    41220     8102 ... celery-beat
[Sun Aug  9 03:14:02 2026] Memory cgroup out of memory: Killed process 31871 (gunicorn) total-vm:793768kB, anon-rss:376840kB

That block is the single most useful artefact in a memory incident, and almost nobody reads it. It names the cgroup, the limit, every task with its RSS, and which one was chosen. Paste it into the ticket verbatim.

Read-only / Safe3. the breakdown of what was using the memory— memory.stat splits usage into anonymous, file-backed and kernel. This is what tells a leak from a cache.

CID=api
PID=$(docker inspect --format '{{.State.Pid}}' "$CID")
CG=$(awk -F: '/^0::/{print $3}' "/proc/$PID/cgroup")
cat /sys/fs/cgroup"$CG"/memory.current
grep -E '^(anon|file|slab|sock|shmem) ' /sys/fs/cgroup"$CG"/memory.stat

The two wrong diagnoses

Wrong diagnosis 1: the process that died is the process that leaked

It usually is not, and understanding why saves hours.

When a cgroup hits memory.max and reclaim fails, the kernel scans the tasks in that cgroup and scores each one, essentially by resident set size adjusted by oom_score_adj. It kills the highest scorer. That is the largest process, which in a typical container is the application server — regardless of which process was allocating.

So a small sidecar with a runaway buffer pushes the cgroup to its limit, and the kernel kills the 400 MB Gunicorn master, because killing it frees the most memory fastest. The kernel is optimising for “recover the machine”, not for “identify the culprit”.

The tell is in the dmesg task table above: read the RSS column for every task, not just the killed one. If the killed process’s RSS is roughly what it always is, and the total is at the limit, the growth is somewhere else in the list. Two containers’ worth of processes in one cgroup — a sidecar pattern, or a --pid=container: share — makes this the default outcome rather than the exception.

Wrong diagnosis 2: `memory.current` at 99% means a leak

memory.current includes page cache. A container that reads or writes files accumulates file-backed pages charged to its cgroup, and the kernel is perfectly happy to let that grow until it touches the limit, because those pages are reclaimable — it will drop them the instant memory is needed.

So a log-heavy or file-serving container sitting at 511 MB of a 512 MB limit is often completely healthy. docker stats shows 99.8% and looks alarming. It is not a leak.

The discriminator is memory.stat:

anon climbing steadily and never falling — a real leak. Anonymous memory is heap; it is not reclaimable and it is what causes OOM kills.
file large, anon flat — page cache. Harmless. If it genuinely bothers you, memory.high or a smaller limit makes the kernel reclaim sooner, at a cost in I/O.
slab climbing — kernel objects, often dentries from a workload that creates and deletes many files. Real, and not visible in any application profiler.

Alerting on memory.current / memory.max produces false pages. Alerting on anon growth, or on memory.events oom_kill increasing, does not.

`memory.high`: throttle instead of kill

memory.max is a wall. memory.high is a brake: when usage crosses it, the kernel puts allocating processes into direct reclaim and throttles them, but never kills. A container that overshoots memory.high gets slow; a container that overshoots memory.max gets SIGKILL.

Docker has no flag for memory.high. runc writes memory.max from --memory, memory.low from --memory-reservation, and memory.swap.max from --memory-swap, and nothing writes memory.high at all.

You can set it by hand for a diagnosis:

Configuration changeset memory.high for an investigation— Writes directly to the cgroup. This does NOT survive a container restart — the cgroup is recreated.

CID=api
PID=$(docker inspect --format '{{.State.Pid}}' "$CID")
CG=$(awk -F: '/^0::/{print $3}' "/proc/$PID/cgroup")
echo 400M | sudo tee /sys/fs/cgroup"$CG"/memory.high
# watch the throttle counter climb instead of an oom_kill
watch -n2 grep high /sys/fs/cgroup"$CG"/memory.events

The restart caveat is the whole reason this is a diagnostic tool and not a production configuration. If you need it permanently, the supported route is a systemd drop-in on the slice, or lowering --memory and accepting the kills.

From flag to cgroup file, exactly

runc writes four files in the container’s cgroup directory and no others:

Docker flag	cgroup v2 file	Conversion
`--memory 512m`	`memory.max`	bytes, `536870912`
`--memory-reservation 256m`	`memory.low`	bytes
`--memory-swap 1g`	`memory.swap.max`	`memory-swap` minus `memory` — cgroup v2 counts swap separately, so `1g` with `--memory 512m` writes `536870912`

Three things are notable by their absence.

There is no memory.high, as discussed.

runc does not set memory.oom.group, which therefore defaults to 0 — meaning the kernel kills one process at a time rather than the whole cgroup. That default is precisely what produces the “running, but OOMKilled: true, with a worker missing” state from the callout above.

And --memory-swappiness is not implemented on cgroup v2. The flag is accepted by the CLI and there is no corresponding write in runc’s cgroup v2 memory code, so on any current host it does nothing. Do not use it as a tuning knob and do not put it in a runbook; if you need to change swap behaviour for a container, use --memory-swap, which is enforced.

--kernel-memory is likewise ignored, and has been deprecated for several releases.

Find the cgroup for any container with:

CID=api
PID=$(docker inspect --format '{{.State.Pid}}' "$CID")
awk -F: '/^0::/{print "/sys/fs/cgroup" $3}' "/proc/$PID/cgroup"

The 0:: prefix is the cgroup v2 unified hierarchy line. On a systemd-driver host the path looks like /sys/fs/cgroup/system.slice/docker-<id>.scope; with the cgroupfs driver it is /sys/fs/cgroup/docker/<id>. Confirm which you have with docker info --format '{{.CgroupDriver}} v{{.CgroupVersion}}'.

What happens when the host, rather than a container, runs out of memory?

Different killer, different rules, worse outcome.

The global OOM killer scans every process on the machine and picks the highest badness score, which is roughly RSS adjusted by oom_score_adj. Your largest container is the obvious candidate, but so are dockerd and the containerd shims — and killing a shim takes its container with it while leaving the daemon confused about the state.

This is the failure mode of running containers without memory limits: an unlimited container has no memory.max to hit, so it grows until the host has nothing left, and then the kernel starts choosing victims across the whole machine.

Prevention, in order of effectiveness:

Every container has a limit. Gate it in CI. An unlimited container is the bug.
The sum of limits leaves the host real headroom. Compare against MemTotal minus what the OS, the daemon, and the page cache actually need — 15–20% is a reasonable starting reservation.
Protect the daemon. --oom-score-adj -500 on the container makes the kernel prefer other victims; the systemd units for docker.service and containerd.service should carry OOMScoreAdjust=-500 so the control plane survives to restart things.
Alert on pressure, not on kills. /proc/pressure/memory shows stall time before anything dies. By the time oom_kill increments, the outage has already happened.

Setting the limits

The exit that explains nothing

Where the evidence actually is

The two wrong diagnoses

memory.high: throttle instead of kill

Knowledge check

`memory.high`: throttle instead of kill