medium riskservice affecting~25 min

Runbook: Container OOM-killed — find what actually consumed the memory

1 · Prerequisites

Confirm every item is in place before any state change.

Shell access on the Docker host, with sudo for dmesg and the kernel log
The container name or ID is known and recorded before anything is restarted
The host runs cgroup v2 (confirm with the pre-checks below); paths differ on cgroup v1
jq is installed, or you are prepared to read raw JSON from docker inspect
You know whether a restart policy is in force, because a restart destroys the evidence in the old container

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· CONTAINER=web
· docker inspect -f '{{.State.OOMKilled}} {{.State.ExitCode}} {{.State.Status}} {{.State.FinishedAt}}' "$CONTAINER"
· docker inspect -f '{{.HostConfig.Memory}} {{.HostConfig.MemorySwap}} {{.HostConfig.MemoryReservation}}' "$CONTAINER"
· docker events --since 2h --until 0m --filter event=oom --filter event=die
· stat -fc %T /sys/fs/cgroup
· free -h
· docker logs --tail 200 --timestamps "$CONTAINER"

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Set CONTAINER to the affected container name and do not restart it yet; expect docker ps -a to still list it in an Exited state.
2Run docker inspect -f '{{.State.OOMKilled}} {{.State.ExitCode}}' "$CONTAINER"; expect true 137 for a kernel OOM kill. false with a non-137 exit code means this is not an OOM incident and you should stop here.
3Record the memory ceiling with docker inspect -f '{{.HostConfig.Memory}}' "$CONTAINER"; expect a byte count for a limited container and 0 for an unlimited one. 0 means only a host-wide OOM can explain the kill.
4Read the cgroup counters with CID=$(docker inspect -f '{{.Id}}' "$CONTAINER") then cat /sys/fs/cgroup/system.slice/docker-"$CID".scope/memory.events; expect a non-zero oom_kill for a cgroup limit kill.
5Run free -h and grep -c 'Out of memory' on the kernel log; expect MemAvailable to be healthy and no host-wide kill record if the cause was the container limit alone.
6Read the kernel record with sudo dmesg -T | grep -iE 'memory cgroup out of memory|out of memory|killed process'; expect a Memory cgroup out of memory line naming the container cgroup for a limit kill, and no cgroup line for a host OOM.
7Read the task table the kernel printed with sudo dmesg -T | grep -A 40 'invoked oom-killer' and sort the rss column; expect the victim to be the largest resident process, which is frequently not the one that grew.
8Correlate the FinishedAt timestamp with deploys, cron jobs, batch imports and traffic; expect one of them to line up within a couple of minutes of the kill.
9Apply the correct control - raise the container limit if the working set is legitimately larger, or fix the growth in the application if it is not - and record which of the two you chose and why.
10Start the container and watch memory.current and memory.events under representative load for at least one full traffic cycle.

4 · Verification

Confirm the procedure actually fixed the problem.

✓docker inspect -f '{{.State.OOMKilled}}' "$CONTAINER" prints false after the container has run for a full traffic cycle
✓docker inspect -f '{{.State.ExitCode}}' "$CONTAINER" is not 137 on any subsequent exit
✓The oom_kill counter in /sys/fs/cgroup/system.slice/docker-"$CID".scope/memory.events stops increasing over a sustained observation window
✓memory.current in that cgroup stays below 80 percent of memory.max at peak, not merely below it at idle
✓docker events --filter event=oom over the following hour returns no lines

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If a raised memory limit starved other containers on the host, lower it with docker update --memory and docker update --memory-swap on that container and re-check free -h
↶If the limit was raised in a Compose file, revert the mem_limit or deploy.resources.limits.memory value and run docker compose up -d for that service only
↶If a memory-related application setting was tuned - heap size, worker count, cache size - restore the previously recorded value and restart the container
↶The OOM kill itself cannot be rolled back. Whatever in-flight work the container held at SIGKILL is lost, because SIGKILL cannot be caught and no shutdown handler ran. Replay or reconcile that work explicitly rather than assuming it completed.

6 · Escalation

When the runbook isn't enough, contact:

· Memory grows steadily under constant load and returns to baseline only on restart: escalate to the application team as a suspected leak, attaching memory.current samples over time and the docker inspect state
· The host itself is OOM-killing processes with no container limit involved: escalate to the platform team as a host capacity incident, not a container incident
· The killed container is a database or any stateful service: escalate to the data team before changing any memory setting, because both the limit and the engine tuning have to move together
· The same container is OOM-killed within minutes of every restart and no configuration changed: escalate immediately rather than restarting again, because each restart destroys the evidence in the previous container

An OOM kill inside a container is the kernel enforcing a limit, not the application failing. The process that died is chosen by resident set size, so it is regularly a healthy process at its normal size rather than the one that grew. And because SIGKILL cannot be caught, the application log usually ends mid-sentence with no error at all. This runbook reads the evidence the kernel and the daemon left behind instead of guessing from the silence.

Symptoms

A container is gone, or has restarted, with nothing useful in its log — the last line is an ordinary one.
docker ps -a shows Exited (137).
Requests failed for the duration of the restart, then recovered.
Monitoring shows memory climbing to a flat ceiling and then a gap.

Step 1: Confirm it was an OOM kill

Read-only / Safecontainer state— Reads the daemon's record of the container. No side effects.

CONTAINER=web

docker inspect -f '{{.State.OOMKilled}} {{.State.ExitCode}} {{.State.Status}}' "$CONTAINER"
docker inspect -f 'finished {{.State.FinishedAt}} started {{.State.StartedAt}}' "$CONTAINER"

# The configured ceiling, in bytes. 0 means no limit was set.
docker inspect -f '{{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}' "$CONTAINER"

.State.OOMKilled is true and .State.ExitCode is 137 for a kernel OOM kill. 137 is 128 plus signal 9, the shell convention for “terminated by SIGKILL”.

Step 2: Cgroup limit kill, or host out of memory?

These look nearly identical from the container’s side and need completely different fixes. Read both sources.

Read-only / Safecgroup counters— Reads cgroup v2 accounting files and the kernel ring buffer.

# Confirm cgroup v2. Expect: cgroup2fs
stat -fc %T /sys/fs/cgroup

CID=$(docker inspect -f '{{.Id}}' "$CONTAINER")
CG=/sys/fs/cgroup/system.slice/docker-"$CID".scope

cat "$CG"/memory.max      # the ceiling, or the word max if unlimited
cat "$CG"/memory.current  # usage right now
cat "$CG"/memory.peak     # high-water mark since the cgroup was created
cat "$CG"/memory.events   # low high max oom oom_kill counters

# Host-wide view
free -h
sudo dmesg -T | grep -iE 'memory cgroup out of memory|out of memory|killed process'

Read the two counters in memory.events carefully, because they answer different questions:

Counter	Meaning	What it tells you
`max`	Times usage was about to exceed `memory.max`	The container is pressing on its ceiling, even if nothing died
`oom`	Times an allocation was about to fail at the limit	The reclaim path could not free enough
`oom_kill`	Processes killed by any OOM killer in this cgroup	Something in this container was actually killed

A rising max counter with oom_kill still at zero is the early warning. It means the container is being forced to reclaim constantly to stay under its limit — which shows up as latency, not as a crash, and is the state worth alerting on.

Step 3: Find the process that grew, not the one that died

The kernel picks its victim by resident set size. In a container running a supervisor, a web server and a worker, the worker leaks and the web server dies, because the web server was bigger. Read the task table the kernel printed at the moment of the kill.

Read-only / Safekill record— Reads the recorded OOM report from the kernel ring buffer.

# The full report, including the per-task table
sudo dmesg -T | grep -B 5 -A 40 'invoked oom-killer'

# The victim line only
sudo dmesg -T | grep -i 'Killed process'

Four things in that output are the diagnosis:

invoked oom-killer — the process whose allocation failed. It requested memory that was not available. Often not the victim.
The task table — pid, rss, name for every task in the cgroup. Sort by rss. The largest is the victim; the one that is abnormal relative to its own usual size is the culprit.
Killed process ... anon-rss:... — the victim and how much it actually held.
Memory cgroup out of memory — present only for a limit kill.

Step 4: Look inside the namespace

With a private cgroup namespace — the default for a modern daemon — the container sees its own limits at the root of /sys/fs/cgroup, so you can read them without knowing the container ID.

Read-only / Safein-container view— Reads cgroup files and process sizes from inside the container. No state change.

docker exec "$CONTAINER" sh -c 'cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current'
docker exec "$CONTAINER" sh -c 'cat /sys/fs/cgroup/memory.events'

# Per-process resident size inside the container, largest first
docker exec "$CONTAINER" ps -eo pid,rss,comm --sort=-rss 2>/dev/null | head

# Or from the host, if the image has no ps
docker top "$CONTAINER" -eo pid,rss,comm --sort=-rss

docker top runs ps on the host against the container’s processes, so it works even for a distroless image that contains no shell and no ps at all. That is the reason to prefer it.

Step 5: Apply the right control

Configuration changeadjust the limit— Changes the running container's memory ceiling. Does not restart it.

# Raise the ceiling on a running container
docker update --memory 1g --memory-swap 1g "$CONTAINER"

# Confirm the daemon and the cgroup agree
docker inspect -f '{{.HostConfig.Memory}}' "$CONTAINER"
cat /sys/fs/cgroup/system.slice/docker-"$CID".scope/memory.max

Set --memory-swap to the same value as --memory to forbid the container swapping. A container that is allowed to swap does not get OOM-killed; it gets slow, and stays slow, which is harder to diagnose and usually worse for a request-serving workload than a fast, obvious kill.

For a Compose-managed service, change the Compose file rather than using docker update, or the next docker compose up silently reverts your change.

Common patterns

Evidence	Diagnosis	Action
`OOMKilled=true`, `dmesg` shows `Memory cgroup out of memory`, host has free RAM	Container limit reached	Raise the limit or reduce the working set. Host RAM is irrelevant
`OOMKilled=true`, no cgroup line, host exhausted	Host-wide OOM	Bound the oversubscribing containers, or add host capacity
`OOMKilled=false`, exit 137	Something sent `SIGKILL`	Check `docker events --filter event=kill` and stop-grace timeouts
`memory.events` `max` climbing, `oom_kill` at 0	Constant reclaim under the ceiling	Latency problem now, kill later. Raise the limit before it becomes an outage
Killed process is not the one that grew	Kernel chose by RSS	Read the task table; size each process against its own baseline
Memory flat for days, then a cliff	Batch job or a large request	Bound the job, or stream instead of buffering
Steady climb over days, reset only by restart	Application leak	Escalate with samples over time; restarting is a workaround
`memory.max` generous, runtime still overshot	Runtime sized its heap from host memory	Configure the runtime’s own limit explicitly