OOM killer decisions - who gets killed and why

When memory runs out and reclaim fails, the OOM killer runs. The decision of who dies is heuristic but tuneable. This lesson covers how to influence it.

OOM selection

The kernel scores every eligible task with oom_badness() and kills the one with the highest score. The score is almost entirely about memory footprint:

score ≈ RSS + swap entries + page-table pages

expressed as a fraction of the memory available to the task, then offset by oom_score_adj. The adjustment is scaled so that ±1000 shifts the score by ±100% of total memory — which is why -1000 makes a task effectively unkillable and +1000 makes it the first choice regardless of size.

Read the live values straight from procfs:

# the current score the kernel would use
cat /proc/<pid>/oom_score

# the operator-set adjustment feeding into it
cat /proc/<pid>/oom_score_adj

# rank every process by score
for p in /proc/[0-9]*; do
  printf '%s %s %s\n' "$(cat $p/oom_score 2>/dev/null)" "${p##*/}" "$(cat $p/comm 2>/dev/null)"
done | sort -rn | head

Selection is still not the same as fairness. The kernel aims to free the most memory by killing the fewest tasks, so the largest consumer usually dies — and with cgroup v2, selection happens within the cgroup that hit its limit rather than across the whole host.

oom_score_adj

/proc/<pid>/oom_score_adj is a per-process adjustment from -1000 to 1000:

• -1000: never kill.
• -500: kill last.
• 0: default.
• +500: kill first.
• +1000: kill always.

# View
cat /proc/<pid>/oom_score_adj

# Set
echo -500 > /proc/<pid>/oom_score_adj

For systemd services:

[Service]
OOMScoreAdjust=-500    # kill last

For critical services (databases, monitoring agents): set to a low value to avoid being killed. For cache / batch processes: set to a high value.

Read OOM kill records

sudo dmesg | grep -i oom
sudo journalctl -k | grep -i oom

Each kill emits two lines. The oom-kill: line says why the allocation failed; the human-readable line says who died.

# cgroup OOM - one unit went over its own memory.max
oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=myapp.service,
  mems_allowed=0,oom_memcg=/system.slice/myapp.service,
  task_memcg=/system.slice/myapp.service,task=java,pid=12345,uid=0
Memory cgroup out of memory: Killed process 12345 (java) total-vm:8234560kB anon-rss:4123456kB

# global OOM - the host itself ran out
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,
  oom_memcg=(null),task_memcg=/system.slice/myapp.service,task=java,pid=12345,uid=0
Out of memory: Killed process 12345 (java) total-vm:8234560kB anon-rss:4123456kB file-rss:0kB shmem-rss:0kB

Read constraint= before you change anything. CONSTRAINT_MEMCG with a named oom_memcg= means raise that unit’s MemoryMax or make the service use less. CONSTRAINT_NONE with oom_memcg=(null) means the host is over-committed — raising one unit’s limit there makes it worse. cat /sys/fs/cgroup/<unit-path>/memory.events attributes the same kill without the log, and survives rotation.

The record shows the killed process’s PID, name, and memory usage. To find which service:

# Substitute your own values before running:
PID=1234

# Find the systemd unit
ps -o pid,unit | grep 12345
# or
systemctl status "$PID"

OOM in cgroups

With cgroups v2, OOM is contained per cgroup. If a cgroup exceeds its memory.max, the OOM killer runs within that cgroup.

[Service]
MemoryMax=4G
OOMPolicy=continue    # OOMs but service stays running
OOMScoreAdjust=-900   # strongly avoid being killed

OOMPolicy=continue means the kernel attempts to reclaim memory but does not necessarily kill. Useful for services that should survive memory pressure at the cost of slowness.

OOMPolicy=kill (default) means OOM kills a process in the cgroup.

cgroup-aware OOM

For critical services:

[Service]
MemoryMax=4G
OOMScoreAdjust=-900

For disposable services (caches, batch jobs):

[Service]
MemoryMax=8G
OOMScoreAdjust=+500

The cgroup limit + score adjustment together give fine control.

Common patterns

Knowledge check