OOM killer behaviour - what happens when memory runs out

When memory runs out, the kernel kills a process. This is the OOM killer. This lesson covers how it works and how to manage it.

What the OOM killer does

When the kernel cannot reclaim memory:

1. It picks a process to kill.
2. It sends SIGKILL.
3. The process dies.
4. Memory is freed.

The picker scores every eligible task with oom_badness() and kills the highest scorer. The score is almost entirely about memory footprint:

score ≈ RSS + swap entries + page-table pages

expressed as a fraction of the memory available to the task, then offset by oom_score_adj.

OOM in cgroups

With cgroups v2, the OOM killer operates per cgroup. If a cgroup exceeds its memory.max, the OOM killer runs within that cgroup, killing the offending process.

The whole-host OOM only triggers when the system has no memory at all (after all cgroup limits are exceeded).

Configure OOM behaviour

OOMPolicy= does not stop the kernel from killing anything. It only decides what the service manager does after a process in the unit has already been OOM-killed.

[Service]
# What systemd does AFTER a process in this unit was OOM-killed:
#   continue - log it; leave the (now degraded) unit running
#   stop     - terminate the unit cleanly; it enters a failed state
#   kill     - kill every remaining process in the unit
#              (sets memory.oom.group=1 on the cgroup)
OOMPolicy=stop
Restart=on-failure
OOMScoreAdjust=-500      # lower oom_score (less likely to be chosen)

The default is whatever DefaultOOMPolicy= is set to in systemd-system.conf, which ships as stop on current distributions — not kill. The one exception is a unit with Delegate= turned on (container and slice managers), where the default is continue. Confirm on the host rather than assuming:

systemctl show --property=DefaultOOMPolicy
systemctl show myapp.service --property=OOMPolicy

OOMScoreAdjust shifts the oom_score by the given amount. Negative values make the process less likely to be killed. Note the limit: when a cgroup OOM fires and every task in that cgroup carries the same adjustment, the adjustment does not influence which of them is chosen. It biases selection between processes with different scores, so it is a host-level tool, not a within-service one.

Per cgroup (in systemd):

[Slice]
MemoryMax=4G
MemoryHigh=3G

MemoryHigh is the directive that actually reduces the chance of an OOM kill. Above it the kernel throttles the cgroup and reclaims aggressively; the service slows down instead of dying. MemoryMax is the hard wall where the cgroup OOM killer runs. Setting MemoryHigh below MemoryMax gives the service a pressure zone to recover in.

The second OOM killer: systemd-oomd

On Ubuntu 22.04 and later, and on Fedora, systemd-oomd is enabled by default. It is a userspace OOM killer that watches PSI pressure and terminates services before the kernel has to act. Its kills leave nothing in the kernel log.

# Is the userspace OOM killer active?
systemctl is-active systemd-oomd
journalctl -u systemd-oomd --since '1 hour ago'
oomctl                      # what oomd monitors, and current pressure

# Opt a unit in or out of systemd-oomd management
[Service]
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit=50%
ManagedOOMSwap=auto

Find OOM kills

# Kernel logs
sudo dmesg | grep -i 'out of memory\|oom'

# systemd journal
sudo journalctl -k | grep -i 'out of memory\|oom'

# Per-service
sudo journalctl -u my-service | grep -i 'killed\|oom'

# The userspace killer - checked SECOND, and it leaves no kernel log
sudo journalctl -u systemd-oomd --since '1 hour ago'

# Audit
sudo ausearch -m SERVICE_STOP

Every kill emits two lines. The oom-kill: line carries the evidence; the human-readable line carries the victim.

# a cgroup hit its own memory.max - the host had memory to spare
oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=myapp.service,
  mems_allowed=0,oom_memcg=/system.slice/myapp.service,
  task_memcg=/system.slice/myapp.service,task=java,pid=12345,uid=0
Memory cgroup out of memory: Killed process 12345 (java) total-vm:8234560kB anon-rss:4123456kB

# the host itself ran out
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,
  oom_memcg=(null),task_memcg=/system.slice/myapp.service,task=java,pid=12345,uid=0
Out of memory: Killed process 12345 (java) total-vm:8234560kB anon-rss:4123456kB

Prevent OOM cascades

A service that grows until it OOMs can affect other services. With cgroups, the OOM is contained.

Without cgroups:

• Service X grows.
• Host memory runs out.
• Host OOM killer runs.
• Service Y (innocent) gets killed.

With cgroups:

• Service X grows.
• X’s cgroup hits memory.max.
• X’s cgroup OOM killer runs.
• X is killed.
• Service Y survives.

The discipline: every service has a cgroup with a memory limit.

Knowledge check