Lab: OOM investigation and prevention

This lab triggers an OOM, investigates it, and applies prevention. By the end you will have the discipline for OOM incidents.

Tasks

Task 0: Install the workload generator

sudo apt install -y stress-ng     # Debian / Ubuntu
sudo dnf install -y stress-ng     # RHEL / Rocky / Alma (EPEL)

Confirm the host runs the cgroup v2 unified hierarchy. Every command in this lab assumes it:

$ stat -fc %T /sys/fs/cgroup

cgroup2fs

Task 1: Bound the memory, not the host

The memory you need to constrain is the test workload’s, not the machine’s. A cgroup limit does that precisely.

# Confirm swap is present and note how much is in use.
# You are NOT going to turn it off - see the warning below.
swapon --show
free -h

Task 2: Trigger an OOM inside a scope

Let systemd own the cgroup hierarchy and place the workload in a transient scope with a real v2 memory limit:

$ sudo systemd-run --unit=oom-test --scope \
-p MemoryMax=256M -p MemorySwapMax=0 \
stress-ng --vm 2 --vm-bytes 200M --timeout 30s

Running scope as unit: oom-test.scope
stress-ng: info:  [4711] setting to a 30 second run per stressor
stress-ng: info:  [4711] dispatching hogs: 2 vm
Killed

The process tries to allocate 400 MB total but the cgroup limit is 256 MB. The OOM killer will run.

Task 3: Find the OOM records

sudo dmesg -T | grep -iE 'oom-kill|out of memory|memory cgroup'
sudo journalctl -k --since '1 hour ago' | grep -iE 'oom-kill|out of memory|memory cgroup'

Output for the cgroup OOM you just triggered:

oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=oom-test.scope,
  mems_allowed=0,oom_memcg=/system.slice/oom-test.scope,
  task_memcg=/system.slice/oom-test.scope,task=stress-ng,pid=12345,uid=0
Memory cgroup out of memory: Killed process 12345 (stress-ng) total-vm:524288kB anon-rss:262144kB

That is not what a host-wide OOM looks like. Learn the difference now, because it is the first question of every OOM incident.

There is a non-log way to attribute the same kill, and it survives log rotation:

$ cat /sys/fs/cgroup/system.slice/myapp.service/memory.events

low 0
high 1043
max 219
oom 3
oom_kill 3

Task 4: Identify the process

# Substitute your own values before running:
PID=1234

# Find the systemd unit (if any)
ps -fp "$PID"
systemctl status "$PID"

# Find what spawned it
ps -o pid,ppid,comm

Task 5: Apply prevention

For a real service that should not be killed:

[Service]
ExecStart=/usr/local/bin/myapp
MemoryMax=2G
OOMScoreAdjust=-900

For a cache or batch job that should be killed first:

[Service]
ExecStart=/usr/local/bin/cache
MemoryMax=512M
OOMScoreAdjust=+500

Task 6: Test the protection

Restart the service with the new settings. Try to OOM again - with the pressure source held inside its own scope so the host survives the test:

$ sudo systemd-run --unit=oom-pressure --scope \
-p MemoryMax=1G -p MemorySwapMax=0 \
stress-ng --vm 4 --vm-bytes 512M --timeout 30s

# Check that the protected service is still running
systemctl status myapp

# Confirm where the kill landed
sudo journalctl -k --since '5 min ago' | grep 'oom-kill'

The protected service should survive, and oom_memcg= in the kill record should name oom-pressure.scope.

Task 7: Clean up

Leave the host exactly as you found it:

# Stop the transient scopes if they are still running
sudo systemctl stop oom-test.scope oom-pressure.scope 2>/dev/null

# Remove the hand-made cgroup, if you built one in Task 2
sudo rmdir /sys/fs/cgroup/oom-test 2>/dev/null

# Nothing to restore for swap - you never turned it off
swapon --show

Confirm no test unit survives:

$ systemctl list-units --all 'oom-*'

0 loaded units listed.

Task 8: Document

OOM LAB
=======
Test environment:
- transient scope: oom-test.scope
- MemoryMax: 256M, MemorySwapMax: 0
- host swap: left enabled, untouched

OOM triggered by:
- stress-ng --vm 2 --vm-bytes 200M
- Tried to allocate 400 MB
- Hit 256 MB limit
- OOM killed stress-ng

Evidence classified:
- constraint=CONSTRAINT_MEMCG  -> cgroup OOM, not host OOM
- oom_memcg=/system.slice/oom-test.scope
- memory.events oom_kill incremented for that cgroup

Prevention applied:
- Protected service: MemoryMax=2G, OOMScoreAdjust=-900
- Disposable service: MemoryMax=512M, OOMScoreAdjust=+500

Verification:
- OOM still occurs under stress
- Protected service survives
- Disposable service is killed first