Lab 11: Observing an OOM kill with cgroup files

Objective

Reproduce a cgroup OOM, then prove it from three independent sources: docker inspect, the cgroup memory.events file, and the kernel log.

Tasks

Task 1: Start a memory-limited container

docker run -d --name oom-lab --memory 128m --memory-swap 128m \
  alpine sleep 3600

Task 2: Find the cgroup

PID=$(docker inspect oom-lab --format '{{.State.Pid}}')
CGROUP=$(cat /proc/$PID/cgroup | awk -F: '{print $3}' | head -1)
echo "PID=$PID"
echo "CGROUP=$CGROUP"
echo "---"
cat /sys/fs/cgroup$CGROUP/memory.max
cat /sys/fs/cgroup$CGROUP/memory.current

Task 3: Try to OOM it

docker exec oom-lab sh -c 'apk add --no-cache stress-ng && \
  stress-ng --vm 1 --vm-bytes 200M --vm-hang 60 --timeout 30s'

(--vm-bytes 200M exceeds the 128 MB cap; stress-ng should be killed.)

Task 4: Confirm via docker inspect

docker inspect oom-lab --format '{{.State.Status}} {{.State.OOMKilled}} {{.State.ExitCode}}'
# exited true 137

Exit code 137 = 128 + 9 (SIGKILL). OOMKilled: true.

Task 5: Confirm via cgroup memory.events

cat /sys/fs/cgroup$CGROUP/memory.events
# oom 1
# oom_kill 1
# oom_group_kill 0

The oom_kill counter is incremented by the kernel on every cgroup OOM kill.

Task 6: Confirm via dmesg / journalctl

sudo dmesg | grep -i 'oom' | tail -3
sudo journalctl -k --since "5 minutes ago" | grep -i oom | tail -3

You should see memory cgroup out of memory: Killed process ... with the PID matching the container’s init.

Task 7: Cleanup

docker rm -f oom-lab