cgroups v2 — the resource side of container isolation · Docker & Containers

Namespaces decide what a container can see. cgroups decide what a container can use. Together they are the two halves of container isolation.

This lesson teaches the cgroups v2 hierarchy that every modern Linux distribution ships by default, and shows exactly how Docker translates --memory, --cpus, and other flags into cgroup constraints.

What cgroups v2 actually is

flowchart TB
  Root["/sys/fs/cgroup (root)"] --> Slice["system.slice / user.slice / docker.slice"]
  Slice --> Service["system-docker.service"]
  Slice --> Container["system-docker-<id>.scope"]
  Container --> Cpu["cpu.max"]
  Container --> Mem["memory.max"]
  Container --> Pids["pids.max"]
  Container --> Io["io.max"]

The cgroup v2 hierarchy is a single tree, mounted at /sys/fs/cgroup by default. Every process is in exactly one leaf cgroup. Controllers (cpu, memory, pids, io, etc.) enforce the limits set on the cgroup tree at each node.

A few things that catch operators out:

The tree is one tree. There is no cpu,cpuacct or cpu,cpuacct,cpuset split; the cgroup v1 mess is gone.
The cgroup that owns a process owns all its children by default. Move a process to a child cgroup and its descendants follow.
The container’s cgroup is created by containerd (or dockerd via containerd) at startup. It is removed when the container is removed (assuming no orphaned cgroups — more on that later).

Inspecting cgroups from the host

Read-only / Safefind a container's cgroup— Every container's PID is in a cgroup under the container runtime's slice. Find it.

PID=$(docker inspect --format '{{.State.Pid}}' CONTAINER)
cat /proc/$PID/cgroup

Read-only / Safecgroup contents— The container's cgroup directory contains all its resource controls.

CGROUP=$(awk -F: '!/^#/{print $3}' /proc/self/cgroup | head -1)
ls /sys/fs/cgroup/$CGROUP/ 2>/dev/null | head -20

Read-only / Safememory limits— memory.max and memory.current — what the container is limited to and what it is using right now.

PID=$(docker inspect --format '{{.State.Pid}}' CONTAINER)
CGROUP=$(cat /proc/$PID/cgroup | awk -F: '!/^0:/{print $3}' | head -1)
cat /sys/fs/cgroup/$CGROUP/memory.max
echo 'current:'
cat /sys/fs/cgroup/$CGROUP/memory.current

Read-only / Safecpu limits— cpu.max is a quota/period pair. A value of 150000 100000 means the container can use 1.5 CPU cores worth of time per 100 ms.

PID=$(docker inspect --format '{{.State.Pid}}' CONTAINER)
CGROUP=$(cat /proc/$PID/cgroup | awk -F: '!/^0:/{print $3}' | head -1)
cat /sys/fs/cgroup/$CGROUP/cpu.max

Read-only / SafePID limits— pids.max caps the number of processes (and threads) the container can create.

PID=$(docker inspect --format '{{.State.Pid}}' CONTAINER)
CGROUP=$(cat /proc/$PID/cgroup | awk -F: '!/^0:/{print $3}' | head -1)
cat /sys/fs/cgroup/$CGROUP/pids.max
echo 'current:'
cat /sys/fs/cgroup/$CGROUP/pids.current

How Docker flags translate into cgroup limits

From `docker run` flags to cgroup v2 files

$ --memory 512m--memory-reservation 256m--cpus 1.5--cpu-shares 512--pids-limit 200--blkio-weight 500--device-read-bps /dev/sda:1mb--device-read-iops /dev/sda:1000--restart unless-stopped

01--memory 512mHard memory limit. The container may not exceed this. Translated into `memory.max = 536870912`.
02--memory-reservation 256mSoft memory target. The kernel tries to reclaim memory from the container when host pressure rises. Translated into `memory.low = 268435456`.
03--cpus 1.5Equivalent to `--cpu-quota=150000 --cpu-period=100000`. Translated into `cpu.max = 150000 100000`.
04--cpu-shares 512Relative weight (default 1024). Only meaningful when CPU is contended. runc converts it with 1 + ((shares - 2) * 9999) / 262142, so 512 becomes `cpu.weight = 20` and the 1024 default becomes 39 (cgroup v2 uses weight 1–10000).
05--pids-limit 200Caps process/thread count. Translated into `pids.max = 200`. A value of -1 or 0 disables the limit.
06--blkio-weight 500Block I/O relative weight (default 500). runc converts it with 1 + (weight - 10) * 9999 / 990, so 500 becomes `io.weight = 4950` — not 500. The v1 and v2 numbers are on different scales.
07--device-read-bps /dev/sda:1mbPer-device read bandwidth cap. Translated into `io.max` for the cgroup.
08--device-read-iops /dev/sda:1000Per-device read IOPS cap. Same target file.
09--restart unless-stoppedNot a cgroup limit. This is a restart policy handled by dockerd, not by the kernel.

Show reconstructed command

--memory 512m --memory-reservation 256m --cpus 1.5 --cpu-shares 512 --pids-limit 200 --blkio-weight 500 --device-read-bps /dev/sda:1mb --device-read-iops /dev/sda:1000 --restart unless-stopped

The controllers and what they limit

Controller	cgroup file	Docker flag	Notes
`cpu`	`cpu.max`	`--cpus`	Quota per period (default 100 ms).
`cpu`	`cpu.weight`	`--cpu-shares`	Relative weight when contended.
`cpu`	`cpu.idle`	(none)	Whether the cgroup may use idle CPU.
`cpuset`	`cpuset.cpus`	`--cpuset-cpus`	Pin to specific CPUs.
`cpuset`	`cpuset.mems`	`--cpuset-mems`	Pin to specific NUMA nodes.
`memory`	`memory.max`	`--memory`	Hard cap.
`memory`	`memory.low`	`--memory-reservation`	Soft target.
`memory`	`memory.swap.max`	`--memory-swap`	Total memory + swap.
`memory`	`memory.high`	(none)	Throttle before OOM.
`pids`	`pids.max`	`--pids-limit`	Process/thread count.
`io`	`io.max`	`--device--bps/--device--iops`	Per-device bandwidth and IOPS.
`io`	`io.weight`	`--blkio-weight`	Relative weight.

The OOM killer in depth

When a container exceeds memory.max, the OOM-killer picks a process to kill. The choice is influenced by:

oom_score_adj — each process has a score adjustment in /proc/<pid>/oom_score_adj. Higher = killed first.
The cgroup’s memory.oom.group — when set, the OOM-killer kills the entire cgroup at once instead of picking individual processes. This is what you want for batch jobs but not for services.

Docker sets neither of these by default, and the consequence is the one that surprises people at 03:00. On a default Engine 28/29 host, memory.oom.group reads 0 for every container scope and the container’s PID 1 has oom_score_adj of 0 — runc’s cgroup v2 memory path writes only memory.max, memory.low and memory.swap.max. Check it on your own host rather than taking it on trust:

CID=$(docker ps -q | head -1)
PID=$(docker inspect --format '{{.State.Pid}}' "$CID")
CG=$(awk -F: '{print $3}' "/proc/$PID/cgroup" | head -1)
cat "/sys/fs/cgroup$CG/memory.oom.group"   # 0
cat "/proc/$PID/oom_score_adj"             # 0

Because memory.oom.group is 0, the kernel kills one process — the largest resident one in the cgroup, which is frequently a worker rather than PID 1. If PID 1 survives, the container stays running with a worker missing. docker ps shows it up, the health check may still pass, and throughput has quietly dropped. That partial OOM is far more common than the clean exit-137 case, and it is the reason memory.events is worth alerting on rather than container restarts.

--oom-score-adj is exposed on docker run if you want to bias the choice; there is no Docker flag that sets memory.oom.group.

The container was OOM-killed and `docker ps` still says it is up

Throughput on a queue consumer halves overnight. Nothing restarted: docker ps shows the container Up 4 days, RestartCount is 0, State.OOMKilled is false, the health check is green because the HTTP listener in PID 1 still answers. Latency is fine — the requests that get served are served quickly. There are simply fewer of them.

Because every restart-shaped signal says “healthy”, the investigation goes to the queue, the database, and the upstream producer. Days later somebody reads dmesg and finds an oom-kill line from Tuesday.

What actually happened. The cgroup hit memory.max. Because memory.oom.group is 0, the kernel did not kill the cgroup — it picked the single largest resident process in it and killed that one. In a multi-process container that is almost never PID 1; it is a worker, a forked child, a thread pool host. PID 1 survives, so the container’s state never changes, so Docker never reports anything. State.OOMKilled describes the container’s exit, and the container did not exit.

Why the obvious signals miss it. docker events, restart counts, exit code 137 and State.OOMKilled all key off the container dying. A partial OOM kills a process inside a container that keeps running, and none of those fire. Alerting on container restarts cannot detect it by construction.

How to tell. memory.events counts what happened to the cgroup, independently of whether anything exited. A non-zero oom_kill against an unchanged StartedAt and a RestartCount of 0 is the signature, and it is unambiguous: the kernel killed something in this container and the container is still the same one.

Read-only / Safedid this container lose a process without anyone noticing?— Read the kernel's counters and Docker's view side by side. They disagree exactly when a partial OOM has happened.

CONTAINER=worker
PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")
CG=$(awk -F: '{print $3}' "/proc/$PID/cgroup" | head -1)
docker inspect --format 'started={{.State.StartedAt}} restarts={{.RestartCount}} oomkilled={{.State.OOMKilled}}' "$CONTAINER"
cat "/sys/fs/cgroup$CG/memory.events"
echo "peak: $(cat "/sys/fs/cgroup$CG/memory.peak")  max: $(cat "/sys/fs/cgroup$CG/memory.max")"

started=2026-08-08T02:11:47.9Z restarts=0 oomkilled=false
low 0
high 0
max 1184
oom 6
oom_kill 6
oom_group_kill 0
sock_throttled 0
peak: 536870912  max: 536870912

Illustrative output

Every line in that output is doing work:

oom_kill 6 with restarts=0 — six processes were killed and the container never restarted. This is the partial OOM.
oom_group_kill 0 — confirms memory.oom.group is off; had it been on, the whole cgroup would have gone at once and the container would have exited 137.
max 1184 — the number of times allocation hit memory.max. This one climbs long before anything is killed, which makes it the early warning oom_kill is not.
peak equal to max — the cgroup has been pinned at its ceiling, not merely brushing it.

A healthy container reads oom 0, oom_kill 0, and a peak comfortably below max. Note that all of these counters live in the cgroup, so a container restart creates a new cgroup and resets them to zero — which is another reason to scrape them into a time series rather than read them after the fact.

Orphan cgroups

A container is deleted but its cgroup is not. This happens when containerd crashes mid-deletion, or when an operator deletes a container with docker rm -f while containerd is down.

You will notice this when docker ps -a shows nothing but /sys/fs/cgroup/system.slice/docker-<id>.scope/ still exists. The processes are gone (the kernel reaped them) but the cgroup directory is still there with cgroup.procs containing stale PIDs.

The fix is usually to clean them up with systemd-cgtop and manual deletion, or to bounce containerd. In production, monitor for orphan cgroups — a slow leak here eats host memory.

Knowledge check

Where next

OverlayFS — how Docker builds container filesystems from image layers. After that, capabilities, then the rest of the internals chapter.