cgroups v2 architecture - the Linux resource control subsystem

cgroups (control groups) are the Linux kernel mechanism for limiting and accounting resources: CPU, memory, I/O, network. This lesson covers cgroups v2 and how to use them.

What cgroups do

cgroups provide:

• Resource limits: cap CPU, memory, I/O bandwidth per cgroup.
• Accounting: measure resource usage per cgroup.
• Prioritisation: assign priorities to cgroups.
• Control: freeze, thaw, kill cgroups.

For a web server, you can limit a process to 4 CPUs and 4 GB of memory. If it tries to use more, the kernel throttles or rejects.

cgroups v2

Modern Linux uses cgroups v2 (unified hierarchy). One hierarchy, multiple controllers:

• cpu and cpuacct (merged): CPU shares and usage.
• memory: memory limits.
• io: block I/O.
• pids: process count.
• cpuset: CPU pinning.

cgroup v2 is a single unified hierarchy with exactly one mount, at /sys/fs/cgroup. Controllers are not mounted separately the way they were in v1; they are enabled per-subtree through cgroup.subtree_control.

Inspect cgroups

# Show current process cgroups
cat /proc/self/cgroup

# Show cgroup hierarchy
ls /sys/fs/cgroup/
ls /sys/fs/cgroup/system.slice/

# Show cgroup memory usage
cat /sys/fs/cgroup/system.slice/nginx.service/memory.current

# systemd-cgtop: live view
systemd-cgtop

Apply limits via systemd

systemd uses cgroups for every service:

[Service]
CPUQuota=200%          # 2 CPUs
MemoryMax=4G           # 4 GB
MemoryHigh=3G          # soft limit
IOWeight=100           # I/O weight

Apply:

sudo systemctl daemon-reload
sudo systemctl restart my-service

systemd creates a cgroup under /sys/fs/cgroup/system.slice/ for the service.

Apply limits via systemd-run

For ad-hoc limits:

systemd-run --scope -p CPUQuota=50% -p MemoryMax=500M -- stress-ng --cpu 4 --vm 2 --vm-bytes 1G

The stress-ng runs in a scope with the given limits.

Apply limits directly

Two v2 rules decide whether this works at all, and neither is obvious from the filesystem.

Rule 1 - a controller only exists in a child if the parent enabled it. memory.max is not present in a new directory unless the parent lists memory in its cgroup.subtree_control. A mkdir that appears to succeed followed by “No such file or directory” writing memory.max is this rule, not a typo.

Rule 2 - no internal processes. Once a cgroup has children with controllers enabled, it may not hold processes of its own. Workloads live in the leaves.

# What can a child of the root actually use?
cat /sys/fs/cgroup/cgroup.subtree_control

# Enable the controllers you need for children of the root
echo '+memory +cpu' | sudo tee /sys/fs/cgroup/cgroup.subtree_control

# Create a leaf. Do NOT name it *.slice - that suffix is
# systemd's namespace and systemd will try to manage it.
sudo mkdir /sys/fs/cgroup/myapp

# Set memory limit
echo 1G | sudo tee /sys/fs/cgroup/myapp/memory.max

# Add a process. Use a PID you actually have - $$ is this shell.
echo $$ | sudo tee /sys/fs/cgroup/myapp/cgroup.procs

# Inspect
cat /sys/fs/cgroup/myapp/memory.current
cat /sys/fs/cgroup/myapp/memory.events

memory.events shows OOM kills and other events.

OOM behaviour

When a cgroup exceeds its memory.max, the kernel triggers OOM. By default, the OOM killer kills the offending process. To reduce the chance of that, give the kernel room to reclaim before it hits the wall:

[Service]
# Reclaim and throttling start here
MemoryHigh=3G
# Hard wall. Crossing it means a cgroup OOM kill.
MemoryMax=4G
# MemorySwapMax is deliberately NOT set: swap is the escape valve.
Restart=on-failure

Above MemoryHigh the kernel reclaims aggressively and throttles the cgroup rather than killing it. Be honest about what that costs: a service whose working set genuinely exceeds MemoryHigh becomes slow enough to fail health checks, and it is still OOM-killed the moment it reaches MemoryMax. MemoryHigh buys time and a warning signal in memory.events; it does not make a leak survivable.

Knowledge check