cgroups v2 and Linux CFS quotas — the kernel primitives

The kubelet’s resource enforcement is delegated to Linux cgroups. To diagnose resource issues at the deepest level, you need to read cgroup files on the node. This lesson covers cgroups v2 (the modern kernel API), the key files for CPU and memory, and the diagnostic patterns for throttling and OOMKill.

cgroups v1 vs v2

Two versions of the cgroups API exist in the kernel:

• cgroups v1: legacy; separate hierarchies for each controller (cpu, memory, blkio, etc.). Each controller has its own mount point and file structure.
• cgroups v2: unified hierarchy. All controllers are under /sys/fs/cgroup/ with consistent naming.

Kubernetes 1.25+ requires cgroups v2 on the node (--cgroupv2 kubelet flag, or detected automatically on modern distros). cgroups v2 provides:

• Unified hierarchy: one tree, all controllers.
• Better resource isolation: less leakage between controllers.
• Pressure stall information (PSI): kernel signals of resource pressure, useful for autoscaling.
• Cleaner file layout: less confusing for operators debugging at the node level.

cgroups v2 file layout

The kubelet creates a cgroup for each container under the Pod’s cgroup:

/sys/fs/cgroup/
├── kubepods.slice/                      # Pod-level cgroup
│   ├── kubepods-burstable.slice/        # QoS: Burstable Pods
│   │   ├── pod<uid>/
│   │   │   ├── container-<id>-scope/    # main container
│   │   │   │   ├── cpu.max
│   │   │   │   ├── memory.max
│   │   │   │   ├── cpu.stat
│   │   │   │   ├── memory.current
│   │   │   │   └── memory.events
│   │   │   └── container-<id>-scope/    # sidecar
│   │   │       ├── ...
│   │   │       └── ...
│   │   └── ...
│   └── kubepods-besteffort.slice/       # QoS: BestEffort
└── kubepods-podslice.slice/             # some distros use this path

The exact path depends on the container runtime (containerd vs CRI-O) and the distro. Use find /sys/fs/cgroup -name "cpu.max" to locate cgroup files.

cpu.max — the CPU limit

$ cat /sys/fs/cgroup/.../container-<id>-scope/cpu.max
50000 100000

The format is quota period in microseconds:

• quota (50000): the container can use 50000 µs of CPU time per period.
• period (100000): the period is 100000 µs (100 ms).

The container’s CPU share is quota / period = 50%. If the container tries to use more, the kernel throttles it.

A container with no CPU limit has cpu.max = max (unlimited):

$ cat /sys/fs/cgroup/.../container-<id>-scope/cpu.max
max 100000

cpu.stat — throttling counters

$ cat /sys/fs/cgroup/.../container-<id>-scope/cpu.stat
usage_usec 12345678901
user_usec 11000000000
system_usec 1345678901
nr_periods 123456
nr_throttled 4567
throttled_usec 987654321
nr_burst 0
burst_usec 0

Key fields:

• usage_usec: total CPU time the container has used.
• nr_periods: number of CFS periods elapsed.
• nr_throttled: number of periods the container was throttled.
• throttled_usec: total time the container spent throttled.

To calculate the throttling ratio:

throttled_pct=$(awk '{print $5 / ($4 * 0.1)}' cpu.stat)

Where $5 is throttled_usec, $4 is nr_periods, and 0.1 is the period in microseconds (100 ms = 100000 µs = 0.1 s). The result is the percentage of time the container was throttled.

A healthy container has throttled_pct < 1%. A throttled container (>10%) is starving for CPU.

memory.max — the memory limit

$ cat /sys/fs/cgroup/.../container-<id>-scope/memory.max
536870912

The value is in bytes: 536870912 = 512 MiB. A container with no memory limit has memory.max = max (unlimited).

memory.current — actual usage

$ cat /sys/fs/cgroup/.../container-<id>-scope/memory.current
268435456

The value is in bytes: 268435456 = 256 MiB. The container is using 256 MiB out of its 512 MiB limit.

memory.events — OOMKill counter

$ cat /sys/fs/cgroup/.../container-<id>-scope/memory.events
low 0
high 0
max 5
oom 0
oom_kill 3
pgscan 1234567
pgmajfault 12

Key fields:

• max: number of times the container hit the memory limit.
• oom: number of times the cgroup was OOM-killed (the cgroup as a whole).
• oom_kill: number of times a process inside the cgroup was OOM-killed.

For a container that OOMKilled 3 times: oom_kill 3. This is the ground truth for OOMKill; the kubelet’s OOMKilled reason comes from this counter.

Reading PSI (Pressure Stall Information)

cgroups v2 exposes PSI for CPU, memory, and IO:

cat /sys/fs/cgroup/.../cpu.pressure
# some avg10=0.00 avg60=0.01 avg300=0.05 total=12345
cat /sys/fs/cgroup/.../memory.pressure
# some avg10=2.34 avg60=1.50 avg300=0.80 total=987654
cat /sys/fs/cgroup/.../io.pressure
# some avg10=0.00 avg60=0.00 avg300=0.00 total=0

The avg10, avg60, avg300 are the percentage of time some task was stalled waiting for the resource. some means at least one task; full means all tasks.

For example, memory.pressure: some avg10=2.34 means 2.34% of the last 10 seconds had at least one task stalled waiting for memory.

Diagnosing resource issues at the node

The full diagnostic for a slow Pod:

# 1. Identify the Pod's cgroup
POD_UID=$(kubectl get pod web-7c8 -o jsonpath='{.metadata.uid}')
CGROUP="/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/pod${POD_UID}"

# 2. Find the container's cgroup
ls $CGROUP

# 3. Read CPU throttling
cat $CGROUP/container-*/cpu.stat

# 4. Read memory usage and events
cat $CGROUP/container-*/memory.current
cat $CGROUP/container-*/memory.events

# 5. Read PSI
cat /sys/fs/cgroup/cpu.pressure
cat /sys/fs/cgroup/memory.pressure

This gives you the ground truth. Compare with the Pod’s declared resources:

kubectl get pod web-7c8 -o jsonpath='{.spec.containers[*].resources}'

If throttled_usec is high and the CPU limit is the declared value, the workload needs more CPU. If memory.events: oom_kill is non-zero, the memory limit is too low.

Cross-course references

• The Linux course part I-Linux-Foundations covers cgroups v1 and v2; the kubelet’s enforcement uses these primitives.
• The Linux course part XXXVII-Linux-Resources covers resource management; cgroups v2 is the modern implementation.
• The Observability course part X-Observability-NodeExporter covers node-exporter’s cgroup metrics; these are the Prometheus surface for the same data.

Quiz

Production discipline

• Run on cgroups v2. Verify with stat -fc %T /sys/fs/cgroup/. v1 is deprecated in 1.34+.
• Read cgroup files when the kubelet’s view is insufficient. Throttling, OOMKill, PSI are the ground truth at the kernel level.
• Monitor throttled_usec and oom_kill counters. These are the leading indicators of resource pressure.
• Use PSI for early warning. Alert on avg10 > 10% as the leading signal before throttling/OOMKill happens.
• Audit cgroup paths. The kubelet creates cgroups under kubepods.slice/kubepods-<qos>.slice/pod<uid>/; know where to look when debugging.