cgroup accounting - proving which limit was actually hit · Linux

Setting a limit is easy. Proving that a limit was hit is the part that closes an incident, and it is a different skill. A service that is slow because it is CPU-throttled and a service that is slow because the database is slow look identical from the application logs. cgroup v2 records the difference, per service, as a counter you can read.

This lesson is about the accounting files, not the limit files. The previous lessons in this part taught you to write CPUQuota=, MemoryMax= and TasksMax=. This one teaches you to read what happened afterwards.

Where the numbers live

Every systemd unit has a cgroup, and every cgroup directory contains the accounting files for the controllers enabled on it. Find the path first:

systemctl show nginx.service -p ControlGroup

ControlGroup=/system.slice/nginx.service

The files are under /sys/fs/cgroup plus that path:

ls /sys/fs/cgroup/system.slice/nginx.service/

cgroup.controllers  cpu.pressure  cpu.stat        io.pressure  memory.current
cgroup.events       cpu.max       cpu.stat.local  io.stat      memory.events
cgroup.procs        cpu.weight    cpu.uclamp.max  io.max       memory.max
cgroup.stat         cpu.idle      cpuset.cpus     io.weight    memory.peak
cgroup.threads      cpu.max.burst cpuset.mems     memory.high  memory.pressure
pids.current        pids.events   pids.max        pids.peak    memory.stat

A file that is absent means the controller is not enabled on that cgroup, which is itself a finding: no io.stat means no I/O accounting for this service, so you cannot answer an I/O question about it until you enable the controller.

cat /sys/fs/cgroup/system.slice/cgroup.subtree_control

cpuset cpu io memory hugetlb pids rdma misc dmem

That line lists the controllers a parent has enabled for its children. Controllers propagate downwards only when the parent enables them in cgroup.subtree_control, so a missing controller deep in the tree is usually a parent that never turned it on.

CPU: throttling is a counter, not an inference

cpu.stat is the single most useful accounting file on the system, because it separates “used a lot of CPU” from “was prevented from using CPU”:

cat /sys/fs/cgroup/system.slice/nginx.service/cpu.stat

usage_usec 86005392007
user_usec 66877054358
system_usec 19128337649
nice_usec 0
core_sched.force_idle_usec 0
nr_periods 0
nr_throttled 0
throttled_usec 0
nr_bursts 0
burst_usec 0

The first four lines are consumption. The interesting ones are the last four:

Field	Meaning
`nr_periods`	Enforcement periods that have elapsed since a quota was set
`nr_throttled`	Periods in which the cgroup exhausted its quota and was stopped
`throttled_usec`	Total microseconds the cgroup spent stopped, waiting for the next period
`nr_bursts` / `burst_usec`	Periods where `cpu.max.burst` credit was consumed

nr_periods 0 above means no quota is configured. With no CPUQuota=, there is no enforcement period, so nothing is counted and nothing can be throttled. That is the expected reading on an unlimited service, and it is also how you confirm that the limit you thought you deployed is not actually in effect.

On a service that does have a quota, the reading looks like this:

nr_periods 145200
nr_throttled 41833
throttled_usec 902114000

Turn it into a percentage, which is the number worth alerting on:

CG=/sys/fs/cgroup/system.slice/api.service
awk '/nr_periods/ {p=$2} /nr_throttled/ {t=$2} END {
  if (p > 0) printf "throttled in %.1f%% of periods (%d of %d)\n", 100*t/p, t, p
  else print "no quota configured, nothing to throttle"
}' "$CG/cpu.stat"

throttled in 28.8% of periods (41833 of 145200)

Nearly 29% of enforcement periods ended with the service stopped and waiting. That is a latency problem the application team cannot see and cannot fix, and it is caused entirely by CPUQuota=.

Throttling happens at well under 100% average CPU use

The intuition that a quota only bites when a service is “using all its CPU” is wrong, and the reason is that the quota is enforced per period - 100 ms by default - not per second.

A service with CPUQuota=100% gets 100 ms of CPU time per 100 ms period. A request handler that briefly fans out across four threads consumes its whole period’s budget in 25 ms of wall clock, then sits stopped for the remaining 75 ms. Averaged over a minute, the service shows perhaps 30% CPU. Averaged over the period, it hit the ceiling repeatedly.

So the dashboard shows a service comfortably under its limit while nr_throttled climbs and p99 latency has a 75 ms floor welded into it. Utilisation cannot see this; only cpu.stat can. This is the same lesson the storage chapter teaches about %util: an average across a window hides a saturation that happens inside the window.

The fix is usually to raise the quota, or to remove it and rely on CPUWeight= instead. CPUWeight= is a share under contention rather than a hard ceiling: an unloaded host lets the service use everything, and a loaded host divides CPU proportionally. A ceiling wastes idle CPU; a weight does not. Reach for CPUQuota= when you need a predictable ceiling for capacity planning or for a tenant guarantee, and for CPUWeight= otherwise.

Memory: events, current, and peak

memory.current is a live gauge and tells you almost nothing on its own. The two files that carry evidence are memory.events and memory.peak.

cat /sys/fs/cgroup/system.slice/api.service/memory.events

low 0
high 0
max 0
oom 0
oom_kill 0
oom_group_kill 0
sock_throttled 0

Each counter names a different event, and they escalate:

Counter	What happened
`low`	Usage went above `memory.low`, so protection from reclaim was dropped
`high`	Usage hit `memory.high`; the kernel throttled the cgroup and reclaimed
`max`	Usage hit `memory.max`; allocation was blocked while the kernel reclaimed hard
`oom`	Reclaim failed and the cgroup entered OOM
`oom_kill`	A process in this cgroup was killed by the cgroup OOM killer
`oom_group_kill`	The whole cgroup was killed together, because `memory.oom.group` was set

The distinction that matters most in practice is high versus max. A rising high count means the service is being slowed down by reclaim - it is alive, it is meeting its limit, and it is paying for it in latency. A rising max count means it is being stopped. Neither produces a log line in the application.

An all-zero memory.events on a service you believe was memory-limited proves the limit was never reached, which usually means the kill you are investigating came from the system-wide OOM killer or from systemd-oomd, not from this cgroup’s limit.

Size the limit from memory.peak

memory.peak is the high-water mark of memory.current since the cgroup was created, and it is the number to size a limit from:

CG=/sys/fs/cgroup/user.slice/user-1000.slice
printf 'current %s\npeak    %s\nmax     %s\n' \
  "$(cat "$CG/memory.current")" "$(cat "$CG/memory.peak")" "$(cat "$CG/memory.max")"

current 5492432896
peak    47019118592
max     max

5.5 GB right now, 47 GB at some point since boot. A limit set from the current reading would have killed this workload; a limit set from the peak plus headroom would not. Read it in human units with numfmt:

numfmt --to=iec 5492432896 47019118592

5.2G
44G

memory.peak is since cgroup creation, and a service restart resets it

The cgroup is created when the unit starts and destroyed when it stops, so memory.peak covers the current invocation only. A service that restarted an hour ago has an hour-old peak, which is useless for sizing a limit against a nightly batch job.

Two consequences. Read memory.peak after the workload’s busiest period has happened within the current invocation, not after a deploy. And record it: a small timer that appends memory.peak per service to a file gives you the weeks of history that sizing a limit actually requires.

On kernels from 6.x, memory.peak can be reset by writing to it, which lets a benchmark harness measure one run cleanly. Check that the write is accepted before relying on it, because on earlier kernels the file is read-only and the write fails.

Processes and I/O

pids.current against pids.max catches fork bombs and thread leaks, and pids.events records the refusals:

CG=/sys/fs/cgroup/user.slice/user-1000.slice
printf 'current %s / max %s / peak %s\n' \
  "$(cat "$CG/pids.current")" "$(cat "$CG/pids.max")" "$(cat "$CG/pids.peak")"
cat "$CG/pids.events"

current 54 / max 124286 / peak 675
max 0

max 0 in pids.events means no fork was ever refused. A non-zero value there is the definitive answer to “why did the service fail to start a worker” - the application will have logged Resource temporarily unavailable and nothing more.

io.stat gives per-device byte and operation counts for the cgroup:

cat /sys/fs/cgroup/system.slice/io.stat

7:4 rbytes=14336 wbytes=0 rios=11 wios=0 dbytes=0 dios=0
259:0 rbytes=48213770240 wbytes=9917349888 rios=1264810 wios=331477 dbytes=0 dios=0

The first field is major:minor; resolve it with lsblk:

lsblk -o NAME,MAJ:MIN,SIZE,TYPE

NAME        MAJ:MIN  SIZE TYPE
nvme0n1     259:0    1.8T disk
├─nvme0n1p1 259:1      1G part
└─nvme0n1p2 259:2    1.8T part

Missing io.stat is not zero I/O - it is no accounting

io.stat reports only devices the cgroup has actually touched, and the file exists only where the io controller is enabled. On a host where the parent slice never enabled io in cgroup.subtree_control, the file is missing entirely, or present with lines reading (unknown).

The failure mode is concluding “this service does no I/O” from an empty file. Check cgroup.controllers in the cgroup first:

cat /sys/fs/cgroup/system.slice/nginx.service/cgroup.controllers

If io is not in that list, the number you want does not exist yet. Enable it by setting IOAccounting=yes on the unit, which makes systemd turn the controller on for the unit and its parents. Note that the io controller has real overhead on a busy host, which is why it is not on everywhere by default.

Attribute the number to a service

Reading one cgroup requires knowing which one to read. systemd-cgtop ranks them:

systemd-cgtop -n 3 -b --depth=2 --order=memory

CONTROL GROUP                            TASKS   %CPU   MEMORY  INPUT/S OUTPUT/S
/                                          687      -     3.0G        -        -
system.slice                               439      -     2.1G        -        -
user.slice                                 248      -     5.1G        -        -
system.slice/snapd.service                   18      -    58.1M        -        -
system.slice/unattended-upgrades.service      2      -    18.4M        -        -

Two things to know before you trust this output.

A dash in a column means no data, not zero. The %CPU column needs two samples to compute a delta, so a single-iteration run shows dashes; -n 3 fixes it. The INPUT/S and OUTPUT/S columns show dashes wherever the io controller is not enabled, for the reason above.

Second, systemd-cgtop is a live ranking, so it is a triage tool rather than an evidence tool. Once it points at a unit, read that unit’s accounting files directly - those are the numbers you can paste into an incident review.

systemctl status presents a subset inline, which is often enough:

systemctl status nginx.service

● nginx.service - A high performance web server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled)
     Active: active (running) since Mon 2026-08-10 09:12:04 UTC; 1 day 6h ago
      Tasks: 9 (limit: 38350)
     Memory: 412.3M (peak: 1.1G, max: 2.0G, available: 1.6G)
        CPU: 4h 21min 8.442s
     CGroup: /system.slice/nginx.service

peak: 1.1G against max: 2.0G is the sizing conversation in one line: the limit is roughly double the observed peak, which is a defensible margin.

CPUAccounting= is deprecated - and that is good news

On systemd 258 and later, CPUAccounting= is deprecated because CPU accounting is always available on the unified cgroup hierarchy and the setting has no effect. systemd-analyze verify says so about units that still carry it:

Support for option CPUAccounting= has been removed and it is ignored

The practical consequence: you do not need to enable anything to read cpu.stat, memory.current, memory.peak or pids.current. They are there right now, on every service on every host you run, with history since the unit started.

IOAccounting= is the exception - the io controller carries overhead and is still opt-in. Everything else in this lesson is free, which means there is no excuse for closing an incident with “we think it ran out of memory”.

A worked attribution

A service is reported slow. Three files answer it in under a minute:

CG=$(systemctl show api.service -p ControlGroup --value)
cd "/sys/fs/cgroup${CG}"

grep -E 'nr_periods|nr_throttled|throttled_usec' cpu.stat
cat memory.events
cat memory.pressure

nr_periods 145200
nr_throttled 41833
throttled_usec 902114000
low 0
high 1204
max 0
oom 0
oom_kill 0
oom_group_kill 0
some avg10=0.00 avg60=0.12 avg300=0.31 total=88214551
full avg10=0.00 avg60=0.08 avg300=0.19 total=61003422

Read together: the service is CPU-throttled in 29% of periods, and it has been throttled by memory reclaim 1204 times without ever hitting a hard limit or an OOM. Both limits are too tight. Neither fact is visible in the application logs, in top, or on a CPU-utilisation dashboard - and both are recorded, for free, on every host.