Where the numbers come from — cgroup v2 metric sources

docker stats and cAdvisor are not two sources of truth. They are two readers of the same files under /sys/fs/cgroup. When they disagree, one of them is doing arithmetic the other is not, and knowing which arithmetic is the difference between a working alert and a false one.

Finding the cgroup

With the systemd cgroup driver — the default on any systemd host — each container gets a scope named after its full ID:

$ docker info --format 'cgroup {{.CgroupVersion}} / driver {{.CgroupDriver}}'
CID=$(docker inspect grafana --format '{{.Id}}')
CG=/sys/fs/cgroup/system.slice/docker-$CID.scope
ls "$CG" | grep -E '^(memory|cpu|pids|io)\.' | head -14

cgroup 2 / driver systemd
cpu.max
cpu.pressure
cpu.stat
cpu.weight
io.pressure
io.stat
memory.current
memory.events
memory.max
memory.peak
memory.pressure
memory.stat
pids.current
pids.max

Everything below is a plain file. cat is a valid monitoring agent.

Memory

$ cat "$CG/memory.current" "$CG/memory.max" "$CG/memory.peak"
grep -E '^(anon|file|inactive_file|active_file|kernel) ' "$CG/memory.stat"
cat "$CG/memory.events"

297140224
536870912
346812416
anon 71393280
file 216993792
inactive_file 93097984
active_file 123895808
kernel 8499200
low 0
high 0
max 0
oom 0
oom_kill 0
oom_group_kill 0

The discrepancy is not a bug, and reproducing the arithmetic is the whole point of this lesson:

memory.current   297140224   283 MiB   everything charged to the cgroup
inactive_file     93097984    88 MiB   reclaimable page cache
                 ----------
current - inactive_file      194 MiB   what docker stats prints

docker stats and cAdvisor’s container_memory_working_set_bytes both report working set — current usage minus the page cache the kernel can evict under pressure. That is the right number to alert on, because it approximates what the container cannot give back.

memory.current is what the kernel compares against memory.max when deciding to OOM-kill. A container whose memory.current is pinned at the limit while its working set is comfortable is not in trouble; it is using its whole allowance as file cache, which is what caches are for.

CPU, and the metric that is actually missing

$ cat "$CG/cpu.max"
cat "$CG/cpu.stat"

100000 100000
usage_usec 4855653170
user_usec 3681038790
system_usec 1174614380
nice_usec 0
core_sched.force_idle_usec 0
nr_periods 3411231
nr_throttled 9
throttled_usec 1488817
nr_bursts 0
burst_usec 0

usage_usec is what every dashboard shows. nr_throttled and throttled_usec are what explain your latency.

CFS enforces a quota per 100 ms period. A container that uses its entire quota in the first 30 ms of a period is stopped for the remaining 70 ms, no matter how idle the host is. Averaged over a minute the same container reports comfortable CPU usage. The p99 latency graph shows the truth and nobody can explain it.

PIDs and block I/O

cat "$CG/pids.current" "$CG/pids.max"
cat "$CG/io.stat"

pids.current against pids.max catches fork bombs and thread leaks before they take the host’s PID table with them. io.stat is per block device, keyed by major:minor:

8:0 rbytes=378687488 wbytes=13639680 rios=2996 wios=1948 dbytes=0 dios=0

Map the device with lsblk -o NAME,MAJ:MIN when the numbers matter.

Pressure stall information

The most useful saturation signal on a modern kernel, and the one almost no Docker monitoring stack collects:

$ cat "$CG/memory.pressure"

some avg10=0.00 avg60=0.00 avg300=0.00 total=884
full avg10=0.00 avg60=0.00 avg300=0.00 total=884

cpu.pressure, memory.pressure and io.pressure answer the question utilisation cannot: not “how much of the resource is in use” but “how much time did work spend waiting for it”. A disk at 60% utilisation with io.pressure some avg60=40.0 is a disk that is hurting you.

Mapping to Prometheus

Everything above has a cAdvisor equivalent. Collect the right-hand column and you have covered the sources.

Sanity check