Auditing resource limits — proving the limit you set is the limit in force

The previous lessons in this part covered how to set CPU, memory, PID and I/O limits. This one starts from an uncomfortable observation: on most hosts, a meaningful fraction of the limits somebody wrote down are not in force, and nothing anywhere reports that.

There is no warning, no log line and no error. The container starts, runs, and is unlimited.

Four layers, and a limit can be lost between any two

flowchart LR
  A[Compose file<br/>declared] --> B[Container config<br/>HostConfig]
  B --> C[cgroup files<br/>enforced]
  C --> D[Application<br/>observed]

• Declared — what the Compose file or run script says.
• Configured — what the daemon recorded when the container was created, visible in HostConfig.
• Enforced — what the kernel has in the cgroup files.
• Observed — what the workload actually did, and whether it ever met the ceiling.

An audit walks all four. Checking only the first is how the gap survives; checking only the last is how you conclude a limit works because nothing has broken yet.

Layer 2: what the daemon recorded

$ docker inspect api --format 'mem={{.HostConfig.Memory}} swap={{.HostConfig.MemorySwap}} nanocpus={{.HostConfig.NanoCpus}} shares={{.HostConfig.CpuShares}} pids={{.HostConfig.PidsLimit}}'

mem=536870912 swap=-1 nanocpus=500000000 shares=0 pids=0

Read the zeroes. 0 means unset, not zero. So this container has a 512 MiB memory limit and half a CPU, and no PID limit at all — a fork bomb inside it can exhaust the host’s process table, as the PIDs lesson in this part describes.

Across the whole host, in one command:

$ docker ps -q | xargs -r docker inspect --format '{{.Name}} {{.HostConfig.Memory}}' | awk '$2 == 0 {print $1}'

/legacy-cron
/adminer
/redis

That list is the actual finding on most hosts, and it is usually longer than anyone expects.

Layer 3: what the kernel is enforcing

HostConfig is the daemon’s record of intent. The cgroup files are the enforcement. They should agree, and checking is cheap:

$ cat /proc/$(docker inspect api --format '{{.State.Pid}}')/cgroup

0::/system.slice/docker-c3f81a04b7d29e6a5f0c8b3d1e7a92f4068c5b1d3a7e9f2b4c6d8e0a2f4b6c8d.scope

$ cd /sys/fs/cgroup/system.slice/docker-c3f81a04b7d2*.scope && grep . memory.max memory.swap.max cpu.max pids.max

memory.max:536870912
memory.swap.max:max
cpu.max:50000 100000
pids.max:max

memory.max matches. cpu.max of 50000 100000 is half a CPU, as configured. pids.max: max confirms the missing PID limit. And memory.swap.max: max is the one worth stopping on.

Layer 4: did it ever actually hit the limit?

This is the part that turns an audit into a decision, and it is where cgroup v2 is genuinely good: the kernel counts.

$ cd /sys/fs/cgroup/system.slice/docker-c3f81a04b7d2*.scope && cat memory.events memory.peak

low 0
high 0
max 1284
oom 3
oom_kill 1
oom_group_kill 0
1073741824

Every line is a fact you cannot get any other way:

• max 1284 — the memory limit was reached 1284 times and the kernel reclaimed to stay under it. The container did not die, and it spent real time doing reclaim instead of work. This is the number that identifies a limit set too tight, and nothing in docker stats shows it.
• oom 3 — three times reclaim was not enough.
• oom_kill 1 — one process was killed. This is the counter that answers “was my container OOM-killed” without relying on the exit code surviving a restart.
• memory.peak — the high-water mark in bytes. 1 GiB here, against a 512 MiB limit, is only possible because swap was unlimited.

The CPU equivalent:

$ cd /sys/fs/cgroup/system.slice/docker-c3f81a04b7d2*.scope && grep -E 'nr_periods|nr_throttled|throttled_usec' cpu.stat

nr_periods 184204
nr_throttled 21847
throttled_usec 43918204

nr_throttled / nr_periods is the throttled fraction: 21847/184204 is about 12%, meaning that in one period in eight the container exhausted its quota and was stopped until the next period began. Above a few per cent, the --cpus value is the cause of your latency, and no amount of application profiling will show it — from inside the container the process simply was not scheduled.

$ cd /sys/fs/cgroup/system.slice/docker-c3f81a04b7d2*.scope && grep . pids.peak pids.max

pids.peak:534
pids.max:max

pids.peak is how you choose a --pids-limit from evidence rather than from a guess: observe the real peak over a busy week, then set the limit at a comfortable multiple of it.

Where limits get lost

There is a fifth that no cgroup file will reveal, because the limit is in force and the application is ignoring it:

An audit you can run on a whole host

#!/usr/bin/env bash
# audit-limits.sh - declared vs enforced vs observed, per container.
set -euo pipefail

printf '%-22s %12s %10s %10s %8s %10s\n' \
  CONTAINER MEM_MAX CPU_MAX PIDS_MAX OOM_KILL THROTTLED

for id in $(docker ps -q); do
  name=$(docker inspect "$id" --format '{{.Name}}' | tr -d /)
  pid=$(docker inspect "$id" --format '{{.State.Pid}}')
  rel=$(sed 's/^0:://' "/proc/$pid/cgroup")
  cg="/sys/fs/cgroup$rel"

  [ -d "$cg" ] || continue

  mem=$(cat "$cg/memory.max")
  cpu=$(awk '{print $1}' "$cg/cpu.max")
  pids=$(cat "$cg/pids.max")
  oom=$(awk '/^oom_kill/ {print $2}' "$cg/memory.events")
  thr=$(awk '/^nr_throttled/ {print $2}' "$cg/cpu.stat")

  printf '%-22s %12s %10s %10s %8s %10s\n' \
    "$name" "$mem" "$cpu" "$pids" "$oom" "$thr"
done

$ ./audit-limits.sh

CONTAINER                   MEM_MAX    CPU_MAX   PIDS_MAX OOM_KILL  THROTTLED
api                       536870912      50000        max        1      21847
postgres                        max        max        max        0          0
redis                     268435456        max       4096        0          0
legacy-cron                     max        max        max        0          0

Three findings in four rows. postgres and legacy-cron have no limits of any kind, so either can take the host down. api is being throttled heavily and has been OOM-killed once, which means its limits are set below what it needs. redis is the only correctly configured row, and it too has no CPU cap.

1. List containers with HostConfig.Memory of 0 — these have no memory limit.
2. For each container that does have one, confirm memory.swap.max is not max, or the ceiling is soft.
3. Read memory.events for oom_kill above zero, and for max climbing — the second is a tight limit nobody has noticed.
4. Read cpu.stat for a throttled fraction above a few per cent.
5. Read pids.peak and set --pids-limit from it rather than from a guess.
6. Re-run after any deploy. A recreated container is a new cgroup and the counters reset to zero.

Knowledge check

Where next

That closes the resource-controls part. The performance part takes these same counters and asks the next question: given that the limits are correct, where is the time actually going?