Memory capacity — sizing limits against what the cgroup actually counts

The overview lesson gave you a formula with a safety margin in it. This lesson gives you the number to put into that formula, because the obvious source — “what docker stats says” — is a derived figure, and the raw figure underneath it counts things that are not your application’s memory at all.

Get this wrong in the generous direction and you waste a third of a host. Get it wrong in the tight direction and the OOM killer takes the container down at 03:00 under exactly the load you sized it for.

The three things inside memory.current

memory.current is the cgroup’s total charged memory. It is the sum of several distinct populations with completely different behaviour under pressure:

Everything in that table is charged to the limit. That is the part people are surprised by: reading a 4 GB file inside a container with a 512 MB limit does not OOM the container, because the kernel reclaims clean page cache instead — but it does push memory.current right up against memory.max while it happens, and every dashboard that plots memory.current / memory.max will show 99%.

Reading the real numbers

CID=web
PID=$(docker inspect --format '{{.State.Pid}}' "$CID")
CG=$(awk -F: '{print $3}' "/proc/$PID/cgroup" | head -1)
echo "/sys/fs/cgroup$CG"

CID=web
PID=$(docker inspect --format '{{.State.Pid}}' "$CID")
CG=$(awk -F: '{print $3}' "/proc/$PID/cgroup" | head -1)
D="/sys/fs/cgroup$CG"
cat "$D/memory.current" "$D/memory.max" "$D/memory.peak"
grep -E '^(anon|file|shmem|slab|inactive_file|active_file) ' "$D/memory.stat"

264437760
536870912
300359680
anon 31805440
file 230162432
shmem 0
slab 1034832
inactive_file 117424128
active_file 112738304

Read that capture carefully, because every number in it matters:

• memory.max is 536870912 — a 512 MiB limit.
• memory.current is 264437760 — 252.2 MiB, or 49% of the limit.
• Of that 252.2 MiB, only anon 31805440 (30.3 MiB) is the application’s own heap and stack.
• file 230162432 (219.5 MiB) is page cache the container happened to touch. It is 87% of the reported usage and it is almost all disposable.
• memory.peak is 300359680 (286.4 MiB) — the high-water mark since the container started, which is the number a five-minute polling interval will miss.

Why docker stats disagrees with memory.current

docker stats for the same container reports 140.2 MiB, not 252.2 MiB:

docker stats --no-stream --format '{{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}' web

web	140.2MiB / 512MiB	27.38%

The difference is exactly inactive_file:

memory.current        264437760  =  252.2 MiB
minus inactive_file   117424128  =  112.0 MiB
                      ---------
docker stats MEM USAGE            =  140.2 MiB

On cgroup v2 Docker subtracts inactive_file — the cold end of the page cache LRU — before reporting. It is a reasonable approximation of “memory that is actually being used”, and it is not the number the kernel compares against memory.max. The kernel compares memory.current.

The sizing formula

The limit has to cover the parts that cannot be reclaimed on demand, plus enough cache for the workload not to thrash, plus margin:

working_set   = peak(anon) + peak(shmem) + peak(slab + pagetables)
cache_floor   = the file cache the workload re-reads (its hot set)
limit         = (working_set + cache_floor) x safety

safety of 1.3 is the working default for a service with a stable allocator. Use 1.5 for a JVM or any runtime with a generational collector, because collection is bursty and the peak between collections is not the average.

Applying it to the captured container: anon peaked around 34 MB across the sample window, shmem is 0, kernel structures are about 1.7 MB, and active_file sits near 107 MiB, which is the cache it keeps re-reading.

working_set = 34 + 0 + 2      =  36 MB
cache_floor = 107             = 107 MB
limit       = (36 + 107) x 1.3 = 186 MB  ->  round up to 256 MiB

The 512 MiB limit that was configured is twice what the evidence supports. On a host running twenty such containers that is 5 GB of RAM budgeted against nothing.

Budgeting the host

Per-container limits are only half of it. The host budget is the constraint that stops one container’s limit from being cashed at the same moment as everyone else’s:

sum(limits)  <=  RAM - host_reserve
host_reserve  =  1 GiB + 1% of RAM     (dockerd, containerd, systemd,
                                        journald, kernel slab, SSH)

TOTAL=$(docker ps -q | xargs -r docker inspect --format '{{.HostConfig.Memory}}' | awk '{s+=$1} END {print s+0}')
RAM=$(awk '/MemTotal/ {print $2 * 1024}' /proc/meminfo)
awk -v t="$TOTAL" -v r="$RAM" 'BEGIN {
  printf "limits:  %.1f GiB\n", t/1073741824
  printf "host RAM: %.1f GiB\n", r/1073741824
  printf "reserve:  %.1f GiB\n", (1073741824 + r*0.01)/1073741824
  printf "committed: %.0f%%\n", 100*t/(r - 1073741824 - r*0.01)
}'

limits:  22.5 GiB
host RAM: 46.0 GiB
reserve:  1.5 GiB
committed: 51%

A container with no --memory reports 0, which the sum silently treats as zero. That is the dangerous case: an unlimited container is not 0 GB of commitment, it is all of it.

docker ps -q | xargs -r docker inspect \
  --format '{{.HostConfig.Memory}} {{.Name}}' | awk '$1 == 0 {print "UNLIMITED:", $2}'

UNLIMITED: /legacy-batch

Overcommit: when it is defensible

Committing more than 100% is defensible only when you can show the peaks do not coincide. A batch job that runs at 02:00 and an API that peaks at 14:00 can share a memory budget; two API replicas behind the same load balancer cannot, because they peak together by definition.

If you overcommit, the mitigation is --memory-reservation, which sets memory.low. Under host pressure the kernel reclaims from cgroups above their memory.low first, so the reservation decides who gets squeezed rather than leaving it to chance.

docker run -d --name api \
  --memory 512m --memory-reservation 256m \
  --memory-swap 512m \
  registry.example.com/api:1.4.0

Setting --memory-swap equal to --memory disables swap for the container. Do that for latency-sensitive services: a service that swaps is a service whose p99 has already left the building, and you would rather have a clean OOM you can see in memory.events than a slow service nobody can explain.

Proving the limit was right

A sizing decision is a hypothesis. These two counters are the test:

CID=web
PID=$(docker inspect --format '{{.State.Pid}}' "$CID")
CG=$(awk -F: '{print $3}' "/proc/$PID/cgroup" | head -1)
cat "/sys/fs/cgroup$CG/memory.events"
cat "/sys/fs/cgroup$CG/memory.pressure"

low 0
high 0
max 0
oom 0
oom_kill 0
oom_group_kill 0
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

Read it as a verdict:

• oom_kill 0 and max 0 — the limit was never reached. If memory.peak is also far below memory.max, the limit is loose and you can reclaim the difference.
• max climbing with oom_kill 0 — the container is repeatedly hitting the ceiling and the kernel is reclaiming its way out. It is surviving on cache eviction. Raise the limit before it stops working.
• oom_kill above 0 — the limit is wrong, full stop. Do not restart and hope.
• memory.pressure full avg300 above zero — everything in the cgroup stalled waiting for memory. That is a latency problem the application will report as slow database calls.

Sanity check