Capacity planning — CPU, RAM, disk, network · Docker & Containers

Capacity planning for Docker is capacity planning for any Linux server, plus two concerns that have no equivalent on a traditional host:

Disk grows from things nobody is watching. Image layers accumulate with every deploy, build cache grows without bound, and container logs are unbounded by default. None of these are the application’s data and all of them are on the same filesystem as it.
Every limit is a cgroup ceiling, not a reservation. Setting memory: 1g on ten containers does not reserve 10 GB. It permits 10 GB. Whether the host can honour all ten at once is a question you have to answer yourself, because nothing checks it for you.

Miss the first and the host fills up. Miss the second and the OOM killer resolves it for you, badly.

What you are sizing

Resource	Bounded by	Failure mode	Where it shows first
Memory	Host RAM	Container OOM kill, or host-wide if unlimited	Exit code 137
CPU	Host cores	Throttling; latency grows, throughput does not	`cpu.stat` throttled counters
Disk space	Filesystem size	Writes fail with ENOSPC across every container at once	`df`, not `docker system df`
Disk IOPS	Device capability	Everything slows; looks like a CPU or network problem	`iostat` await
Network	Link and NIC	Latency and drops	`ss -s`, interface counters
PIDs	`pids_limit`	`fork()` fails; the container cannot even run a shell	`pids.events`

Memory: measure, do not sum

The sum of every container’s memory limit is an upper bound on what the host could be asked for. It is not what the host needs, because containers do not peak simultaneously. Sizing from that sum buys hardware you will never use; sizing from average usage buys an outage. The number you want is the peak working set, measured.

Why cgroup v2 makes containers look larger than they are

Under cgroup v2, a container’s memory accounting lives in /sys/fs/cgroup/system.slice/docker-<id>.scope/:

memory.current — total charged to the cgroup. This includes page cache: every file the container has read recently, plus the dirty pages it has written. A container that streams a 5 GB file through tar will show several gigabytes here while its actual heap is 40 MB.
memory.stat — the breakdown. anon is anonymous memory, the heap and stacks, the part that cannot be reclaimed. file is page cache, which the kernel will drop under pressure.
memory.max — the ceiling. Hitting it triggers reclaim first, and only an OOM kill if reclaim cannot free enough.
memory.events — counters, and this is the one worth alerting on. high and max count how often the limit was hit; oom and oom_kill count actual kills.

docker stats compensates for the page cache problem by subtracting inactive file pages from memory.current before reporting MEM USAGE, so its number is closer to a working set than the raw counter. It is still a snapshot: run it once and you have measured one instant.

Size from anon plus a working allowance, and alert on memory.events. A container whose memory.events shows a rising max count is being reclaimed at its ceiling — it has not been killed yet, but it is spending time in reclaim instead of doing work, and that is the warning before exit code 137.

Read-only / Safeworking set— Sample real usage over time instead of guessing from one snapshot.

# Sample every 30s for an hour; keep the peak per container
for i in $(seq 1 120); do
docker stats --no-stream --format '{{.Name}} {{.MemUsage}} {{.CPUPerc}}'
sleep 30
done | tee /tmp/usage.log

# The counter that says a limit is too tight, per container
for c in $(docker ps -q); do
name=$(docker inspect --format '{{.Name}}' "$c")
echo "== $name"
cat "/sys/fs/cgroup/system.slice/docker-$(docker inspect --format '{{.Id}}' "$c").scope/memory.events" 2>/dev/null
done

A container whose oom_kill counter is above zero has already been killed at least once, whatever its uptime says. That is a fact docker ps will not tell you, because the restart policy brought it straight back.

The memory rule that actually holds

Two inequalities, both of which have to be true:

Sum of measured peak working sets, plus host overhead, must fit in physical RAM with room to spare. Host overhead is about 1 GB for the kernel and page cache the host itself needs, plus 300–500 MB for dockerd and containerd. Target 70% of RAM for the workload total.

Sum of memory limits may exceed RAM, but only knowingly. Overcommitting limits is normal and fine — that is what makes them ceilings rather than reservations. What is not fine is overcommitting by an amount that lets the host reach OOM before any individual container does, because then the global OOM killer chooses the victim by footprint and takes your database instead of the service that misbehaved.

And do not run production with swap enabled on a Docker host unless you have a specific reason. Swap turns a fast, obvious OOM kill into a slow, mysterious latency collapse that is much harder to diagnose.

CPU: throttling is the signal, not utilisation

cpus: 2.0 sets cpu.max to a quota-and-period pair — 200000 microseconds of CPU per 100000 microsecond period. When a container exhausts its quota inside a period, every one of its threads is stopped until the period rolls over.

This matters because CPU throttling does not look like a CPU problem. Throughput stays flat and latency develops a long tail, because requests that happen to land near the end of a period wait for the next one. The p99 goes bad while the average and the CPU graph look fine.

Read-only / Safethrottling— The counter that distinguishes throttling from slowness.

$ cat /sys/fs/cgroup/system.slice/docker-$(docker inspect --format '{{.Id}}' api).scope/cpu.stat

usage_usec 48213904
user_usec 39120441
system_usec 9093463
nr_periods 182400
nr_throttled 21877
throttled_usec 194330012

Illustrative output

nr_throttled as a fraction of nr_periods is the number to watch. Twelve percent, as above, is a service that is being stopped regularly. Either its limit is too low or its concurrency is too high; either way, no amount of tuning inside the application will fix it.

Size CPU from measured utilisation with headroom for burst. Unlike memory, overcommitting CPU is safe in the sense that nothing gets killed — it just gets slower, and slower is a decision you can make deliberately.

Disk: the term everybody forgets

This is where Docker capacity planning genuinely differs, and where hosts actually die. Four things grow on the Docker filesystem, and only one of them is your data.

disk required =
    unique image layers          (NOT the sum of image sizes)
  + container writable layers    (small, unless someone writes outside a volume)
  + volumes                      (your actual data, growing at its own rate)
  + build cache                  (unbounded; only exists if you build here)
  + container logs               (unbounded by default; NOT in docker system df)
  + headroom                     (30%)

Sized from `docker stats`, full in ninety days

The plan was reasonable. Somebody measured the running containers: six services, 4.2 GB of memory, about 3 GB of image, a 20 GB database volume. They provisioned 100 GB and considered it generous — 4× the measured footprint.

Ninety days later the host is full, and the breakdown is not what anyone expected:

Read-only / Safesystem df— Where 100 GB went.

$ docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          214       6         41.6GB    38.1GB (91%)
Containers      9         6         112MB     18.4MB (16%)
Local Volumes   4         4         23.8GB    0B (0%)
Build Cache     1839      0         27.2GB    27.2GB (100%)

Illustrative output

Nobody sized for 214 images. Two deploys a day for three months, each leaving the previous image behind because a container was replaced but the old image was never pruned. Nobody sized for 27 GB of build cache, because the plan was made before anyone decided to build on the production host.

And the total in that output is 92 GB against a 100 GB disk, while df says the filesystem is full — because the container logs are not in docker system df at all. That gap is the last few gigabytes, and it is the one that made writes start failing.

The first diagnosis is usually “the database grew”. It did not; it is 23.8 GB and behaving exactly as modelled. 91% of the image space and 100% of the build cache are reclaimable, which means this host was never short of disk — it was short of a retention policy.

Modelling the layer term honestly

docker images shows each image’s full logical size, so adding them up double-counts every shared base layer. docker system df reports what is actually on disk. Model growth from deploy rate instead:

image growth per month =
    (deploys per month) × (size of the layers that change per deploy)

For a typical application image, the layers that change are the application layer and its dependency layer — often 50–200 MB — not the whole image. So 60 deploys a month at 150 MB is about 9 GB a month if nothing is ever pruned, and close to zero if a weekly docker image prune runs. That retention policy is worth more than the disk you would otherwise buy.

Modelling logs

log bytes per day = (lines/sec) × (bytes/line) × 86400 × (containers)

A service emitting 50 structured JSON lines a second at 400 bytes each is 1.7 GB per day, per container. With max-size: 10m and max-file: 3 it is capped at 30 MB and the rest is discarded — which is why the cap has to be paired with actually shipping the logs somewhere, or you have solved a disk problem by throwing away your evidence.

Read-only / Safegrowth baseline— Record a daily datapoint so the trend is measurable rather than remembered.

STAMP=$(date +%F)
USED=$(df --output=used -B1 /var/lib/docker | tail -1)
LOGS=$(du -sb /var/lib/docker/containers 2>/dev/null | cut -f1)
printf '%s\t%s\t%s\n' "$STAMP" "$USED" "$LOGS" >> /var/log/docker-capacity.tsv
tail -5 /var/log/docker-capacity.tsv

Two weeks of that file tells you the growth rate, and the growth rate tells you the date you run out. That is a far more useful number than a percentage full, because it is the one that says whether you have a month to act or a weekend.

Headroom, and what it is for

Run production hosts at 60–70% of capacity. The remaining 30–40% is not waste; each part of it is doing a job:

Burst. Real traffic is not flat. Plan for peak, not mean.
Failover. If this host takes another’s load, it needs room for it.
Operations. An upgrade pulls new images before removing old ones, so disk usage goes up during a deploy. A forensic capture needs somewhere to write. A database restore needs room for both copies.

That last point is the one people find out about during an incident: you cannot restore a 24 GB database onto a filesystem with 12 GB free, and the moment you need to is exactly the moment there is no time to make room.

Alert at 80%, not 95%. At 80% you have a change window. At 95% you have an incident, and at 100% you may not be able to log in to fix it.

What if you set no limits at all?

Everything works, until it does not, and the failure is maximally confusing.

Memory. Every container can allocate until the host is out. The global OOM killer then picks by oom_score, which is weighted by memory footprint — so it kills the largest process, which is very often the database rather than the service that leaked. The evidence is in dmesg/journalctl -k, and it names the victim, not the cause. Teams have replaced database hardware over this.

CPU. No container is throttled, so one busy service starves the rest. Because nothing is at a limit, every per-container metric looks healthy while every service is slow, which sends the investigation to the network.

PIDs. A runaway worker pool or a fork bomb exhausts the host PID space. You cannot SSH in, because that requires a fork. This is the failure that turns a fixable problem into a trip to the console, and pids_limit: 500 prevents it for free.

Disk. Nothing warns you and everything fails at once.

The asymmetry is the point: with limits, one container misbehaving is one container restarting, with a clear cause and a clear culprit. Without limits, one container misbehaving is a host-wide incident whose evidence points at the wrong service.

Measure peak working set per container over at least a week; do not use a single docker stats snapshot.
Set memory limits above measured peak with margin, and alert on memory.events rather than only on kills.
Set cpus from measured utilisation, then watch nr_throttled over nr_periods.
Set pids_limit on every service; a few hundred is generous.
Model disk as layers + writable + volumes + build cache + logs + 30% headroom.
Bound the log driver, and ship the logs somewhere before you cap them.
Put image and build-cache pruning on a schedule; a retention policy is cheaper than disk.
Record a daily usage datapoint so you can compute the date you run out, not just the percentage full.
Alert at 80% of disk and re-plan quarterly.