Capacity planning for Docker hosts · Docker & Containers

Capacity planning has a bad reputation because most of it is a peak multiplied by 1.5. That is not planning; it is a guess with a decimal point in it.

Real capacity planning produces a date. “This host runs out of disk in approximately nine months, driven by volume growth of 1.4 GB per week” is an answer you can act on. “We should probably get a bigger box” is not.

Measure with the right tool

Each resource has one tool that answers the question and several that look like they do.

Read-only / Safethe measurement set— What to run, and what each one is actually telling you.

# Per-container CPU, memory, network and block I/O, one sample.
# CPU % here is a fraction of ONE core, so 250% means two and a half cores.
docker stats --no-stream \
--format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}'

# Host memory. The number that matters is "available", not "free".
free -m

# Disk BY SUBTREE. Do not use du on /var/lib/docker - see below.
docker system df -v

# Filesystem and inodes. Both fill, independently.
df -h /var/lib/docker
df -i /var/lib/docker

# Real network throughput over 10 seconds, per interface.
# ss shows sockets, not bytes; it cannot answer a throughput question.
sar -n DEV 10 1 2>/dev/null || cat /proc/net/dev

`du -sh /var/lib/docker` is slow and the number is wrong

Two independent problems.

It takes minutes and generates heavy I/O. overlay2 stores every layer as a directory of files; a host with forty images has millions of inodes to walk, and du walks all of them while your production workload competes for the same disk.

The total is not what you want. overlay2 hard-links identical files between layers to save space. du counts a hard-linked file once per directory entry it encounters unless it recognises the link, and its recognition depends on traversal order, so the number is neither the space used nor the space you could reclaim.

docker system df -v asks the daemon, which knows which layers are shared. Its SHARED SIZE and UNIQUE SIZE columns are the two numbers a capacity plan needs: SIZE tells you how big an image is, and only UNIQUE SIZE tells you how much you get back by deleting it.

Memory: the headroom that is not slack

This is where capacity plans are wrong most often, and it is one idea.

Linux uses all free memory as page cache, holding recently read file contents so the next read does not touch the disk. Page cache is reclaimable — the kernel evicts it instantly under pressure — which makes it look like memory you can allocate away. It is not free in the sense that matters: evicting it turns a memory read into a disk read, and the two differ by roughly four orders of magnitude.

A host with no page cache does not report a problem. It reports normal memory usage and mysteriously bad I/O latency.

Read-only / Safehow much cache does this host actually have

$ free -m

               total        used        free      shared  buff/cache   available
Mem:           32109       18442        1204         312       12463       13051
Swap:              0           0           0

Illustrative output

Read available, not free. free is memory doing nothing at all, which on a healthy Linux host is always small and is not a problem. available is the kernel’s own estimate of what a new allocation could obtain, including the cache it would be willing to evict — 13,051 MB here.

The worked memory plan

A host with 32 GB, running twelve containers.

Component	Amount	How it was obtained
Total RAM	32,109 MB	`free -m`
Kernel and slab	1,500 MB	`total` minus `used` minus `buff/cache` minus `free`
Host services (sshd, journald, dockerd, exporters)	1,200 MB	`systemd-cgtop` on the non-container slices
Sum of container working sets at peak	14,800 MB	7 days of `container_memory_working_set_bytes`, peak
Page cache reservation	6,400 MB	20% of total — justified below
Committed	23,900 MB
Uncommitted	8,209 MB	25.6% of total

The 20% page-cache reservation is a floor, not a rule of nature. Justify it per workload:

A service whose data fits in RAM — a 4 GB Postgres dataset on a 32 GB host — wants enough cache to hold the whole working set, so reserve for the dataset, not a percentage.
A service that streams data it never re-reads — a video transcoder, a log shipper — gains nothing from cache and can run with far less.
A build host genuinely wants a lot: BuildKit reads and writes layer files constantly, and a build host with no cache spends its time in I/O wait.

Twenty percent is the number to use when you have not measured. Measure by watching major page faults: node_vmstat_pgmajfault climbing means the kernel is going to disk for pages it used to hold, which is the direct symptom of insufficient cache.

The sum of memory limits is allowed to exceed RAM, and nothing warns you

Docker does not check. You can start twelve containers with --memory=4g each on a 32 GB host, and all twelve start successfully.

Nothing is wrong until they all use what they were promised. Then the kernel OOM killer selects a victim by heuristic — not the container that grew, not the least important one, but the one with the worst oom_score — and kills it. On a host running a database and eleven stateless workers, the database is frequently the biggest process and frequently the one chosen.

Plan against two numbers and know which you are looking at:

Sum of limits. The worst case. If it exceeds physical RAM you have oversubscribed and are relying on the containers not being busy at once.
Sum of observed peaks. The likely case, and what the table above uses.

Oversubscribing memory is a defensible choice for workloads whose peaks genuinely do not correlate. It is indefensible when the containers are replicas of the same service, because their peaks correlate perfectly — a traffic spike hits all of them.

If you do oversubscribe, protect the important process: docker run --oom-kill-disable is not the answer (it can freeze the container instead of killing it). Set the database’s memory limit to a real value with headroom, and give the workers limits tight enough that they are the ones the kernel finds first.

CPU: quota is not the saturation signal

CPU is the resource where oversubscription is normal and safe, because CPU starvation is recoverable and memory starvation is not. A container waiting for CPU is slow; a container denied memory is dead.

So the plan is different in shape: the sum of --cpus values across containers routinely exceeds the core count, and that is fine. What you plan against is the run queue, not the sum of the quotas.

Read-only / Safethe CPU number that matters

$ cat /proc/pressure/cpu

some avg10=4.28 avg60=3.91 avg300=2.55 total=1842771991

Illustrative output

Pressure Stall Information reports the percentage of time some task was runnable but waiting for CPU. avg10=4.28 means 4.28% of the last ten seconds had something waiting. Below roughly 10% is comfortable; sustained above 20% is a host that needs cores, and it says so long before load average does — because load average counts uninterruptible sleep too and conflates I/O waits with CPU contention.

For the per-container view, the throttle ratio from the monitoring part is the equivalent: a container throttled more than a quarter of its scheduling periods needs more quota regardless of what the host has spare.

Disk: four things growing at four different rates

Read-only / Safethe baseline measurement

$ docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          22        9         24.8GB    11.2GB (45%)
Containers      14        12        1.9GB     0.3GB (15%)
Local Volumes   9         8         61.4GB    2.1GB (3%)
Build Cache     184       0         18.6GB    18.6GB

Illustrative output

Images

Image growth is driven by build rate multiplied by unique layer size, not by image size, because base layers are shared.

Take a CI host building six times a day:

Distinct base images on the host:     3
Base layers, total on disk:           2.4 GB   (shared by everything)
Unique application layer per build:   180 MB   (docker system df -v, UNIQUE SIZE)
Builds per day:                       6
Tags retained:                        30 days

Steady-state image store
  = base layers + (builds/day x retention x unique size)
  = 2.4 GB + (6 x 30 x 0.180 GB)
  = 2.4 GB + 32.4 GB
  = 34.8 GB

Two things fall out of that arithmetic that are not obvious from the numbers going in:

Retention dominates. Halving retention to 15 days saves 16.2 GB; halving the image size saves 16.2 GB too, and is far more work. Retention policy is the cheapest capacity lever you have.
Adding a fourth base image costs its full size once, not once per build. A host standardised on one base image and a host using four differ by a couple of gigabytes, not by a factor of four. Standardising base images is worth doing for security and reproducibility; do not sell it as a disk saving.

Build cache

Build cache is the fastest-growing thing on a build host and the only one with a supported cap. BuildKit’s garbage collection is on by default with a 20 GB target, and you should set it explicitly rather than inherit it:

Configuration change/etc/docker/daemon.json— Bound the build cache in the daemon rather than relying on a cron prune. GC runs continuously; a cron job runs after the disk is already full.

{
"builder": {
  "gc": {
    "enabled": true,
    "defaultKeepStorage": "20GB"
  }
}
}

Setting this is strictly better than a nightly docker builder prune, because garbage collection is continuous. A prune job scheduled for 02:00 does nothing about a build storm at 14:00 that fills the disk by 16:00.

If you do prune, bound it rather than emptying it:

Destructivebounded cache prune— Keeps 10 GB of cache. A bare `docker builder prune -a` empties it and the next build starts from nothing.

docker builder prune --keep-storage 10GB --filter 'until=168h' --force

Logs

This is where the arithmetic surprises people, so do it properly.

Request rate:                    500 req/s
Log lines per request:           1
Bytes per JSON log line:         400 B     (structured logging is verbose)

Per second:   500 x 400 B                = 200 KB/s
Per day:      200 KB/s x 86,400          = 17.3 GB/day
Per month:                               = 518 GB/month

Seventeen gigabytes a day, from one container, written to the root filesystem — not to the volume you sized for data. The default json-file driver does not rotate. This container fills a 200 GB disk in under twelve days and takes the daemon down with it.

With rotation configured:

max-size: 50m, max-file: 3  ->  150 MB per container, hard cap
12 containers               ->  1.8 GB total, permanently

The same workload, bounded, costs 1.8 GB forever instead of 518 GB a month. This single setting is the highest-leverage line in a Docker capacity plan.

`log-opts` in daemon.json applies at container creation only

Setting log-opts in /etc/docker/daemon.json and restarting the daemon does not rotate the logs of containers that already exist. Each container carries the logging configuration it was created with, forever.

So the host with the 400 GB log file is very often a host where somebody already fixed the daemon configuration, verified it on a test container, and closed the ticket — while the container that was actually the problem kept its original unbounded config because nobody recreated it.

Find them:

docker ps -a --format '{{.Names}}' | while read -r C; do
  OPTS=$(docker inspect -f '{{json .HostConfig.LogConfig.Config}}' "$C")
  case "$OPTS" in
    *max-size*) ;;
    *) echo "UNBOUNDED LOGS: $C" ;;
  esac
done

The fix is to recreate each container listed. There is no way to change a running container’s log configuration in place.

Volumes

Volumes are the only row you cannot manage with a retention policy, because the contents are the point. Measure the trend and extrapolate; there is no formula.

Read-only / Safevolume growth rate— Record this weekly to a file. Two samples give you a rate; twelve give you a trend you can trust.

STAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker system df --format '{{.Type}} {{.Size}} {{.Reclaimable}}' \
| sed "s|^|$STAMP |" >> /var/log/docker-capacity.log
tail -5 /var/log/docker-capacity.log

Putting it together: the date

The output of a capacity plan is a date. Here is the full calculation for the 200 GB host measured above.

Filesystem total                       200.0 GB
  OS, packages, Docker binaries          8.0 GB
  Images (steady state, 30d retention)   34.8 GB
  Build cache (capped by GC)             20.0 GB
  Container logs (capped by log-opts)     1.8 GB
  Volumes (today)                        61.4 GB
  ------------------------------------------------
  Currently committed                   126.0 GB

Act-by threshold at 85% of the filesystem
                                       170.0 GB
Headroom before the threshold           44.0 GB

Volume growth, measured over 8 weeks     1.4 GB/week
Weeks of headroom  = 44.0 / 1.4        = 31 weeks
                                       ~ 7 months

Seven months. That is the deliverable. It is a number you can put in a budget, and it is a number that changes when an input changes — so when someone proposes doubling log verbosity, you can price it in weeks rather than arguing about it.

The 85% threshold rather than 100% is deliberate: the last 15% is consumed during the emergency, by the image pull needed to deploy the fix, and by the fact that filesystems degrade as they approach full. Plan to act at 85% and you are ordering hardware; plan to act at 98% and you are running docker system prune -a on a production host at 03:00 and deleting the rollback image.

Why image sizes do not add up, and what that does to a plan

docker images reports nginx:1.27 as 190 MB and nginx:1.28 as 192 MB. Both are on the host. The disk does not hold 382 MB of nginx.

overlay2 stores each layer once, keyed by its content digest, under /var/lib/docker/overlay2/. Two images built from the same base share every base layer — the same directories on disk, referenced by both images’ lower chains. A container adds one more directory, its writable upper layer, and the daemon assembles the union at merged.

So:

Summing the SIZE column overestimates, sometimes by several times, because every image counts its shared base layers in full.
UNIQUE SIZE from docker system df -v is the reclaimable figure, and it is usually a small fraction of SIZE. An image reporting 800 MB may free 40 MB when deleted.
The last image using a base layer frees the whole base. This produces the confusing pattern where deleting nine images frees almost nothing and deleting the tenth frees 700 MB.

For a capacity plan this means the honest model is the one used above: count base layers once, then count per-build unique layers. Anything derived from summing image sizes will overstate your image store, sometimes by enough to buy hardware you did not need.

What happens when /var/lib/docker actually fills?

Not a graceful degradation. A cluster of simultaneous, confusing failures:

Containers cannot start. Creating the writable layer needs disk. New deploys fail with errors about the storage driver that do not mention disk space.
Running containers begin failing writes — including any that write to the container filesystem rather than a volume, and including the log driver.
docker logs may return nothing, because the log file cannot be appended to, so the tool you would reach for to diagnose the problem is the one that stops working.
The daemon may fail to write its own state, and in the worst case will not restart, because it cannot persist container metadata during startup.
docker system df itself may hang, because it walks the filesystem.

The recovery order that works, from most to least reversible:

docker builder prune --force — build cache is pure cache. Fastest win, zero risk, often tens of gigabytes.
truncate -s 0 on the largest container log file. Instant, and it loses only logs you have presumably shipped elsewhere. Do not delete the file: the daemon holds the file descriptor and deleting it frees nothing until the container stops.
docker image prune — dangling images only, still safe.
Only then consider docker image prune -a, and read the maintenance part on what it takes with it.

Never start at step 4. docker system prune -a on a full production host deletes the rollback image, and it is the command people reach for first because it frees the most.

The real answer is that you should never be here, because the seven-month number told you in March.

Measure for a full week minimum, covering a weekly peak. A Tuesday afternoon sample plans for Tuesday afternoons.
Take peaks, not averages, for memory, and averages plus pressure for CPU. Memory has no graceful degradation; CPU does.
Reserve page cache explicitly as a line item, so nobody later reads it as spare capacity and allocates it.
Cap what can be capped — build cache in daemon.json, logs in log-opts, images by retention policy. Three settings remove three growth curves.
Extrapolate only what remains, which is almost always volumes.
Publish a date and the inputs that produced it, so the plan can be recomputed when an input changes rather than redone.
Re-measure quarterly, and immediately after any change to logging verbosity, retention, or the number of services.

Measure with the right tool

Memory: the headroom that is not slack

The worked memory plan

CPU: quota is not the saturation signal

Disk: four things growing at four different rates

Images

Build cache

Logs

Volumes

Putting it together: the date

Knowledge check