Disk capacity — modelling how /var/lib/docker grows

Docker hosts do not run out of disk suddenly. They run out of disk on a schedule that was set months earlier, by four independent processes that grow at four different rates, none of which anyone measured.

This lesson replaces “add a monitor at 85%” with a model that tells you when the disk will fill and how large it should have been.

The four growth processes

Three of the four are bounded only if you bound them. The default for each is “grow forever”.

Reading the actual split

docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          34        10        17.98GB   14.46GB (80%)
Containers      10        9         192.5kB   49.15kB (25%)
Local Volumes   47        8         2.901GB   1.684GB (58%)
Build Cache     87        0         7.446GB   5.124GB

That host is carrying 28.3 GB of Docker state, of which 21.3 GB is reclaimable. Ten images are in use and twenty-four are not; eight volumes are attached and thirty-nine are orphaned. Neither number is a crisis today and both are a crisis in four months.

docker system df -v | sed -n '/Images space usage/,/Containers space usage/p' | head -8

Images space usage:

REPOSITORY          TAG              IMAGE ID       SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS
app-web             dev              0ef9d88b4b73   1.05GB    384.8MB       669.4MB       0
debian              bookworm-slim    abd67ffcfa54   116MB     0B            115.8MB       0
app-web             ci               bea532850da8   964MB     424.8MB       539.3MB       0

The model

Size the filesystem from the four rates plus a floor:

required = base_images
         + deploys_per_day x image_delta x image_retention_days
         + volume_growth_per_day x volume_retention_days
         + containers x log_max_size x log_max_files
         + build_cache_ceiling
         + headroom

headroom is not decoration. OverlayFS is a copy-on-write filesystem and every write to an existing file in a container copies the whole file up into the container layer first. At high utilisation the allocator fragments and copy-up slows down measurably. Budget 20% and treat 80% as the number your alert fires at, not 90%.

Worked example

A host runs 12 containers of one application, deployed 4 times a day. The image is 400 MB with a 60 MB delta per build. Postgres grows 1.5 GB a month. Logging is json-file with max-size=50m and max-file=3. Builds run on this host with a 10 GB cache ceiling.

base_images    = 400 MB
image churn    = 4 deploys/day x 60 MB x 14 days retention  = 3.4 GB
volume growth  = 1.5 GB/month x 12 months                   = 18.0 GB
logs           = 12 containers x 50 MB x 3 files            = 1.8 GB
build cache    = 10.0 GB ceiling                            = 10.0 GB
                                                            --------
subtotal                                                      33.6 GB
headroom (subtotal / 0.8)                                     42.0 GB

A 64 GB filesystem for /var/lib/docker gives 18 months of runway. A 40 GB one, which looks generous next to the 28 GB the host uses on day one, fills in under a year and does so during a deploy.

Bounding each term

Logs

Unbounded json-file logging is the most common way a Docker host fills, because it is the only term that scales with traffic rather than with your deploy cadence. Set the bound in the daemon so it applies to every container, including the ones nobody reviewed:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  }
}

docker ps -aq | while read -r c; do
  P=$(docker inspect --format '{{.LogPath}}' "$c")
  [ -n "$P" ] && [ -f "$P" ] && printf '%10s  %s\n' "$(du -h "$P" | cut -f1)" "$(docker inspect --format '{{.Name}}' "$c")"
done | sort -rh | head

      2.1G  /legacy-batch
     50M  /api
     50M  /web

The daemon-level log-opts apply to containers created after the setting, so a long-lived container started before the change keeps growing. Finding the 2.1 GB file is the point of the check.

Images

# Remove images not referenced by a container and older than 14 days.
docker image prune -a --force --filter 'until=336h'

until takes a duration or a timestamp. Prefer a duration in hours in automation: 336h is unambiguous, 14d is not accepted by every filter.

Build cache

# Trim the BuildKit cache to a 10 GB ceiling.
docker builder prune --force --reserved-space 10GB

--reserved-space is the current spelling. Older material and older CI scripts use --keep-storage, which still works but prints Flag --keep-storage has been deprecated. There are two siblings worth knowing: --max-used-space caps the cache from above, and --min-free-space prunes until the filesystem has that much free, which is the one you want on a shared host.

docker buildx du | tail -3

Private:	5.124GB
Reclaimable:	7.446GB
Total:		7.446GB

Volumes

Volume pruning is the one to be careful with. docker volume prune removes anonymous volumes not attached to a container, and an anonymous volume is exactly how a database that was started without -v keeps its data. Always prune volumes with --filter on a label you control, never bare.

# Only volumes this project created and labelled as ephemeral.
docker volume prune --force --filter 'label=lifecycle=ephemeral'

Inodes: the other way to run out

overlay2 stores every file of every layer as a real file. An image with a large node_modules or a Python site-packages tree can carry hundreds of thousands of small files, and each one is an inode.

df -h /var/lib/docker
df -i /var/lib/docker

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       126G   57G   65G  47% /
Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/sda2      8388608 1151104 7237504   14% /

No space left on device with 60 GB free is inode exhaustion. Budget roughly 1 million inodes per 100 GB for a Docker filesystem, and check df -i in the same alert as df -h — most monitoring templates ship with only the byte check.

1. Measure the four rates over a fortnight. docker system df daily into a file is enough; you need the slope, not precision.
2. Bound logs at the daemon, then find and recreate containers created before the setting.
3. Set an image retention filter and run it from a timer, not by hand.
4. Cap the build cache with --reserved-space on any host that builds.
5. Alert on 80% bytes and 80% inodes, both, on /var/lib/docker specifically rather than on /.
6. Re-run the model quarterly, because the deploy cadence is the term that changes most and nobody announces it.

Sanity check