Filesystem maintenance for /var/lib/docker

/var/lib/docker is not an ordinary directory. On a busy host it holds hundreds of thousands of small files across thousands of overlay layer directories, it is written to constantly, and — on most installations — it is on the root filesystem, where filling it takes down sshd, the journal and your ability to fix it.

Four maintenance concerns follow from that: where it lives, inodes, discard, and how to move it when it outgrows its home.

Where it should live

$ findmnt -T /var/lib/docker -o TARGET,SOURCE,FSTYPE,SIZE,USED,AVAIL,USE%

TARGET          SOURCE     FSTYPE SIZE  USED AVAIL USE%
/               /dev/sda2  ext4   126G   57G   65G  47%

TARGET showing / means the data root is on the root filesystem. That is the default and it is the wrong layout for a production host, for one reason: when Docker fills the disk, everything else on the machine fails at the same time.

The layout worth building:

Separating volumes from the rest of the data root is the one people skip, and it is what makes a daemon-state rollback possible without a data rollback — the constraint from the upgrade rollback lesson.

Inodes: the exhaustion nobody watches

Every overlay layer is a directory tree, and image layers are made mostly of small files. A host with many images consumes inodes far faster than bytes.

$ df -i /var/lib/docker

Filesystem       Inodes   IUsed    IFree IUse% Mounted on
/dev/sda2       8388608 6710886  1677722   80% /

80% of inodes used with 47% of bytes used. That host will report “disk fine” on every byte-based check right up until the moment writes start failing with No space left on device — the same errno for both conditions, which is why the diagnosis goes wrong so often.

# Total inodes under each top-level Docker directory
sudo du --inodes -d 1 /var/lib/docker 2>/dev/null | sort -n

# Both dimensions in one alert-friendly line
df -h --output=target,pcent /var/lib/docker | tail -1
df -i --output=target,ipcent /var/lib/docker | tail -1

Discard on SSDs

overlay2 creates and deletes layer directories constantly. Without discard, the SSD firmware never learns those blocks are free, and write amplification and latency climb over months in a way that is easy to blame on the application.

# Most distributions ship this timer; confirm it is enabled
systemctl status fstrim.timer
sudo systemctl enable --now fstrim.timer

# Trim everything eligible now, verbosely
sudo fstrim -av

$ sudo fstrim -av

/var/lib/docker: 41.2 GiB (44242247680 bytes) trimmed on /dev/mapper/vg0-docker
/: 8.1 GiB (8697308774 bytes) trimmed on /dev/sda2

Use the weekly fstrim.timer rather than the discard mount option. Inline discard issues a TRIM on every delete, which on a workload that deletes as many small files as overlay2 does adds latency to exactly the operations you care about. Batch discard once a week does the same job off the hot path.

Moving the data root

The maintenance task with the largest downside if done wrong. The whole operation must happen with the daemon stopped.

#!/usr/bin/env bash
set -euo pipefail

NEW=/srv/docker

# 1. Stop the socket first, then the service, then containerd
sudo systemctl stop docker.socket docker.service
sudo systemctl stop containerd.service

# 2. Confirm nothing is still holding the tree open
sudo lsof +D /var/lib/docker 2>/dev/null | head

# 3. Copy, preserving EVERYTHING. -a alone is not enough.
sudo mkdir -p "$NEW"
sudo rsync -aHAX --numeric-ids --info=progress2 /var/lib/docker/ "$NEW"/

# 4. Point the daemon at the new location
#    /etc/docker/daemon.json:  { "data-root": "/srv/docker" }
sudo dockerd --validate --config-file /etc/docker/daemon.json

# 5. Start and verify BEFORE deleting anything
sudo systemctl start containerd.service docker.service
docker info --format 'data-root={{.DockerRootDir}}'
docker image ls
docker volume ls
docker ps -a

# 6. Only after verification, and not on the same day:
#    sudo rm -rf /var/lib/docker.old

Filesystem checks

fsck needs the filesystem unmounted, which for a dedicated Docker volume means stopping the daemon. That is a real constraint and it is a further argument for the separate-filesystem layout: you can check and repair the Docker volume without taking the whole host to single-user mode.

sudo systemctl stop docker.socket docker.service containerd.service
sudo umount /var/lib/docker

# ext4: force a check even if the filesystem is marked clean
sudo fsck.ext4 -f -y /dev/vg0/docker

# xfs: repair is a separate tool; xfs_check does not exist
sudo xfs_repair /dev/vg0/docker

sudo mount /var/lib/docker
sudo systemctl start containerd.service docker.service

For xfs, confirm the ftype=1 requirement is satisfied while you are there — overlay2 needs it and a filesystem formatted without it will have been failing in confusing ways:

sudo xfs_info /var/lib/docker | grep -o 'ftype=[01]'

Knowledge check