Skip to main content
RunBook Academy

← All break/fix scenarios in Docker & Containers

advancedStorage~25 min

Break/Fix 16: "No space left on device" with 40% of the disk free

Reported symptoms

  • ●`docker pull` fails part-way through extraction: `write /var/lib/docker/overlay2/.../diff/...: no space left on device`.
  • ●New containers fail to start with `error creating overlay mount ... no space left on device`.
  • ●`df -h /var/lib/docker` shows 41% used with 320 GB free, so the alert never fired.
  • ●Containers already running keep serving traffic; only operations that create new files fail.

Evidence

  • · `df -i /var/lib/docker` — `IUse%` is 100%, `IFree` is 0.
  • · `docker system df` reports 62 GB of images on a 550 GB filesystem — nowhere near full by bytes.
  • · Counting files per layer shows several `diff` trees with 200,000+ entries each (`node_modules`, `site-packages`, vendored Go module caches).
  • · `tune2fs -l /dev/mapper/vg0-docker | grep -i inode` shows the filesystem was created with the ext4 default of one inode per 16 KiB.
Diagnosis and resolutionclick to reveal

Root cause

overlay2 unpacks every image layer into a real directory tree under /var/lib/docker/overlay2. Inode consumption therefore tracks the number of files in the images, not their size. ext4 fixes the size of the inode table at mkfs time and cannot grow it, so a host that pulls many-small-file images exhausts inodes with most of its blocks still free. The kernel returns ENOSPC for a create that has no free inode — the same errno it returns for a full disk — so Docker reports "no space left on device" while `df -h` looks healthy and everyone hunts for a disk-fill that is not there.

Remediation

Reclaim inodes first so the host can serve again: `docker image prune -a --filter until=168h` and `docker builder prune --filter until=168h`. That is a stopgap; the filesystem will exhaust again at the same file count. The durable fix is to rebuild /var/lib/docker with an inode budget that matches the workload — `mkfs.ext4 -i 8192` doubles the count, and XFS allocates inodes dynamically so the ceiling disappears. Migration means stopping the daemon, copying the tree with a hardlink-preserving copy, and remounting.

Verification

`df -i /var/lib/docker` shows IUse% back below the alert threshold with a working margin. The pull that failed completes. A new container starts and writes a file. `docker system df` and `df -i` are both recorded in the ticket so the next occurrence has a baseline.

Prevention

Alert on `df -i` alongside `df -h` — a host with free bytes and no free inodes is invisible to every disk alert most teams ship. Prune images and build cache on a schedule with an age filter. Reduce the file count at source with multi-stage builds and distroless or slim bases. Put /var/lib/docker on XFS on new hosts.

Reported symptoms

The deploy pipeline fails on one host in the fleet. The error names a full disk:

failed to register layer: write /var/lib/docker/overlay2/9f3c.../diff/usr/
lib/node_modules/.package-lock.json: no space left on device

Everyone opens the disk dashboard, sees 41% used, and concludes the error message is wrong or that Docker is broken.

Diagnosis

The error message is not wrong. It is reporting the wrong resource because the kernel only has one errno for both.

df -h /var/lib/docker
df -i /var/lib/docker
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vg0-docker  550G  221G  302G  43% /var/lib/docker

Filesystem              Inodes  IUsed IFree IUse% Mounted on
/dev/mapper/vg0-docker  36M     36M   0     100%  /var/lib/docker

IUse% is the whole finding. Blocks are the resource everyone watches; inodes are the resource that ran out.

Find which layers own the inodes. This walks the entire overlay2 tree and takes several minutes on a large host — it is read-only, but run it with nice on a host that is still serving:

cd /var/lib/docker/overlay2
for d in */diff; do
  printf '%s\t%s\n' "$(find "$d" | wc -l)" "${d%/diff}"
done | sort -rn | head -10

The top of that list is almost always the same shape: an application image built FROM node or FROM python that ships its dependency tree unpruned, rebuilt on every commit, with every historical tag still resident on the host.

Confirm the ceiling is structural rather than transient:

DEV=$(findmnt -no SOURCE --target /var/lib/docker)
sudo tune2fs -l "$DEV" | grep -iE 'inode count|free inodes|block count'

An ext4 filesystem cannot grow its inode table. Whatever count it was created with is the count it has for life.

Resolution path

  1. Reclaim inodes so the host can deploy again.
  2. docker image prune -a --filter until=168h removes images not used by a container in the last week.
  3. docker builder prune --filter until=168h removes the BuildKit cache, which on a build host is usually the larger half.
  4. Re-check df -i /var/lib/docker after each step; stop when you have working headroom.
  5. **Do not reach for docker system prune -a --volumes under pressure.**
  6. It removes named volumes that no running container references, which on a host with a stopped database container is the database.
  7. Record the file count that caused this.
  8. The prune bought time proportional to the images you deleted, not to the problem. Put the inode count and the top layers in the ticket.
  9. Size the durable fix.
  10. Target inode count = peak resident image file count x 3. Most build hosts need 4-8 KiB per inode, not the ext4 default of 16 KiB.
  11. Rebuild the filesystem, or move to XFS.
  12. systemctl stop docker docker.socket, copy /var/lib/docker to a staging path with cp -a or rsync -aHAX (hardlinks and xattrs matter), mkfs.xfs or mkfs.ext4 -i 8192 on the device, copy back, systemctl start docker.
  13. XFS allocates inodes dynamically, which removes the ceiling rather than raising it.
  14. Verify before handing the host back.
  15. Pull the image that failed. Start a container. Write a file inside it.

Verification

  1. **df -i /var/lib/docker shows IUse% below the alert threshold.**
  2. The failing pull completes. docker pull on the image named in the original error.
  3. A new container starts and writes. docker run --rm alpine sh -c "touch /tmp/probe && echo ok"
  4. **Both df -h and df -i are recorded in the ticket** so the next responder has a baseline.

Prevention

  • Alert on inode usage, not just block usage. node_exporter exposes node_filesystem_files_free; most teams graph node_filesystem_avail_bytes and nothing else, which is exactly why this failure gets three hours of misdirected investigation.
  • Prune on a schedule with an age filter, so the job is bounded and predictable rather than an emergency action.
  • Attack the file count at source. A multi-stage build that copies a built artefact into a distroless base ships hundreds of files where the single-stage image shipped hundreds of thousands.
  • Provision /var/lib/docker on XFS on new hosts and stop having this conversation.