Storage failures — out of disk, permission denied · Docker & Containers

Storage failures in Docker come in three flavours: out of disk, permission denied, and storage-driver-specific issues. Each has a distinct diagnostic.

What makes them expensive is that the error messages collapse several causes into one string. ENOSPC — “no space left on device” — is what the kernel returns when bytes run out, when inodes run out, when a filesystem quota is hit, and when a tmpfs mount reaches its size limit. Three of those four show plenty of free space in df -h.

Out of disk: four causes, one message

Read-only / Safewhich kind of full— Run all four. Only one of them will be interesting, and it is rarely the first.

# 1. Bytes, on the filesystem holding the data root
DATA_ROOT=$(docker info --format '{{.DockerRootDir}}')
df -h "$DATA_ROOT"

# 2. Inodes on the same filesystem
df -i "$DATA_ROOT"

# 3. Inside the container: its own mounts may be full while the host is not
docker exec web df -h
docker exec web df -i

# 4. Quotas, if the filesystem enforces them
sudo repquota -a 2>/dev/null | head
sudo xfs_quota -x -c 'report -h' 2>/dev/null | head

Illustrative output of the case people miss:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda2       196G  118G   69G  64% /var/lib/docker

Filesystem       Inodes  IUsed IFree IUse% Mounted on
/dev/vda2      12845056 12845056    0  100% /var/lib/docker

Sixty-four percent full and every write fails. overlay2 creates a directory tree per layer, and images built from many small files consume inodes far faster than bytes. Node.js images with a deep node_modules tree are the classic producer.

The `df -h` that says everything is fine

The incident goes like this. Containers start failing writes. Somebody runs df -h, sees 64%, and concludes the disk is not the problem. The next two hours go into the application, the database, the network.

df -i was one character away and would have ended it immediately.

Inode exhaustion is worth internalising because its recovery is different too. Pruning images frees inodes; deleting a few large log files does not, because a 40 GB log is one inode. If you are out of inodes, the fix is removing many small files, and the largest population of those on a Docker host is unused image layers.

The same reasoning applies in reverse: if you are out of bytes and the consumer is a single unrotated container log, pruning images will free almost nothing and you will have deleted your images for no benefit.

Identify the resource that is exhausted before you choose what to delete.

Attribute the consumption before pruning

Read-only / Safewho is using it— Category totals, then the individual offenders.

# Category totals, with per-object detail
docker system df
docker system df -v

# Container log files, largest first. json-file logs live beside the
# container's state in the data root.
DATA_ROOT=$(docker info --format '{{.DockerRootDir}}')
sudo du -h "$DATA_ROOT"/containers/*/*-json.log 2>/dev/null | sort -h | tail -10

# The biggest directories in the data root, one level down
sudo du -xh --max-depth=1 "$DATA_ROOT" | sort -h | tail

# Writable layer size per container (the -s flag is the point)
docker ps -as --format 'table {{.Names}}\t{{.Size}}'

docker system df reports TYPE, TOTAL, ACTIVE, SIZE and RECLAIMABLE. The RECLAIMABLE column is the number to read: it tells you what a prune would actually free, so you can decide whether the prune is worth it before running it. A host with 118 GB used and 4 GB reclaimable does not have a pruning problem — it has a sizing problem, and pruning will buy an hour.

docker ps -as is the one people forget. The SIZE column shows the container’s writable layer, which is data written inside the container rather than to a volume. A container with a 30 GB writable layer is an application logging to a file path instead of stdout, or writing temporary data that a volume should be absorbing. The writable layer dies with the container, so this is also silent data loss waiting to happen.

Where the bytes physically are

With the classic overlay2 graph driver, each layer lives under /var/lib/docker/overlay2/<id>/:

diff/ — the layer’s own contents. This is the directory whose size du reports.
link — “contains the name of the shortened identifier”, used to keep the mount option string short enough for the kernel.
lower — present on higher layers, “denotes its parent”.
merged/ — the unified view the container sees, populated only while the container is running.
work/ — OverlayFS’s internal scratch area for atomic operations.

The driver “natively supports up to 128 lower OverlayFS layers” and “consumes fewer inodes on the backing filesystem” than its predecessor — fewer, not few, which is exactly why inode exhaustion is still reachable.

Two properties follow that matter operationally. OverlayFS “supports page cache sharing”, so ten containers from the same image reading the same file share one page cache entry: memory accounting that assumes ten copies will overstate usage badly. And a write to a file that exists in a lower layer triggers copy-up — the whole file is copied into the upper layer before the write proceeds. Appending one line to a 2 GB file that came from the image costs 2 GB of disk and a long pause, which is a performance mystery with a storage cause.

On Engine 29.0 and later with the containerd image store as the default, image content and snapshots move to /var/lib/containerd instead, so a du of /var/lib/docker on such a host will understate the total dramatically. Check docker info for the storage driver before you trust a path.

Destructiveprune with a decision behind it— Look first, then remove the narrowest category that solves the problem.

# What would be freed, by category, before deciding
docker system df

# Narrowest first: build cache only. Almost always safe and often enough.
docker builder prune --filter 'until=168h'

# Then dangling images only (no -a): layers no tag points at
docker image prune

# Stopped containers, listed before removal
docker ps -a --filter status=exited --format 'table {{.Names}}\t{{.Status}}\t{{.Size}}'
docker container prune --filter 'until=24h'

# Only if the above was not enough, and only after checking what you
# still need to be able to roll back to:
# docker image prune -a --filter 'until=720h'

Escalating narrowest-first is the whole technique. Build cache is regenerable by definition. Dangling images are unreferenced by definition. -a is the only step that can take something you needed, so it is the last one and it gets a --filter until= so it cannot take last week’s release.

What happens when the host runs out of disk?

Containers fail writes; new containers may not start; the daemon’s API may become unresponsive. Some processes get ENOSPC and crash; others hang waiting for I/O.

The specifically nasty part is that the daemon’s own recovery paths need disk. docker commands that write to the data root — creating a container, pulling an image, even some inspect paths — can fail, so the tools you would use to fix it stop working. Log files stop being written, so the record of what happened while you were fixing it does not exist.

The order that works when you are already at 100%:

Free something outside the data root first, so the daemon can breathe: rotate host logs with journalctl --vacuum-size=200M, clear the package cache.
Truncate the largest container log in place rather than deleting it. Deleting a file an open process holds frees nothing until that process closes it, and the daemon holds those files open:
```
sudo truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log
```
Truncation frees the blocks immediately with the descriptor still open. This is the single most useful command in a disk-full Docker incident.
Only then prune, with the daemon able to function again.

The fix afterwards: set max-size and max-file so this cannot recur, and monitor RECLAIMABLE rather than raw usage.

The log rotation default is the one that bites

The json-file driver’s defaults are worth stating exactly, because they are the opposite of what most operators assume:

Option	Default	Consequence
`max-size`	`-1` (unlimited)	A container’s log grows without bound, forever
`max-file`	`1`	And is “only effective when `max-size` is also set”
`compress`	`false`	Rotated files are not compressed

So out of the box there is no rotation at all, and setting max-file alone does nothing. A chatty container on a long-lived host will fill any disk you give it. This is the most common single cause of a Docker host running out of bytes.

Configuration changerotation by default for every container— Daemon-wide default. Existing containers keep their old setting until recreated.

sudo tee /etc/docker/daemon.json >/dev/null <<'JSON'
{
"log-driver": "json-file",
"log-opts": {
  "max-size": "50m",
  "max-file": "5",
  "compress": "true"
}
}
JSON

sudo dockerd --validate --config-file=/etc/docker/daemon.json
sudo systemctl reload docker

# Verify the daemon took it
docker info --format 'driver={{.LoggingDriver}}'

# Verify a NEW container inherits it (an existing one will not)
docker run --rm -d --name logtest alpine:3.20 sh -c 'while :; do echo x; sleep 1; done'
docker inspect logtest --format '{{.HostConfig.LogConfig}}'
docker rm -f logtest

The verification step matters because the change is not retroactive. Containers created before the daemon change keep their original log configuration until they are re-created, so a host can be “fixed” and still be filling up from the one container nobody restarted.

Permission denied: five causes

docker exec web touch /data/test
# touch: cannot touch '/data/test': Permission denied

Cause	Discriminating check
UID mismatch	`docker exec C id` against `stat -c '%u:%g %a' /host/path`
Mounted read-only	`docker inspect C --format '{{json .Mounts}}'` — look for `"RW":false`
Read-only root filesystem	`docker inspect C --format '{{.HostConfig.ReadonlyRootfs}}'`
SELinux label	`ls -Z /host/path`, plus `ausearch -m AVC -ts recent`
AppArmor denial	`journalctl -k --since '5 min ago' \| grep -i apparmor`

Read-only / Safethe UID comparison— The container's view and the host's view, side by side.

CONTAINER=web
HOSTPATH=/srv/myapp-data

# Who is the process inside?
docker exec "$CONTAINER" id

# Who owns the directory on the host, numerically?
stat -c 'uid=%u gid=%g mode=%a %n' "$HOSTPATH"

# And what does the container see on the mount point?
docker exec "$CONTAINER" stat -c 'uid=%u gid=%g mode=%a %n' /data

# Every mount, with its read-write flag and propagation
docker inspect "$CONTAINER" --format '{{json .Mounts}}' | python3 -m json.tool

Compare numeric IDs, not names. A bind mount carries the host’s numeric ownership into the container, and the container’s /etc/passwd maps those numbers to entirely different names — or to nothing at all. ls -l inside the container showing an owner of 1000 where the host shows deploy is the same user; ls -l showing appuser in both places proves nothing, because they may be different numbers.

Fix ownership, not permissions

The reflex fix for permission denied is chmod 777. It works, and it is the wrong tool: it grants write access to every process on the host that can reach the path, including other containers, and it will be copied into the next runbook by someone who does not know why it was there.

The two correct fixes:

Match the UID. Run the container as the owner of the data: --user 1001:1001, or set USER in the Dockerfile and chown the host path to that numeric UID once at provisioning time.
Match the group. Where several services share a path, give the directory a group, add the container’s UID to it, and set the setgid bit so new files inherit it.

For SELinux the answer is a label, not a mode: :z for a path shared between containers, :Z for one used exclusively by this container. Note that :Z relabels the host path and will break other consumers of that path — do not use it on /home or a shared dataset.

Turning SELinux off, or running --privileged, is not a fix for a label problem. It removes the mechanism that was reporting the problem correctly.

Storage driver issues

docker info --format 'driver={{.Driver}} root={{.DockerRootDir}}'
docker info | grep -A5 'Storage Driver'
mount | grep -w overlay | head

Common issues:

Backing filesystem out of space. overlay2 uses the host’s filesystem for layers. If that filesystem is full, no new layers can be created — and this is reported as a pull or build failure rather than as a disk error.
d_type not supported. The driver “is supported on xfs backing filesystems, but only with d_type=true enabled”. An XFS filesystem formatted with ftype=0 cannot host overlay2 correctly. Check with xfs_info /var/lib/docker | grep ftype; ftype=0 requires a reformat, not a remount.
Inode exhaustion. Covered above; df -i.
The data root is on a filesystem you did not intend. A host provisioned with a small root partition and a large /data will fill the root the first time somebody pulls a big image. Move the data root deliberately with "data-root" in daemon.json rather than discovering the layout during an incident.

Diagnosing OverlayFS for one container

docker inspect web --format '{{json .GraphDriver.Data}}' | python3 -m json.tool

Illustrative output:

{
    "LowerDir": "/var/lib/docker/overlay2/9f3...c1/diff:/var/lib/docker/overlay2/4a2...7e/diff",
    "MergedDir": "/var/lib/docker/overlay2/b71...aa/merged",
    "UpperDir": "/var/lib/docker/overlay2/b71...aa/diff",
    "WorkDir": "/var/lib/docker/overlay2/b71...aa/work"
}

LowerDir is a colon-separated stack of image layers, ordered topmost first. UpperDir is where every write the container has made lives — du -sh on it is the true size of the writable layer. MergedDir is populated only while the container runs; on a stopped container it is an empty directory, which is normal and not corruption.

If a path listed in LowerDir does not exist on disk, the image’s layers are genuinely damaged. Re-pull the image; if the pull is a no-op because the daemon believes it already has the layers, remove the image first with docker image rm and pull again.

Verification

Read-only / Safeverify the storage fix— Proves a write succeeds and that headroom exists on both axes.

#!/usr/bin/env bash
set -euo pipefail
CONTAINER=web
DATA_ROOT=$(docker info --format '{{.DockerRootDir}}')

# Bytes and inodes both under 85%
for flag in -h -i; do
pct=$(df $flag --output=pcent "$DATA_ROOT" | tail -1 | tr -dc '0-9')
echo "df $flag: $pct percent"
[ "$pct" -lt 85 ] || { echo "FAIL: df $flag at $pct percent" >&2; exit 1; }
done

# The container can actually write where it needs to
docker exec "$CONTAINER" sh -c 'touch /data/.writetest && rm /data/.writetest' \
|| { echo 'FAIL: container cannot write to /data' >&2; exit 1; }

# Log rotation is in force for this container
maxsize=$(docker inspect "$CONTAINER" \
--format '{{index .HostConfig.LogConfig.Config "max-size"}}')
[ -n "$maxsize" ] && [ "$maxsize" != '<no value>' ] \
|| { echo 'FAIL: no max-size on this container, logs are unbounded' >&2; exit 1; }

echo OK

The write test is the part that can fail for the right reason. Checking df alone verifies the host; touching a file inside the container verifies the whole path including the mount flags, the UID and the LSM label.