The storage driver on disk — overlay2, and what replaced it · Docker & Containers

docker-overlayfs covers the kernel mechanism: lower, upper, work, merged, and copy-up. This lesson covers the layer above it — how dockerd decides which directories to pass to that mount, where it keeps them, and what happens to all of it when the storage driver changes underneath you.

The mount, from the daemon’s side

When a container starts, the daemon assembles one mount -t overlay call. Everything under /var/lib/docker/overlay2/ exists to make that call possible.

Read-only / Safethe four directories— On a daemon using the classic overlay2 graph driver. Note that UpperDir ends in /diff, not /upper.

$ docker inspect web --format '{{json .GraphDriver}}' | python3 -m json.tool

{
  "Data": {
      "LowerDir": "/var/lib/docker/overlay2/9c1f...e4a-init/diff:/var/lib/docker/overlay2/7b30...c19/diff:/var/lib/docker/overlay2/2ad8...f70/diff",
      "MergedDir": "/var/lib/docker/overlay2/9c1f...e4a/merged",
      "UpperDir": "/var/lib/docker/overlay2/9c1f...e4a/diff",
      "WorkDir": "/var/lib/docker/overlay2/9c1f...e4a/work"
  },
  "Name": "overlay2"
}

Illustrative output

Three details in that output surprise people:

LowerDir is ordered, highest first. The leftmost path is the layer closest to the container’s root; the rightmost is the base image layer. Reading it right to left reconstructs the Dockerfile.

UpperDir ends in diff. The container’s writable layer lives in a directory called diff — the same name image layers use for their content. There is no directory named upper in overlay2; that was the older overlay driver. Scripts written against the old layout silently find nothing.

The first LowerDir entry ends in -init/diff. That is not an image layer.

The `l/` symlink farm

/var/lib/docker/overlay2/l/ is full of short symlinks with names like 6Y5IM2XC7TSNIJZZFLJCS6I4I4, each pointing at ../<64-hex-cache-id>/diff.

Upstream states the reason directly: the shortened identifiers are “used to avoid hitting the page size limitation on arguments to the mount command”.

Each layer directory also contains a link file holding its own short name, and — for every layer above the base — a lower file listing its parents, in the same short form.

Digests are not directory names

docker image inspect reports layer diff IDs as sha256: digests. None of those digests appear as a directory name under overlay2/.

The directories are named with cache IDs: random identifiers the daemon allocates when it unpacks a layer. The mapping from content-addressed digest to cache ID lives in the daemon’s layer database under /var/lib/docker/image/overlay2/layerdb/.

This is a deliberate separation. The digest identifies content and must be stable across every host in the world; the cache ID identifies one local unpacked copy and is meaningless anywhere else.

The practical consequence: you cannot go from a digest to a directory by string matching, and a directory copied from another host means nothing without that host’s layer database. It is the mechanical reason a /var/lib/docker tar is not portable — docker-what-is-actually-stateful covers the operational side.

The driver changed

Docker 29 moved the default to the containerd image store. The symptom on a modern host:

Read-only / Safecaptured on Docker 29.7.1— The driver is now a containerd snapshotter named overlayfs, and GraphDriver on a container inspect is null.

$ docker info --format 'driver={{.Driver}}'
docker info | grep -A1 'Storage Driver'
docker inspect web --format '{{json .GraphDriver}}'

driver=overlayfs
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
null

Read that last line carefully. .GraphDriver is null — not empty, not a different shape. Every script, dashboard and monitoring check that does docker inspect --format '{{.GraphDriver.Data.UpperDir}}' fails on this host, and it fails with a Go template error rather than with anything that explains why.

This breaks tooling silently in the places you check least

The things that read GraphDriver are exactly the things nobody looks at until an incident: a disk-usage script, a forensic capture helper, a layer-inspection one-liner in a runbook.

They are written once, work for years, and then a routine engine upgrade turns them into an error message inside a larger script that may or may not have set -e.

Before an engine major upgrade, grep your automation:

grep -rn 'GraphDriver' /usr/local/bin /etc/cron.d /opt 2>/dev/null

and check the storage driver on a test host first:

docker info --format '{{.Driver}}'

The containerd store is the better design — layers are shared with containerd rather than duplicated, and image handling matches the rest of the ecosystem. It is still a change to a stable interface, and it deserves the same treatment as any other one.

Reading disk usage correctly

du on /var/lib/docker double-counts nothing, but it also does not tell you what is reclaimable. Ask the daemon:

Read-only / Safeshared vs unique— SHARED SIZE is layers this image has in common with others. Deleting the image reclaims only UNIQUE SIZE.

$ docker system df -v | head -12

Images space usage:

REPOSITORY   TAG             IMAGE ID       CREATED        SIZE     SHARED SIZE   UNIQUE SIZE   CONTAINERS
app          dev             0ef9d88b4b73   5 days ago     1.05GB   384.8MB       669.4MB       0
app          cimatch         bea532850da8   10 days ago    964MB    424.8MB       539.3MB       0
debian       bookworm-slim   abd67ffcfa54   8 days ago     116MB    0B            115.8MB       0
redis        7-alpine        e7723ff73d96   2 weeks ago    57.8MB   12.15MB       45.67MB       0

Illustrative output

SHARED SIZE is why removing a 630 MB image often frees far less than 630 MB: most of it is base-image layers other images are still using. Capacity planning that sums image sizes overestimates, sometimes by a large factor.

What if the filesystem reports free space but writes fail with ENOSPC?

Almost always inodes. Layered images create very large numbers of small files, and each one consumes an inode. An overlay2 directory with a few hundred layers can exhaust the inode table on a filesystem formatted with default settings long before the blocks run out.

df -h /var/lib/docker
df -i /var/lib/docker

If df -i shows IUse% at 100 while df -h shows free space, that is the diagnosis. docker system prune recovers inodes as well as bytes. The durable fix is formatting the Docker filesystem with a higher inode density, or giving /var/lib/docker its own volume sized for the file count rather than for the byte count.

The second, rarer cause is the mount argument limit described above, which surfaces as a mount failure at container start rather than as an application write error. docker history showing 100-plus layers in a single image points at it.

Sanity check

Why this matters

Two practical things come out of knowing this layer.

The first is that you can answer disk questions precisely. Which image is really costing you 600 MB and which is 90% shared base layers; whether the failure is bytes or inodes; whether the writable layer of one container is quietly growing because the application is logging into its own filesystem. All of that is one docker inspect and one df -i away once you know what you are looking at.

The second is more important and less obvious: this layout is an implementation detail that has already changed once. Docker 29 moved to the containerd snapshotter, GraphDriver became null, and every tool that read it broke. The lesson generalises — anything you build on the on-disk layout is coupled to a version, and the coupling only announces itself during an upgrade. Prefer the API surface, pin what you cannot, and check docker info --format '{{.Driver}}' before you trust a path.