Storage drivers — overlay2 and what the others are for

The storage driver is the implementation behind docker image pull, docker run, and every file write inside a container. On modern Linux, overlay2 is the right choice; knowing why, and what the alternatives do, is the difference between debugging a production incident and guessing.

What the storage driver does

The storage driver:

1. Stores image layers in /var/lib/docker/overlay2/<layer-id>/.
2. Composes them into a container’s root filesystem at start time.
3. Tracks the container’s writable layer.
4. Cleans up when the container is removed.

For overlay2, each layer is a directory; the merged view is mounted into the container’s mount namespace.

docker info --format 'Driver: {{.Driver}}'
docker info --format 'DriverStatus: {{json .DriverStatus}}'

overlay2

flowchart LR
  L1["lowerdir: layer 1 diff"] --> M
  L2["lowerdir: layer 2 diff"] --> M
  L3["lowerdir: layer 3 diff"] --> M
  U["upperdir: container diff"] --> M
  W["workdir: copy-up scratch"] --> M
  M["merged: the container root"]

overlay2:

• Is the classic default on Linux (kernel 4.0 and later; RHEL and CentOS from 3.10.0-514).
• Supports page cache sharing — multiple containers reading the same file from the same image layer share one page cache entry.
• Handles up to 128 lower layers natively, which is why it consumes fewer inodes than the legacy overlay driver it replaced.
• Has copy-up semantics on first write to a lower-layer file.

What overlay2 actually writes to disk

Every layer is a directory under /var/lib/docker/overlay2/. The entries inside it are not arbitrary; each one has a job.

# Pick any layer directory that exists on this host
LAYER=$(ls -1 /var/lib/docker/overlay2 | grep -v '^l$' | head -1)

ls -la "/var/lib/docker/overlay2/$LAYER"
echo '--- short id ---'
cat "/var/lib/docker/overlay2/$LAYER/link"; echo
echo '--- layers below it ---'
cat "/var/lib/docker/overlay2/$LAYER/lower" 2>/dev/null || echo '(this is a bottom layer)'

$ ls -la /var/lib/docker/overlay2/$LAYER

total 24
drwx--x--- 4 root root 4096 Aug 12 09:14 .
drwx--x--- 9 root root 4096 Aug 12 09:14 ..
drwxr-xr-x 5 root root 4096 Aug 12 09:14 diff
-rw-r--r-- 1 root root   26 Aug 12 09:14 link
-rw-r--r-- 1 root root   28 Aug 12 09:14 lower
drwx------ 3 root root 4096 Aug 12 09:14 work

--- short id ---
GKTX7QW5PZLRD2M4NHVBYJ3ACE

--- layers below it ---
l/UPQK4ZLN6XTMY2HRBVWD53JCFA

You can read the live composition of a running container the same way, straight out of the kernel’s mount table:

CONTAINER=web

# The merged path the daemon mounted for this container
docker inspect --format '{{.GraphDriver.Data.MergedDir}}' "$CONTAINER"

# And the mount options the kernel recorded for it
grep -F "$(docker inspect --format '{{.GraphDriver.Data.UpperDir}}' "$CONTAINER")" /proc/self/mountinfo

If GraphDriver.Data comes back empty, the host is not using overlay2 — see the Engine 29 section below.

The alternatives

The bottom three rows matter mostly when you inherit a host. A daemon.json carried forward from a 2019 install that still says "storage-driver": "devicemapper" will not start Engine 25 or later — the driver is gone, not merely discouraged, and the failure arrives as a daemon that will not come up after an upgrade rather than a warning before it.

Engine 29 stopped saying “overlay2”

This is the change most likely to surprise you on a current host. Docker Engine 29.0 and later uses the containerd image store by default for fresh installations. Layers are managed by a containerd snapshotter rather than by the Engine’s own graph driver, and docker info says so:

$ docker info | grep -A1 'Storage Driver'

 Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1

overlayfs here is the containerd snapshotter name. It is the same kernel filesystem doing the same job; what changed is which userspace component owns the layer store. Existing hosts that upgrade to Engine 29 keep their classic graph driver and keep reporting overlay2 — the new default applies to fresh installations only.

Verify the backing filesystem supports overlay2

overlay2 needs the underlying filesystem to record file types in directory entries. On ext4 this is always true. On XFS it is a format-time decision, and an XFS filesystem made with ftype=0 cannot host overlay2 at all.

DOCKER_ROOT=/var/lib/docker

# What filesystem is it on?
findmnt -no FSTYPE,SOURCE --target "$DOCKER_ROOT"

# If that said xfs, ftype must be 1
if [ "$(findmnt -no FSTYPE --target "$DOCKER_ROOT")" = xfs ]; then
xfs_info "$DOCKER_ROOT" | grep -o 'ftype=[01]'
fi

ftype=1 passes. ftype=0 means the only fix is to back up /var/lib/docker, reformat the filesystem with mkfs.xfs -n ftype=1, and restore — there is no online conversion. Find this at provisioning time, not at 03:00.

Inspecting the storage driver state

docker system df

$ docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          34        10        17.98GB   14.46GB (80%)
Containers      10        9         192.5kB   49.15kB (25%)
Local Volumes   47        8         2.905GB   1.684GB (57%)
Build Cache     87        0         7.446GB   5.124GB

Read that as four separate budgets. Containers at 192 kB is the sum of every container’s writable layer — it is tiny here because these containers write their state to volumes, which is the design you want. A host where Containers is measured in gigabytes is a host where applications are writing into the upper layer, and that data will vanish with the next docker rm.

Switching storage drivers

You can set the storage driver in daemon.json:

{
  "storage-driver": "overlay2"
}

Changing it is not a tuning knob. When you change the storage driver, every existing image and container becomes inaccessible, because their layers are stored in a format the new driver cannot read.

# 1. Record what exists now, so you can prove what came back
docker image ls --format '{{.Repository}}:{{.Tag}}' | sort > /root/images-before.txt
docker ps -a --format '{{.Names}}' | sort > /root/containers-before.txt

# 2. Change the setting
#    edit /etc/docker/daemon.json, then:
systemctl restart docker

# 3. Verify: this should now report the new driver
docker info --format '{{.Driver}}'

# 4. Verify the cost: this list should be EMPTY of anything you needed
comm -23 /root/images-before.txt <(docker image ls --format '{{.Repository}}:{{.Tag}}' | sort)

Step 4 is the verification that can fail. “The daemon restarted cleanly” tells you nothing — it restarts cleanly with every image hidden.

Knowledge check