Storage drivers and image layout · Docker & Containers

The storage driver is the implementation behind every image pull, every container start, and every byte a container writes to its own filesystem. overlay2 is the default on Linux and has been for years. Knowing what it does well, what it does badly, and which of your problems it is not responsible for is the difference between debugging a performance issue and guessing at one.

What is still here, and what is gone

This table is more useful than it looks, because half the storage-driver advice on the internet recommends drivers that no longer exist in the engine.

Driver	Status in current Engine
`overlay2`	Supported. The classic default on Linux.
containerd snapshotters	Default for Engine 29.0 and later on fresh installs.
`fuse-overlayfs`	Supported, for rootless Docker on hosts where `overlay2` cannot run rootless.
`btrfs`	Supported on a Btrfs backing filesystem.
`zfs`	Supported on a ZFS backing filesystem.
`vfs`	Supported, for testing. No copy-on-write; every layer is a full copy.
`aufs`	Removed in v24.0. Deprecated in v19.03.
`overlay` (v1)	Removed in v24.0. Deprecated in v18.09.
`devicemapper`	Removed in v25.0. Deprecated in v18.09, disabled by default in v23.0.

Three practical consequences:

“Fall back to devicemapper” is no longer advice. It is not in the engine. A host that cannot run overlay2 runs vfs, which has no copy-on-write at all — every container gets a full physical copy of the entire image — or it does not run current Docker.

An upgrade across a removal boundary is a data event. Upgrading a host that was still on devicemapper to Engine 25.0 leaves the daemon unable to read any of its images. The images are still on disk; nothing can open them. Check docker info before an engine upgrade, not after.

Engine 29 changed the default. New installations use the containerd snapshotter rather than the classic overlay2 graph driver. If a lesson, runbook or script of yours reasons about /var/lib/docker/overlay2 paths, it is reasoning about the classic layout, which is still what you get on an upgraded 28.x host and on 29.x hosts that were upgraded rather than reinstalled.

The on-disk layout

Read-only / Safewhat am I running— Driver, backing filesystem, and the two flags that matter.

docker info --format 'driver={{.Driver}}'
docker info | sed -n '/Storage Driver/,/^ [A-Z]/p'

Read-only / Safedocker info storage section— The fields to read. Supports d_type must be true.

$ docker info

Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false

Illustrative output

Each layer is a directory under /var/lib/docker/overlay2/:

/var/lib/docker/overlay2/
├── l/                          # short symlinks to every diff dir
│   └── ZLNGY6...  -> ../<layer-id>/diff
└── <layer-id>/
    ├── diff/                   # this layer's actual files
    ├── link                    # this layer's short name, as used in l/
    ├── lower                   # colon-separated list of parent layers
    ├── merged/                 # the unified view, mounted only while running
    └── work/                   # OverlayFS scratch space for copy-up

The l/ directory of two-character-ish symlinks looks like an odd design choice until you know why it exists: the mount syscall takes its options as a single page of text, and an image with forty layers produces a lowerdir= argument of forty 64-character paths, which overflows it. The short symlinks keep the mount options inside the limit.

Read-only / Safea container's actual mount— Resolve a running container to its overlay mount and read the real lowerdir/upperdir arguments.

CONTAINER=web

docker inspect --format '{{.GraphDriver.Name}}' "$CONTAINER"
docker inspect --format '{{.GraphDriver.Data.UpperDir}}' "$CONTAINER"

# The same thing as the kernel sees it.
MERGED=$(docker inspect --format '{{.GraphDriver.Data.MergedDir}}' "$CONTAINER")
findmnt -no SOURCE,FSTYPE,OPTIONS "$MERGED" | tr ',' '\n' | head -20

GraphDriver.Data.UpperDir is the single most useful path in container debugging. It is the container’s writable layer, on the host, readable with ordinary tools while the container runs. Everything the container has created or modified since it started is in there, and du -sh on it answers “how much has this container written into its own filesystem” directly.

What copy-up costs

Reads of unmodified files are nearly free: OverlayFS looks up the path in upper, misses, and falls through to the lower layers. One extra directory lookup, then a normal read from the backing filesystem.

The first write to a file that lives in a lower layer triggers a copy-up: the entire file is copied from the lower layer to upper before the write is applied. Not the modified block — the whole file.

Operation	Cost
Read a file never modified	Near-native
First write to a 4 KB config file	Copy 4 KB, then write. Negligible.
First write to a 2 GB database file in the image	Copy 2 GB, then write 8 KB
Subsequent writes to the same file	Near-native; it is already in `upper`
Delete a file from a lower layer	Whiteout entry in `upper`. Cheap.
Create a new file	Straight into `upper`. Native speed.

The third row is the one that ends up in an incident report. A container appends one record to a large file that shipped inside the image, and the first append takes as long as copying the whole file.

Service impact possiblemeasure copy-up on your host— Build an image containing a large file, then time the first and second modification of it inside a container.

cat > /tmp/copyup.Dockerfile <<'EOF'
FROM alpine:3.20
RUN dd if=/dev/zero of=/big.dat bs=1M count=512 2>/dev/null
EOF

docker build -q -t copyup-demo -f /tmp/copyup.Dockerfile /tmp

docker run --rm copyup-demo sh -c '
echo "--- first write (triggers copy-up of 512 MB) ---"
time dd if=/dev/zero of=/big.dat bs=1M count=1 conv=notrunc 2>/dev/null
echo "--- second write (already in upper) ---"
time dd if=/dev/zero of=/big.dat bs=1M count=1 conv=notrunc 2>/dev/null
'

Read-only / Safethe difference— Same one-megabyte write, twice. Only the first one pays for the file.

$ docker run --rm copyup-demo sh -c '...'

--- first write (triggers copy-up of 512 MB) ---
real	0m 2.41s
user	0m 0.00s
sys	0m 0.71s
--- second write (already in upper) ---
real	0m 0.01s
user	0m 0.00s
sys	0m 0.00s

Illustrative output

Two hundred times slower for an identical operation. That ratio is the whole argument for volumes.

The database in the writable layer

Somebody runs PostgreSQL or MySQL without a volume — docker run -d postgres:16 with no -v — and it works. Data goes to the anonymous volume the image declares, or, for an image without a VOLUME declaration, straight into the writable layer.

In the second case every one of these is true at once:

Every write to a data file that came from the image pays copy-up first.
The upper layer competes for the same partition as images, build cache and logs, and when that partition fills the database hits ENOSPC mid-write, which for a database frequently means corruption rather than an error.
docker rm destroys the data. Including the docker rm in the deploy script that recreates the container to pick up a new log driver.
There is no snapshot, no backup hook, and no way to move the data to another host without docker cp from a container that may no longer start.

The first diagnosis is always “overlay2 is slow for databases”. overlay2 is not the problem; putting persistent state in an ephemeral layer is. Named volumes bypass OverlayFS entirely — a bind of a directory under /var/lib/docker/volumes straight into the container’s mount namespace, at full native filesystem speed with no copy-up anywhere.

Page cache sharing, and the metacopy option you do not have

Page cache sharing is overlay2’s best and least-known property. Multiple containers reading the same file from the same lower layer share one page cache entry, because they are reading the same inode on the backing filesystem. Fifty containers from the same 200 MB base image consume 200 MB of page cache between them, not 10 GB.

This is why container density on a shared base image is so much better than the equivalent count of virtual machines, and it is a property vfs destroys completely: vfs gives every container its own physical copy, so every container gets its own page cache entries and its own disk blocks.

Copy-up breaks the sharing for the file it copies, which is another reason write-heavy workloads in the writable layer scale badly: each container’s copy is a distinct inode with distinct cache pages.

metacopy is a kernel overlayfs option that makes a chown or chmod on a lower-layer file copy up only the metadata, leaving the data in place. For an image where the entrypoint runs a recursive chown over a large tree — a very common pattern in images that support arbitrary --user values — it turns gigabytes of copying into kilobytes.

docker info reports Using metacopy: false, and that is the default. Docker does not enable it because it has known interactions with layer distribution: a layer created on a metacopy host can be incomplete when read on one without it. The practical takeaway is not to turn it on. It is to stop doing recursive chown in entrypoints, and instead set ownership at build time in the Dockerfile, where it becomes part of the layer rather than a copy-up storm at every container start.

The backing filesystem requirement

overlay2 needs the underlying filesystem to distinguish file types in directory entries — d_type support. On ext4 this is always present. On XFS it depends on how the filesystem was formatted, and this is the trap.

XFS created with ftype=0 — the default for mkfs.xfs on some older distributions, and the layout on plenty of long-lived hosts — does not provide d_type. overlay2 on such a filesystem does not refuse to start. It runs, and misbehaves: whiteouts do not work correctly, so deleted files can reappear, and layer extraction can produce a filesystem that does not match the image.

Read-only / Safecheck d_type— Two independent checks. Both must agree before you trust overlay2 on XFS.

docker info | grep -E 'Backing Filesystem|Supports d_type'

# If the backing filesystem is xfs, confirm at the filesystem level too.
DEV=$(findmnt -no SOURCE --target /var/lib/docker)
sudo xfs_info "$DEV" 2>/dev/null | grep -o 'ftype=[01]' \
|| echo 'not xfs, or xfs_info unavailable'

ftype=1 and Supports d_type: true is the only acceptable result. ftype=0 cannot be changed in place: XFS cannot be reformatted online and there is no xfs_admin flag for it. The fix is to back up, recreate the filesystem with mkfs.xfs -n ftype=1, and restore — which is why you check this when you build the host, not when you are debugging phantom files six months later.

Choosing, and changing

For essentially every host: leave it alone.

ext4 or XFS with ftype=1, kernel 4.0+: overlay2. No reason to change.
Btrfs or ZFS root filesystem: the matching driver, if you want the filesystem’s own snapshot tooling to see container layers. overlay2 on top of them also works and is simpler.
Rootless Docker where overlay2 will not run: fuse-overlayfs.
Fresh Engine 29 install: you already have the containerd snapshotter and should keep it.
Anything else: you are on vfs, which is a correctness fallback rather than a production choice, and the real fix is the host.

What happens if you change the storage driver on a host with running workloads?

Layer stores are per-driver and mutually unreadable. Each driver keeps its content in its own tree with its own metadata, and no driver can read another’s.

Between classic graph drivers — overlay2 to btrfs, say — the daemon starts on the new driver with an empty layer store. Every image reports as absent. Every container that was created against the old driver fails to start. The old data is still under /var/lib/docker/<olddriver>, consuming disk, doing nothing. You must re-pull every image and recreate every container.

To the containerd snapshotter the wording is gentler and the effect is the same day-to-day: existing images and containers from overlay2 remain on disk but become hidden. Nothing is deleted, and switching back reveals them again, but while the snapshotter is active they cannot be used.

The safe sequence, whichever direction:

Inventory first. docker image ls --digests and docker ps -a, saved to a file off the host. That list is the only record of what needs to exist afterwards.
Confirm every image you need is in a registry you can reach. Anything built locally and never pushed does not survive. docker save it now.
Schedule an outage window. This is not an opportunistic change.
Stop the daemon, edit daemon.json, start the daemon.
Re-pull, recreate, verify.
Reclaim the old tree only after the new one is proven, because until then it is your rollback.

Data-loss riskpre-change inventory— Run before any storage driver change. Without this list you cannot prove what you had.

OUT=/var/tmp/pre-driver-change-$(date +%F)
mkdir -p "$OUT"

docker image ls --digests --format '{{.Repository}}:{{.Tag}} {{.Digest}}' > "$OUT/images.txt"
docker ps -a --format '{{.Names}} {{.Image}} {{.Status}}'                  > "$OUT/containers.txt"
docker volume ls --format '{{.Name}}'                                      > "$OUT/volumes.txt"
docker info --format '{{.Driver}}'                                         > "$OUT/driver.txt"

echo "copy $OUT off this host before proceeding"

Verification that can fail

After any change, or as a periodic health check:

Read-only / Safestorage health assertions— Exits non-zero on the two conditions that silently corrupt or silently degrade.

FAIL=0

DRIVER=$(docker info --format '{{.Driver}}')
echo "driver: $DRIVER"
case "$DRIVER" in
vfs) echo 'FAIL: vfs has no copy-on-write; every container is a full image copy'; FAIL=1 ;;
esac

if docker info 2>/dev/null | grep -q 'Supports d_type: false'; then
echo 'FAIL: backing filesystem lacks d_type; overlay2 behaviour is undefined'
FAIL=1
fi

# Any warning the daemon emits about storage is worth failing on.
docker info 2>&1 | sed -n '/^WARNING/p'

exit "$FAIL"

And to check the thing that actually causes storage-driver performance complaints — workloads writing into the writable layer instead of a volume:

Read-only / Safewho is writing to their own layer— docker ps -s reports the writable layer size. Anything above a few megabytes is state in the wrong place.

$ docker ps -s --format 'table {{.Names}}\t{{.Size}}'

NAMES        SIZE
nginx        1.09kB (virtual 48.3MB)
api          32.8kB (virtual 214MB)
worker       4.1kB (virtual 214MB)
legacy-db    8.4GB (virtual 8.72GB)

Illustrative output

legacy-db is the finding. Eight gigabytes in a writable layer means every one of those bytes is on the Docker partition, unbacked-up, deleted with the container, and paying copy-up on any file that came from the image. That is not a storage driver problem, and switching drivers will not help it.

Knowledge check

Why this matters

Almost every “overlay2 is slow” report is a workload doing something overlay2 was never meant to carry: persistent state in the writable layer, a recursive chown across a large tree at every start, or a large file from the image being appended to. In each case the fix is in the image or the volume configuration, and changing the driver would cost an outage and improve nothing.

The two storage-driver decisions that genuinely matter are both made once, at host build time, and are both expensive to reverse:

Format the Docker filesystem correctly. ftype=1 on XFS is one mkfs flag that cannot be added later without recreating the filesystem, and getting it wrong produces a data-correctness fault that presents as haunted files rather than as an error.

Do not upgrade the engine across a driver removal boundary without checking what you are running first. An engine that has dropped your driver leaves the host with images on disk that nothing can open, and it does so during a maintenance window, at whatever hour you scheduled it for.