OverlayFS — the filesystem underneath every container · Docker & Containers

A container’s root filesystem looks like a normal Linux tree. It isn’t. It is an OverlayFS composition of multiple read-only image layers plus one read-write upper layer that captures every change.

Understanding OverlayFS is the difference between “my container is 1.2 GB” being an accurate size and a misleading one. And between “why is apt-get install slow the second time” being a mystery and a known consequence of copy-up.

The four directories

flowchart TB
  L1["Lower (image layer 1)"] --> M[Merged]
  L2["Lower (image layer 2)"] --> M
  L3["Lower (image layer 3 - Ubuntu)"] --> M
  W[Work] --> M
  U["Upper (container read-write)"] --> M
  W -.-> U

OverlayFS combines three (or more) directories into one visible filesystem:

lower — read-only image layers, stacked. The lowest is the bottom of the image; the highest is closest to the container’s root.
upper — read-write. Every change the container makes lands here. This is what docker diff reports.
work — internal scratch directory OverlayFS uses during copy-up. Must be on the same filesystem as upper.
merged — the resulting view. What the container sees.

The visible filesystem is the merged view. From inside the container, the layered nature is invisible. From the host, it is visible via /var/lib/docker/overlay2/<id>/.

Copy-up semantics

When the container writes to a file that originated from a lower layer, OverlayFS does not modify the lower layer (read-only). Instead, it performs a copy-up:

The file is copied from the lower layer to the upper layer.
The change is made to the copy in the upper layer.
Subsequent reads see the upper-layer copy.

Copy-up happens lazily on the first write, and only for the file being modified (not the whole layer). The cost is the read from the lower layer plus the write to the upper.

The part that surprises people is the unit. Copy-up is per file, not per block. Modifying one byte in the middle of a file copies the entire file, start to finish, before your write(2) returns. There is no partial copy-up and no lazy block fetch: OverlayFS has to materialise the whole object in the upper layer because that is the only place it can hold a modified version of it.

For directories: a mkdir triggers a copy-up of the directory metadata. The directory contents are not copied until each file is modified. This is why a fresh container starts fast and gets slower as more files are modified.

For permissions: copy-up uses the original file’s mode. If the file was 0644 in the image, the upper copy is also 0644, regardless of umask. If a file in the image is a device or setuid binary, the copy is too. This is a security concern for setuid binaries; mitigate with explicit chmod in your Dockerfile.

Mounting OverlayFS manually

This is the single most useful debugging tool for understanding containers. You can do everything OverlayFS does without Docker.

Destructivemanual overlay mount— Mount an overlayfs with three lower layers, an upper, a work, and a merged directory. Read-only lower layers, read-write upper.

mkdir -p /tmp/lower1 /tmp/lower2 /tmp/upper /tmp/work /tmp/merged
echo 'from lower1' > /tmp/lower1/file.txt
echo 'from lower2' > /tmp/lower2/file.txt
mount -t overlay overlay -o lowerdir=/tmp/lower2:/tmp/lower1,upperdir=/tmp/upper,workdir=/tmp/work /tmp/merged
cat /tmp/merged/file.txt
echo 'new content' > /tmp/merged/file.txt
cat /tmp/upper/file.txt
umount /tmp/merged

Read-only / Safeobserve copy-up— After a write, the upper layer contains the modified file; the lower layer is untouched.

$ mkdir -p /tmp/lower1 /tmp/lower2 /tmp/upper /tmp/work /tmp/merged
echo 'from lower1' > /tmp/lower1/file.txt
echo 'from lower2' > /tmp/lower2/file.txt
mount -t overlay overlay -o lowerdir=/tmp/lower2:/tmp/lower1,upperdir=/tmp/upper,workdir=/tmp/work /tmp/merged
echo '--- merged view ---'
cat /tmp/merged/file.txt
echo '--- writing through merged ---'
echo 'overwritten' > /tmp/merged/file.txt
echo '--- upper dir after write ---'
ls /tmp/upper/
cat /tmp/upper/file.txt
echo '--- lower1 unchanged ---'
cat /tmp/lower1/file.txt
umount /tmp/merged

--- merged view ---
from lower2
--- writing through merged ---
--- upper dir after write ---
file.txt
overwritten
--- lower1 unchanged ---
from lower1

What this means for Docker images

A Docker image is a list of layers. Each FROM, RUN, COPY, or ADD instruction produces a new layer. When a container starts:

All image layers become lower directories.
A new empty upper directory is created.
A work directory is created.
The merged view is mounted at the container’s root.

When the container deletes a file that exists in a lower layer, the deletion is recorded as a “whiteout” file in the upper layer. The file still exists in the lower; it’s just invisible in the merged view.

One byte written to a 4 GB file costs 4 GB and a 40-second stall

The image ships a large file — a preloaded SQLite database, an ML model, a seeded search index, a geo-IP dataset. The application opens it read-write on startup and updates a header field: a version number, a last-opened timestamp, a dirty flag. Eight bytes.

The container hangs on startup for the better part of a minute, the health check fails its first two probes, and the host’s disk-usage alert fires. Restarting the container clears it — the second start is instant — so it gets written off as a slow cold start. Then the next deploy pulls a fresh container and it happens again, on every replica, at once.

What is actually happening. That file came from an image layer, so it lives in a read-only lower directory. The first write triggers copy-up: OverlayFS reads all 4 GB from the lower layer and writes all 4 GB into the upper layer before the eight-byte write completes. The stall is the copy. The disk growth is the copy. Multiply by replica count for the disk maths, because every container gets its own upper layer and therefore its own 4 GB.

Why the first diagnosis is wrong. The disk alert says the writable layer grew by 4 GB, and the reflex is to look for runaway logs — so people go through the logging driver, rotation settings, and the application’s own log level, and find nothing wrong with any of them. Nothing logged anything. A file was copied.

It is not only writes. Anything that modifies a file’s inode triggers the same full copy: chmod, chown, touch, truncate, utimes. An entrypoint that runs chown -R app:app /data over a directory baked into the image copies the entire tree, byte for byte, having changed nothing but ownership. The kernel does support a metacopy mode that copies metadata alone, but it is off by default and Docker does not enable it — check /sys/module/overlay/parameters/metacopy on your host and expect N.

How to tell. docker diff marks a copied-up file C. If a large path from the image shows up as C shortly after start, that is the copy, and the size of the upper directory will match it.

Read-only / Safewhat has been copied up, and what did it cost?— C means changed - the file was copied from a lower layer into the upper one. A large image path marked C is a copy-up you paid for in both time and disk.

CONTAINER=web
docker diff "$CONTAINER" | head -20
echo '--- size of the writable layer ---'
docker ps --size --filter "name=$CONTAINER" --format '{{.Names}}\t{{.Size}}'

C /usr
C /usr/local
A /usr/local/etc
A /usr/local/etc/redis
A /usr/local/etc/redis/redis.conf
web	4.02GB (virtual 4.31GB)

Illustrative output

Read the prefixes: A is a file that only ever existed in the upper layer, which costs exactly what it contains. C is a copy-up, which costs the size of the original. D is a whiteout, which costs nothing but does not reclaim anything either. The docker ps --size figure is the writable layer alone — the number before “virtual” — and it is the one that should be near zero on a well-behaved container.

The three fixes, in the order you should reach for them:

Put mutable data in a volume. A volume is a bind of a host directory into the container; it is not part of the overlay, so writes go straight to the filesystem with no copy-up at any size. This is the answer for databases, indexes and anything the application opens read-write.
Do the mutation at build time. If the file only needs its header rewritten once, rewrite it in the Dockerfile so the layer ships in its final state. The copy then happens once during the build instead of once per container per start.
**Do not chown -R image content at runtime.** Set ownership in the Dockerfile with COPY --chown=, which bakes the right UID into the layer and removes the reason for the recursive copy entirely.

Inspecting a running container’s layers from the host

Read-only / Safefind the container's overlay directories— Ask the daemon rather than guessing at paths. The overlay2 directory names are cache IDs, not container IDs, so grepping /var/lib/docker/overlay2 for a container ID never matches anything.

CONTAINER=web
docker inspect --format '{{json .GraphDriver}}' "$CONTAINER" | jq .
UPPER=$(docker inspect --format '{{.GraphDriver.Data.UpperDir}}' "$CONTAINER")
echo "upper: $UPPER"
sudo du -sh "$UPPER"

{
"Data": {
  "LowerDir": "/var/lib/docker/overlay2/2f9c.../diff:/var/lib/docker/overlay2/8ab1.../diff",
  "MergedDir": "/var/lib/docker/overlay2/61de.../merged",
  "UpperDir": "/var/lib/docker/overlay2/61de.../diff",
  "WorkDir": "/var/lib/docker/overlay2/61de.../work"
},
"Name": "overlay2"
}
upper: /var/lib/docker/overlay2/61de.../diff
4.0G	/var/lib/docker/overlay2/61de.../diff

Illustrative output

On Engine 29 those paths are somewhere else entirely

Engine 29 made the containerd image store the default. On such a host the layers do not live under /var/lib/docker/overlay2 at all; they live under /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/, docker info reports the driver as overlayfs with a io.containerd.snapshotter.v1 driver type rather than overlay2, and docker inspect returns null for .GraphDriver — so the command above prints nothing useful.

docker diff and docker ps --size work on both, which is a good reason to reach for them first. When you do need the real directories on a containerd-store host, read them out of /proc/mounts:

grep -m1 ' overlay ' /proc/mounts | tr ',' '\n' | grep -E 'upperdir|workdir'

Everything this lesson says about copy-up, whiteouts and the lower/upper/work model is unchanged — it is the same kernel filesystem underneath. Only the paths moved.

Read-only / Safethe largest files in the writable layer— The direct answer to 'what filled the container layer'. Anything here that also exists in the image is a copy-up.

CONTAINER=web
UPPER=$(docker inspect --format '{{.GraphDriver.Data.UpperDir}}' "$CONTAINER")
sudo find "$UPPER" -type f -printf '%s\t%p\n' 2>/dev/null | sort -rn | head -10 | numfmt --field=1 --to=iec

4.0G	/var/lib/docker/overlay2/61de.../diff/opt/model/index.db
12M	/var/lib/docker/overlay2/61de.../diff/var/log/app/current
4.0K	/var/lib/docker/overlay2/61de.../diff/etc/app/config.yaml

Illustrative output

A 4 GB file at the top of that list, with the same path present in the image, is a copy-up and not a leak. docker image inspect the base image and look for the same path: if it is there, no amount of log rotation will help, and the file belongs in a volume.

Read-only / Safeimage size— The total image size is the sum of the lower layers, not the upper. The upper is empty when the container starts.

docker image inspect nginx --format '{{.Size}}' | numfmt --to=iec
docker image history nginx --no-trunc --format '{{.CreatedBy}}  {{.Size}}' | head -10

Performance characteristics

Reads of unchanged files: one extra indirection (look up in upper, fall through to lower). Negligible cost.
Writes to unchanged files: copy-up from lower to upper, then write. This is the cost most operators underestimate.
Writes to already-copied files: write to upper only. Cheap.
Deletes of files in lower: whiteout file in upper. Cheap.
Many small writes to a fresh container: every file gets copy-up before the write. This is why apt-get install is slow on first run after pulling a new layer.

For workloads that write a lot of small files, tmpfs (which is not OverlayFS) or a direct bind-mount to a fast filesystem may be faster.

The four directories

Mounting OverlayFS manually

What this means for Docker images

Inspecting a running container’s layers from the host

Performance characteristics

Knowledge check