Layers and the copy-on-write filesystem · Docker & Containers

An image is not a tarball. It is an ordered list of immutable layers plus a manifest that describes how to compose them. The filesystem visible inside a container is the union of these layers, mounted via OverlayFS.

What an image is

flowchart TB
  M[Manifest] --> C[Image config]
  M --> L1[Layer 1: FROM ubuntu]
  M --> L2[Layer 2: apt-get install nginx]
  M --> L3[Layer 3: COPY nginx.conf]
  M --> L4[Layer 4: ENTRYPOINT nginx]

Every image has:

A manifest describing the image, including its size, creation date, and layer list.
An image config describing the runtime configuration: env vars, entrypoint, command, working dir, exposed ports, labels.
One or more layers. Each layer is a tar archive of the filesystem changes introduced by a Dockerfile instruction.

When a container is created, the image’s layers become the container’s lower OverlayFS layers; the container gets a new empty upper layer.

Inspecting an image

docker image inspect nginx:1.27

The output includes:

Id — the image’s content-addressable identifier.
RepoDigests — the digest this image was pulled from.
RootFS.Layers — the SHA256 of each layer.
Config — the runtime configuration.
Os, Architecture, Size — the platform details.
Created — when the image was built.

Use jq for specific fields:

docker image inspect nginx:1.27 \
  --format '{{json .RootFS.Layers}}' | jq '.[]' | head -5

Image size: virtual vs on-disk

A 1 GB image does not take 1 GB per container. Layers are shared between containers started from the same image.

docker image ls
# SIZE column shows the image's virtual size

docker system df
# Shows the actual on-disk size after shared-layer deduplication

docker system df gives the truth:

Images — total size of all pulled images.
Containers — size of writable layers.
Local Volumes — total size of named volumes.
Build Cache — BuildKit’s local cache.

Read-only / Safedisk reality— docker system df is the only command that shows the real disk consumption after shared-layer deduplication.

$ docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          14        4         2.731GB   1.842GB (67%)
Containers      4         0         78.3MB    78.3MB (100%)
Local Volumes   8         1         312.4MB   156.2MB (50%)
Build Cache     92        0         1.273GB   1.273GB

Deleting in a later layer deletes nothing

A layer records the changes an instruction made. There is no mechanism for a layer to reach backwards and modify an earlier one — they are immutable and content-addressed, and an earlier layer’s digest is baked into everything above it. So a layer cannot express “that file is gone”. It can only express “hide it”.

Whiteouts in the OCI layer format

When an instruction deletes a file that came from a lower layer, the resulting layer tar does not omit the file. It adds an entry:

path/to/.wh.secret.env

The .wh. prefix is the OCI image layer format’s whiteout marker. At unpack time the runtime sees it and hides path/to/secret.env from the merged view. For a deleted directory whose contents should all vanish, there is .wh..wh..opq — an opaque marker on the directory itself.

Both are additions. A layer that deletes a 300 MB directory is a layer that adds a handful of small marker files, and the image grows very slightly. The 300 MB is still in the layer beneath it, still shipped on every pull, still on disk after every unpack, and still readable by anyone who has the image.

The scanner reports a credential in a file the image does not have

A supply-chain scan flags myorg/api:1.4.0 for a private key at /app/deploy_key. You check immediately:

$ docker run --rm myorg/api:1.4.0 ls -l /app/deploy_key
ls: cannot access '/app/deploy_key': No such file or directory

The file is not there. The Dockerfile clearly removes it. The obvious conclusion is a scanner false positive, and the finding gets suppressed — which is the wrong call, and the reason it is wrong is three lines up in the Dockerfile:

COPY deploy_key /app/deploy_key
RUN git clone git@github.com:myorg/private.git /src
RUN rm -f /app/deploy_key

Three instructions, three layers. The first layer contains the key. The third layer contains a whiteout marker that hides it. Every docker pull of this image transfers the key, and anybody who can pull it can extract it with docker save and tar. The runtime view is genuinely empty; the artefact is not.

Why “no such file” is such a convincing wrong answer. Every tool that inspects a running container — docker exec, docker run ... ls, a shell in the image — reads the merged view, where the whiteout has done its job. To see the truth you have to look at the layers, and almost nothing looks at the layers by default.

The && version is genuinely different. RUN copy && use && rm in a single instruction produces one layer whose final state has no key in it, because the layer records the filesystem after the whole command ran. Same three operations, same order — one layer instead of three, and the key never exists in any shipped artefact. This is the entire reason the “chain your RUN commands” advice exists; the image-size benefit is the smaller half of it.

And the response is not just a rebuild. That key was published. Rebuild the image correctly and rotate the credential — the old image is in the registry, in every layer cache, and on every host that pulled it. Removing the tag does not remove the blob.

Read-only / Safewhat is actually inside the layers?— Unpack the image and look at the layer tars directly. This is the only view that sees past whiteouts, and it is what an attacker with pull access has.

IMAGE=myorg/api:1.4.0
WORK=$(mktemp -d)
docker save "$IMAGE" | tar -C "$WORK" -x
find "$WORK" -type f ! -name '*.json' -print0 \
  | xargs -0 -I{} sh -c 'tar -tf "{}" 2>/dev/null | sed "s|^|{} |"' \
  | grep -E 'deploy_key|\.wh\.'
rm -rf "$WORK"

.../blobs/sha256/9f3a...  app/deploy_key
.../blobs/sha256/c418...  app/.wh.deploy_key

Illustrative output

Two lines, and together they are the whole finding: one layer holds the file, a later layer holds the whiteout that hides it. If the image were clean, the first line would not exist. That is a check whose output distinguishes “the scanner is wrong” from “the scanner is right and we have a key to rotate”, and it takes about ten seconds.

docker history gives the cheaper, coarser version of the same signal — a rm step whose layer size is a few bytes has not reclaimed anything:

Read-only / Safedid that rm reclaim anything?— Compare the size of the instruction that created the file with the size of the instruction that removed it. A near-zero removal layer means the bytes are still in the image.

IMAGE=myorg/api:1.4.0
docker history --no-trunc --format '{{.Size}}\t{{.CreatedBy}}' "$IMAGE" | head -12

0B	CMD ["/app/api"]
0B	USER 10001
32B	RUN /bin/sh -c rm -f /app/deploy_key # buildkit
0B	RUN /bin/sh -c git clone git@github.com:myorg/private.git /src # buildkit
3.38kB	COPY deploy_key /app/deploy_key # buildkit
77.8MB	/bin/sh -c #(nop) ADD file:... in /

Illustrative output

The rm layer is 32 bytes — the whiteout marker. The COPY layer below it is still 3.38 kB, and that 3.38 kB is the key. Layer sizes in docker history never go down, and a “cleanup” step that shows a tiny positive size is a cleanup that did not happen.

The layer cache

Each layer is content-addressable: its identifier is the SHA256 of its tar archive. When you build an image, Docker checks whether each layer is already in the cache (by digest). If yes, the instruction that produced it is skipped (“layer already exists”).

This is why Dockerfile instruction order is critical:

Place frequently-changing instructions late.
Place rarely-changing instructions (apt-get update, package install) early.
Combine related operations into a single RUN to keep the layer count low.

A typical mistake:

# Bad — apt-get update cached for a long time, then invalidated
# when you change any later line.
RUN apt-get update
COPY app.py /app/
RUN pip install -r /app/requirements.txt

vs:

# Good — change the order so layer cache is invalidated only
# when the actual dependencies change.
COPY requirements.txt /app/
RUN pip install -r /app/requirements.txt
COPY app.py /app/