Image size and the cost of layers · Docker & Containers

Image size is the most quoted number in container operations and the most loosely reasoned about. It is a genuine cost. It is a cost on specific things, and being precise about which ones is the difference between an optimisation that pays and an afternoon spent shaving 40 MB off an image that is pulled once a month and then never again.

What image size costs, and what it does not

Image size drives	It does not drive
Cold pull time on a host that lacks the image	Runtime CPU or memory of the running process
Decompression and extraction time on that host	Request latency
Disk consumed under `/var/lib/docker`	Application startup time once the image is present
Registry storage and egress cost	Anything at all on a warm host
Number of packages a scanner reports as vulnerable	Whether the application is actually exploitable

The right-hand column deserves emphasis because it is where the folk belief lives. A 2 GB image and a 20 MB image, both already present on the host, run at identical speed.

Why a large image does not make a running container slower

The container’s root filesystem is an OverlayFS composition of read-only image layers. Nothing is copied at start, and nothing is loaded eagerly. The layers are directories on the backing filesystem, and files are read into the page cache only when the process touches them.

A 2 GB Python image whose service imports 60 MB of libraries faults in 60 MB. The other 1.94 GB — the compiler toolchain, the documentation, the locale data, the .pyc files for modules nobody imports — sits on disk, never read, consuming no memory and costing no time.

There is a second-order effect in the opposite direction: shared lower layers share page cache entries across containers, so a fat base image used by twenty containers costs its resident set once, not twenty times. Density is helped by sharing, not by smallness.

What size genuinely costs at runtime is disk, and disk is what fills, so the cost is real. It is just not latency, and treating it as latency sends people to optimise the wrong thing.

Three numbers, all called “size”

Read-only / Safethe three sizes— Same image, three legitimate answers, differing by more than 3x.

$ docker image ls python:3.12-slim && docker system df -v | grep python

REPOSITORY   TAG          IMAGE ID       CREATED       SIZE
python       3.12-slim    6c4dd321d176   2 weeks ago   179MB

REPOSITORY   TAG         IMAGE ID       SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS
python       3.12-slim   6c4dd321d176   179MB     176.9MB       2.179MB       0

Illustrative output

Compressed size is what crosses the network. It is not in either command above; it is in the registry manifest. Roughly one third of the logical size for a typical Debian-based image.
Logical size — the SIZE column of docker image ls — is every layer uncompressed, added together. It double-counts layers shared with other images, which is why summing the column exceeds du on /var/lib/docker.
Unique size is what deleting this image would actually free. python:3.12-slim above has 179 MB of logical size and 2.2 MB of unique size, because everything else is shared with other images on the host.

Read-only / Safeget all three— Compressed from the manifest, logical from inspect, unique from system df.

IMAGE=python:3.12-slim

docker manifest inspect "$IMAGE" \
| awk '/"size"/ {gsub(/[^0-9]/,"",$2); s+=$2} END {printf "compressed: %.1f MB\n", s/1048576}'

docker image inspect "$IMAGE" --format 'logical:    {{.Size}}' | numfmt --field=2 --to=iec

docker system df -v | awk '/^REPOSITORY/{p=1} p && /python/ {print "unique:     " $6}'

Quote the right one for the question. Cold pull time is governed by compressed size. Host disk pressure is governed by unique size. The SIZE column that everyone quotes governs neither on its own.

What dominates, in order

Base image. One line, an order of magnitude.
Build toolchain left in the runtime image. Compilers, headers, build-essential, the Go SDK. Hundreds of megabytes, entirely avoidable with multi-stage.
Package manager caches. /var/lib/apt/lists, /root/.cache/pip, /root/.npm, /go/pkg/mod. Frequently 100–400 MB.
Application artefacts. Usually the smallest part, and the only part anybody wanted.
Layer metadata. Negligible.

Read-only / Safemeasure your own base images— Do not trust a table on the internet, including this one. Pull and measure.

for I in ubuntu:24.04 debian:bookworm-slim alpine:3.20 gcr.io/distroless/static-debian12; do
docker pull -q "$I" > /dev/null 2>&1 || { echo "$I: pull failed"; continue; }
printf '%-42s %s\n' "$I" "$(docker image inspect "$I" --format '{{.Size}}' | numfmt --to=iec)"
done

Expect roughly 80 MB, 75 MB, 8 MB and 2 MB of logical size, with compressed sizes near a third of each. The precise figures move with every base image release, which is exactly why the command matters more than the numbers.

Deleting a file in a later layer makes the image bigger

This is the single most common Dockerfile mistake and it is invisible in the final docker image ls output, which is why it survives review.

RUN apt-get update && apt-get install -y build-essential
RUN make && make install
RUN apt-get purge -y build-essential && rm -rf /var/lib/apt/lists/*

The final image still contains every byte of build-essential. Each RUN produces a layer, layers are immutable and stacked, and a deletion in layer 3 is recorded as a whiteout entry that hides the files in layer 1 from the merged view. The data is still there, still shipped, still pulled, still on disk. You have added a layer and removed nothing.

The same applies to secrets. A credential written in one layer and deleted in the next is fully recoverable by anyone who can pull the image, using nothing more exotic than docker save and tar.

Two correct forms. Delete inside the same RUN that created the file, so the layer never contains it:

RUN apt-get update \
 && apt-get install -y --no-install-recommends build-essential \
 && make && make install \
 && apt-get purge -y build-essential \
 && rm -rf /var/lib/apt/lists/*

Or, better, put the toolchain in a stage that never ships at all.

Multi-stage: the fix that actually works

# syntax=docker/dockerfile:1
FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -trimpath -o /out/app ./cmd/app

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/app /app
USER nonroot:nonroot
ENTRYPOINT ["/app"]

Only what COPY --from names ends up in the final image. The 800 MB Go toolchain in the build stage is not a hidden layer of the result; it is a separate image that never leaves the build host.

The cache mounts are the second half of the technique. --mount=type=cache gives the build a persistent directory that is not part of any layer, so the module cache survives between builds without ever being committed. That is strictly better than the old rm -rf dance: fast rebuilds and nothing in the image.

Read-only / Safefind the fat layers— docker history attributes size to the instruction that created it. Sort mentally by the size column.

IMAGE=myapp:1.4.2

docker history --no-trunc --format 'table {{.Size}}\t{{.CreatedBy}}' "$IMAGE" \
| head -25

Read-only / Safea history worth reading— One instruction accounts for 71% of the image.

$ docker history --format 'table {{.Size}}\t{{.CreatedBy}}' myapp:1.4.2

SIZE      CREATED BY
12.4MB    COPY /out/app /app # buildkit
0B        ENV PYTHONUNBUFFERED=1
684MB     RUN /bin/sh -c pip install -r requirements.txt # buildkit
41.2MB    RUN /bin/sh -c apt-get update && apt-get install -y gcc g++ # buildkit
0B        CMD ["python" "-m" "app"]
77.8MB    /bin/sh -c #(nop) ADD file:... in /

Illustrative output

684 MB in one pip install, in an image whose application is 12 MB. That is almost certainly build dependencies compiled at install time and left behind. The fix is a build stage that produces wheels, and a runtime stage that installs only the wheels.

Choosing a base image on compatibility, not megabytes

The smallest base that runs the workload — not the smallest that exists.

Alpine is smaller and is not always cheaper

Alpine uses musl libc rather than glibc. That saves a lot of megabytes and introduces differences that cost real time:

Python has no manylinux wheels for musl in many cases. A pip install that downloads a prebuilt wheel on Debian compiles from source on Alpine. The build needs gcc, g++, headers and a Rust toolchain for some packages, so the “small” image acquires a 400 MB build stage and a ten-minute build, and frequently ends up larger than the Debian slim equivalent.

DNS resolution differs. musl historically queries the nameservers in resolv.conf in parallel and handles search domains and truncated UDP responses differently from glibc. Applications that worked in every other environment can fail to resolve internal names, and it presents as an intermittent application bug rather than a base image choice.

Threading and stack sizes differ. musl’s default thread stack is much smaller than glibc’s, which surfaces as stack overflows in deeply recursive code that has run fine for years.

Alpine is an excellent base for Go and Rust static binaries, for shell utilities, and for anything with no native dependencies. For a Python or Java service, debian:bookworm-slim is usually smaller in practice once you count the build stage, and is always less surprising.

Distroless images have the opposite trade: they contain the runtime and nothing else — no shell, no package manager, no ps. That is a real security improvement and a real operability cost, because docker exec gives you nothing. The mitigation is the technique from the performance-tools lesson: nsenter from the host, or a debug container sharing the target’s namespaces. Adopt distroless and establish the debugging path, in the same change.

Verification that can fail

Read-only / Safeimage size gate— A CI gate that fails when an image grows past its budget. Budgets that are not asserted always drift upward.

IMAGE=myapp:1.4.2
BUDGET_MB=150

SIZE_MB=$(( $(docker image inspect "$IMAGE" --format '{{.Size}}') / 1048576 ))
echo "logical size: $SIZE_MB MB (budget $BUDGET_MB MB)"

if [ "$SIZE_MB" -gt "$BUDGET_MB" ]; then
echo 'FAIL: image exceeds budget. Largest layers:'
docker history --format '{{.Size}}\t{{.CreatedBy}}' "$IMAGE" | head -5
exit 1
fi
echo 'PASS'

And the check that catches the deleted-but-still-present problem, which no size number reveals:

Read-only / Safefind files hidden by whiteouts— Export the image layers and look for whiteout entries. Anything hidden is still being shipped.

IMAGE=myapp:1.4.2
WORK=$(mktemp -d)

docker save "$IMAGE" -o "$WORK/img.tar"
tar -tf "$WORK/img.tar" > "$WORK/entries.txt"

# Layer tars containing whiteout markers indicate content deleted in a later
# layer but still present in an earlier one.
grep -c '\.wh\.' "$WORK/entries.txt" \
&& echo 'whiteouts present: something is deleted in a later layer and still shipped' \
|| echo 'PASS: no whiteout entries at the top level'

rm -rf "$WORK"

What happens if you optimise size and startup gets no faster?

That is the expected outcome, and it is worth predicting before the work rather than explaining after it.

Shrinking an image improves phases 1 to 3 of container startup — resolve, download, extract. Those phases run once per host per image digest. On any host that already has the image, a smaller image changes nothing measurable, because the container skips straight to snapshot preparation.

So the honest expected result of halving an image is:

Cold hosts start faster. New nodes, scale-out events, and the first container of a new deploy on each host.
Deploys get faster only if the changed layers got smaller. If your application layer was always 12 MB and you halved the base, deploys are unaffected — the base was already cached everywhere.
Disk pressure improves by the unique size, not the logical size. Halving a 400 MB image that shares 380 MB with three others frees 10 MB.
Warm start time does not move at all.

If your goal was faster deploys, measure which layers change between builds first. The fix is usually Dockerfile ordering — dependencies before source, so the dependency layer is cached — rather than a smaller base. If your goal was lower cold start, size is the right lever and a registry mirror is a bigger one.

Knowledge check

Why this matters

Image size optimisation is popular because it produces a number that goes down and a graph anyone can read. That makes it easy to fund and easy to over-invest in.

The value is real but narrowly targeted: distribution, storage, and the size of the surface a vulnerability scanner has to report on. It is not runtime performance, it is not latency, and on a fleet whose hosts already hold the base layers it is frequently not deploy speed either.

The version of this work that pays is precise about the target. If deploys are slow, order the Dockerfile so only the application layer changes and confirm the base is already cached fleet-wide. If new nodes are slow to become useful, pre-pull during provisioning. If the disk is filling, look at unique size and at retention policy rather than at any single image. And if the goal is fewer CVEs, drop the packages you do not run — which shrinks the image as a side effect, and is worth doing for its own reasons.

Pick the smallest base that actually runs the workload, not the smallest that exists. A 12 MB image nobody on the team can debug at 03:00 is not an optimisation.