OverlayFS for containers - the layered filesystem

OverlayFS is the layered filesystem used by containers. It stacks multiple directories into a single merged view. This lesson covers how it works and the operational implications.

OverlayFS architecture

OverlayFS has three directories:

• lower: the read-only base (image layers).
• upper: the read-write container changes.
• work: a scratch directory for atomic operations.

The merged view is presented at a mount point. Reads check upper first, then lower. Writes go to upper.

Mount an OverlayFS

mkdir -p /tmp/overlay/{lower,upper,work,merged}
echo "from lower" > /tmp/overlay/lower/file
mount -t overlay overlay -o lowerdir=/tmp/overlay/lower,\
    upperdir=/tmp/overlay/upper,workdir=/tmp/overlay/work \
    /tmp/overlay/merged

cat /tmp/overlay/merged/file
# from lower

echo "from upper" > /tmp/overlay/merged/file
cat /tmp/overlay/upper/file
# from upper
cat /tmp/overlay/lower/file
# from lower (unchanged)

The lower file is unchanged; the upper file is the new content.

OverlayFS in containers

A container image is a stack of layers:

• Base layer: the root filesystem.
• Application layer: the application and dependencies.
• Container layer: the changes made by the container.

When the container runs, OverlayFS merges all the layers. The container’s writes go to the container layer.

# On the host
mount | grep overlay

# Find the container\'s mounts
cat /proc/<pid>/mounts | grep overlay

Inspect OverlayFS

The container’s root filesystem is an OverlayFS mount. The layers are visible in /var/lib/docker/overlay2/ (for Docker):

ls /var/lib/docker/overlay2/

# Each layer is a directory
# The merged layer is the container\'s root

Operational considerations

Image size

Each layer is a diff. The image size is the sum of layers. A large number of layers makes the image larger.

Use multi-stage builds to reduce image size:

FROM golang:1.20 AS build
WORKDIR /app
COPY . .
RUN go build

FROM gcr.io/distroless/base
COPY --from=build /app/server /server

The final image contains only the binary, not the build tools.

Layer caching

Layers are cached. A change in a layer invalidates that layer and all subsequent layers.

# Bad: copy everything, then build
COPY . /app
RUN go build

# Good: copy dependencies first, then build
COPY go.mod go.sum /app/
RUN go mod download
COPY . /app
RUN go build

The first COPY only invalidates the go build layer; the dependency download is cached.

Troubleshoot OverlayFS

Common issues:

• “No space left on device”: the upper dir is full. Check /var/lib/docker/overlay2/.
• “Read-only filesystem”: the container’s mount is read-only. Check the mount options.
• “Permission denied”: the upper dir has wrong permissions.

For each:

• Check disk space on the host.
• Check the container’s mount options.
• Check the file permissions in the upper dir.

Knowledge check