Read-only root filesystems and no-new-privileges

Two hardening flags are cheap, effective, and rarely break workloads. Every production container should have them.

They are also the two most commonly claimed and least commonly verified. Both are visible in docker inspect, and both are also visible in the kernel — and when a Compose file and the kernel disagree, the kernel is right. This lesson covers applying them, the failure they cause on the night nobody expected, and how to prove they are in force.

Read-only root filesystem

docker run -d --name web --read-only nginx:1.27-alpine

The container’s / is mounted read-only. Any process that tries to write to a path that is not an explicit mount fails with EROFS (Read-only file system).

That last clause is the important one. --read-only applies to the container’s own root mount. It does not apply to volumes, bind mounts, or tmpfs mounts, which are separate mounts stacked on top of it. This is what makes the flag usable: you mark the root immutable and then punch a hole exactly where the application needs one.

docker run -d --name web \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --tmpfs /var/cache/nginx:rw,noexec,nosuid,size=32m \
  --tmpfs /var/run:rw,noexec,nosuid,size=8m \
  nginx:1.27-alpine

The container can write only to those three paths. Everything else, including /etc, /usr, and the application’s own install directory, is immutable for the life of the container.

Finding the paths the application writes to

Do not guess. Run the workload once without --read-only and ask the storage driver what changed.

$ docker diff web

C /etc
C /etc/nginx
C /etc/nginx/conf.d
A /etc/nginx/conf.d/default.conf
C /run
A /run/nginx.pid
C /var/cache/nginx
A /var/cache/nginx/client_temp
A /var/cache/nginx/proxy_temp
C /tmp

Read it as a work list. A and C entries under /var/cache, /run, and /tmp become --tmpfs mounts. A entries that must survive a restart become named volumes. A entries in /etc that come from an entrypoint templating a config are the awkward case — either mount the rendered config in read-only, or give the entrypoint a small tmpfs at exactly that directory.

Run docker diff after a realistic exercise of the workload, not five seconds after start. A path that is only written during log rotation or during the first request of the day will not appear otherwise, and that is precisely the path that breaks you later.

Sizing tmpfs, and why an unbounded one is a memory bug

A tmpfs mount with no size= option defaults to 50% of the host’s total RAM. That is a kernel default, not a Docker one, and it is a bad fit for a container.

Worse, tmpfs pages are charged to the container’s memory cgroup. Files written into /tmp count against memory.max exactly as though the application had allocated them on the heap, and unlike page cache they are not reclaimable — the kernel cannot drop them, because there is no backing store to drop them to.

So a container run with --memory 512m --tmpfs /tmp (no size) will be OOM-killed the moment something writes 512 MB into /tmp, and the application log will show nothing at all, because the process was SIGKILLed mid-write. Always give --tmpfs a size=, and size the sum of your tmpfs mounts well below the memory limit.

CID=web
docker exec "$CID" sh -c 'grep tmpfs /proc/mounts'

No-new-privileges

docker run -d --name web \
  --security-opt no-new-privileges=true \
  nginx:1.27-alpine

By default, a process in a container can gain privileges it did not start with, through a setuid binary or a binary carrying file capabilities. no-new-privileges sets the kernel’s PR_SET_NO_NEW_PRIVS flag, and from that point on execve() will never grant a process more privilege than the process that called it.

The property that makes this valuable is that it is a one-way latch. It is inherited by fork() and preserved across execve(), and there is no prctl call to turn it off. An attacker who lands code execution inside the container cannot clear it, even as UID 0 in the container.

This blocks:

• setuid and setgid binaries — su, sudo, mount, passwd, chsh, and whatever else the base image happens to ship.
• file capabilities — an xattr-based grant such as cap_net_raw+ep on /bin/ping, which is how modern distributions ship ping instead of setuid-root.

It does not block:

• a process that already holds the privilege it wants. If you gave the container CAP_SYS_ADMIN, no-new-privileges does not take it away.
• a kernel vulnerability. This is a policy on execve(), not a sandbox.
• privilege gained through the Docker socket, a host bind mount, or a shared namespace. Those paths never call execve() in the container at all.

Verification that can fail

docker inspect reports what was requested. /proc reports what the kernel actually did. Check both, and expect them to agree.

$ docker inspect web --format 'ro={{.HostConfig.ReadonlyRootfs}} secopt={{.HostConfig.SecurityOpt}}'

ro=true secopt=[no-new-privileges=true]

$ PID=$(docker inspect --format '{{.State.Pid}}' web); grep NoNewPrivs "/proc/$PID/status"

NoNewPrivs:	1

For --read-only, the test that distinguishes working from broken is a write that must fail:

$ docker exec web sh -c 'touch /usr/local/CANARY' ; echo "exit=$?"

touch: /usr/local/CANARY: Read-only file system
exit=1

An exit code of 0 is the finding. Run the same test against a path you expect to be writable — /tmp — and confirm that one succeeds. Two tests, opposite expected outcomes, and you have proved the boundary is where you think it is.

Combining the two

docker run -d --name web \
  --read-only \
  --security-opt no-new-privileges=true \
  --cap-drop=ALL \
  --cap-add=NET_BIND_SERVICE \
  --user 10001:10001 \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --pids-limit 200 \
  nginx:1.27-alpine

The same thing in Compose, which is where it will actually live:

services:
  web:
    image: nginx:1.27-alpine
    read_only: true
    user: "10001:10001"
    pids_limit: 200
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /tmp:rw,noexec,nosuid,size=64m
      - /var/cache/nginx:rw,noexec,nosuid,size=32m

Note the separator: the CLI documents no-new-privileges=true and Compose examples conventionally use no-new-privileges:true. Both parse. What does not work is writing the option and omitting the value in a Compose file where the YAML parser then hands the daemon something it ignores — so always write the true.

Making it the host default

no-new-privileges is also a daemon-wide setting, which is the better place for it. daemon.json defaults it to false; set it once and every container on the host gets the flag whether or not the person who wrote the Compose file remembered:

{
  "no-new-privileges": true
}

sudo systemctl restart docker
docker run --rm alpine:3.20 grep NoNewPrivs /proc/self/status

There is no equivalent daemon-wide default for --read-only, because a read-only root breaks images that were never designed for one. That control stays per container, which is why the audit for it matters more.

Knowledge check