Storage performance — overlay2, drivers, and tuning

Storage performance in a container is host storage performance plus overlay2 overhead. That overhead is usually negligible and occasionally enormous, and the difference between the two cases is worth understanding precisely, because it decides where you put the data rather than how you tune the host.

The one overlay2 cost that is not small

Measure, do not guess

Three numbers answer “which container is doing the I/O” and all three come from the kernel.

CONTAINER=web
CID=$(docker inspect --format '{{.Id}}' "$CONTAINER")

cat "/sys/fs/cgroup/system.slice/docker-$CID.scope/io.stat"

$ cat /sys/fs/cgroup/system.slice/docker-$CID.scope/io.stat

8:0 rbytes=372002816 wbytes=0 rios=968 wios=0 dbytes=0 dios=0

rbytes=372002816 with wbytes=0 is a read-only workload — 354 MB read since start, nothing written. A container in the middle of a copy-up shows both counters climbing together at the same rate, which is the signature: overlay2 reads from lower and writes to upper in lockstep, and no application does that.

The other two views:

# Live rates for every container, one sample
docker stats --no-stream --format 'table {{.Name}}\t{{.BlockIO}}\t{{.MemUsage}}'

# What is being written into the container layer rather than a volume?
CONTAINER=web
docker diff "$CONTAINER" | head -40

# And how big is that layer? Non-zero here is a design smell.
docker ps --size --format 'table {{.Names}}\t{{.Size}}'

docker ps --size reports the writable layer size and the virtual size together. A container whose writable layer is measured in gigabytes is writing data that will vanish on docker rm, and is paying overlay2 costs to do it. That is almost always logs, an upload directory, or a cache that should have been a volume or a tmpfs.

Host tuning: what is actually worth doing

Much of the standard advice was written for hosts that no longer exist. Check before you change anything.

Mount options

findmnt -no FSTYPE,OPTIONS --target /var/lib/docker

$ findmnt -no FSTYPE,OPTIONS --target /var/lib/docker

ext4 rw,relatime

relatime is the default on every current Linux distribution, and it already suppresses almost all access-time writes: it updates atime only when the existing value is older than mtime/ctime or more than a day old. Moving from relatime to noatime eliminates the remaining once-per-file-per-day metadata write. It is a real but small win, worth taking on a dedicated filesystem and not worth a maintenance window on its own.

The advice that noatime is a major improvement dates from strictatime, which was the default before Linux 2.6.30. If your findmnt output says relatime, you have already had most of the benefit for fifteen years.

I/O scheduler

for dev in /sys/block/[sv]d*/queue/scheduler /sys/block/nvme*/queue/scheduler; do
[ -e "$dev" ] && printf '%s %s\n' "$dev" "$(cat "$dev")"
done

$ cat /sys/block/sda/queue/scheduler

[none] mq-deadline

For NVMe and most SSDs the kernel already selects none, because the device reorders better than the scheduler can and the queue depth makes merging pointless. If [none] is already in brackets, there is nothing to do.

Where a scheduler still earns its place is a rotational disk, or a host where one container’s I/O must not starve another’s. bfq provides that isolation and is also the scheduler required for --blkio-weight to have any effect.

TRIM

For SSDs and NVMe, periodic discard keeps write performance from degrading as the device fills.

systemctl enable --now fstrim.timer
systemctl status fstrim.timer

Prefer the weekly timer to the discard mount option: inline discard issues a TRIM on every delete, which on some devices costs more than it saves. Most distributions ship fstrim.timer enabled already — check rather than assume.

Databases and write-heavy workloads

The rule is one line: the data directory goes in a volume, not in the container filesystem.

docker run -d --name db \
  --mount type=volume,src=pgdata,dst=/var/lib/postgresql/data \
  -e POSTGRES_PASSWORD=REPLACE_ME \
  postgres:16

A volume mount bypasses the storage driver entirely, so there is no copy-up, no upper layer growth, and no whiteout bookkeeping. The database gets the host filesystem’s performance and its own durability guarantees back — which matters, because a database that believes fsync reached stable storage is making a promise that depends on nothing being layered underneath it.

This is also why the answer to “should I switch to the btrfs or zfs driver for a database?” is no. Those drivers change how the container layer works, and a correctly configured database does not use the container layer at all.

Direct I/O is a property of how the application opens its files — O_DIRECT on the open(2) call — and is configured in the database, not in Docker. Postgres exposes it through debug_io_direct; MySQL through innodb_flush_method. No docker run flag enables it, and none is needed: pass the database’s own setting and mount its data directory as a volume.

Throttling a noisy container

When one container’s I/O is starving the rest, limit it. Never respond by removing a security control.

DEV=/dev/sda

docker run -d --name batch --device-write-bps "$DEV:20mb" --device-read-bps "$DEV:40mb" --device-write-iops "$DEV:500" myorg/batch:1.0.0

--blkio-weight (10 to 1000, or 0 to disable) is the proportional alternative and shares spare capacity instead of capping it — but it only takes effect under a scheduler that implements weighting, which in practice means bfq. Under none it silently does nothing, which is the usual reason people conclude it “doesn’t work”.

The hard --device-*-bps and --device-*-iops limits do not have that dependency, so start with those when you need a guarantee rather than a preference.

Knowledge check