I/O controls and block device limits · Docker & Containers

Block I/O limits cap the bandwidth or the operation rate a container may use against a specific block device. They are the least reliable of the four resource controls in this part, not because the kernel does them badly, but because three things have to line up before they do anything at all — the right device, the right kind of I/O, and the right scheduler.

Getting a limit that appears to work and does not is the normal outcome of a first attempt. This lesson is mostly about how to tell.

Setting the limits

Configuration changebandwidth cap— Read and write bandwidth against one host device. The path is the HOST device, not a path inside the container.

docker run -d --name backup \
  --device-read-bps /dev/sda:20mb \
  --device-write-bps /dev/sda:10mb \
  myorg/backup:1.0.0

Configuration changeoperation-rate cap— IOPS rather than bandwidth. The right control when the device's limit is seeks or queue depth rather than throughput.

docker run -d --name backup \
  --device-read-iops /dev/sda:1000 \
  --device-write-iops /dev/sda:500 \
  myorg/backup:1.0.0

Configuration changerelative weight— Proportional share rather than a cap. Accepted range is 10 to 1000; 0 disables it, which is the default.

docker run -d --name batch --blkio-weight 250 myorg/batch:1.0.0

All four --device-* flags accept the same <host-device-path>:<value> shape and may be repeated for several devices.

The device you name is the host device

`/dev/sda` is often the wrong device, and the limit silently does nothing

The path in --device-read-bps /dev/sda:1mb is resolved by the daemon to a major:minor number on the host, and the kernel applies the throttle to that block device. It has no relationship to anything the container sees, and the container does not need a /dev/sda of its own.

So the limit does nothing whenever the container’s actual I/O goes somewhere else — which is common:

The container’s writable layer lives under /var/lib/docker, which may be on a different disk from /.
A named volume lives wherever the volume driver put it.
A bind mount points at an NFS or CephFS path, which is not a block device at all and cannot be throttled this way.
The filesystem sits on LVM or dm-crypt, so the I/O the container issues is against /dev/dm-3, and /dev/sda sees it only as merged writeback from the device-mapper layer.

Nothing warns you. docker run accepts the flag, docker inspect shows it, and the workload runs at full speed.

Find the device the workload actually uses before setting anything:

docker info --format '{{.DockerRootDir}}'
df -h /var/lib/docker | tail -1
lsblk -o NAME,MAJ:MIN,SIZE,TYPE,MOUNTPOINT

Then confirm the number the kernel received, rather than the one you typed.

Read-only / Safewhat the kernel actually recorded— io.max is keyed by major:minor. Compare against lsblk to prove the limit landed on the device you meant.

$ PID=$(docker inspect --format '{{.State.Pid}}' backup); CG=$(awk -F: '/^0::/{print $3}' "/proc/$PID/cgroup"); cat /sys/fs/cgroup"$CG"/io.max; lsblk -o NAME,MAJ:MIN | head

8:0 rbps=20971520 wbps=10485760 riops=max wiops=max
NAME    MAJ:MIN
sda       8:0
├─sda1    8:1
└─sda2    8:2
nvme0n1 259:0

Illustrative output

8:0 is sda. If /var/lib/docker is on nvme0n1 (259:0), that limit governs nothing the container does. An empty io.max means no limit was applied at all.

Why the write limit “doesn’t work”

This is the second, larger trap, and it has a clean demonstration.

Read-only / Safethe limit that looks broken— A 10 MB/s write cap, and dd reports over a gigabyte per second. Nothing is wrong with the limit.

$ docker exec backup dd if=/dev/zero of=/data/testfile bs=1M count=512

512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 0.41 s, 1.3 GB/s

Illustrative output

dd did not write to the disk. It wrote to page cache, and returned as soon as the data was in memory. The actual disk writes happen later, asynchronously, performed by kernel writeback threads. The block-layer throttle sits below page cache and never saw the write() calls at all.

Repeat with direct I/O, which bypasses page cache:

Read-only / Safethe same test, honestly— oflag=direct issues the writes to the block layer. Now the throttle is visible: 512 MB at 10 MB/s is about 51 seconds.

$ docker exec backup dd if=/dev/zero of=/data/testfile bs=1M count=512 oflag=direct

512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 51.4 s, 10.4 MB/s

Illustrative output

Always verify an I/O limit with oflag=direct (or iflag=direct for reads). A buffered test proves nothing, and it is the reason so many teams conclude the flags are broken.

That leaves the real question: does the limit apply to a normal application’s buffered writes, once writeback flushes them? It depends on cgroup writeback, which requires two things:

The memory and io controllers both enabled on the cgroup, so the kernel can attribute a dirty page to the cgroup that dirtied it.
A filesystem that implements it. The kernel documentation lists ext2, ext4, btrfs, f2fs and xfs. On any other filesystem, “all writeback IOs are attributed to the root cgroup” — meaning your container’s write limit governs nothing that goes through page cache.

Read-only / Safeare both controllers enabled here— cgroup.controllers must list both io and memory for writeback attribution to work. Check at the container's level, not just the root.

PID=$(docker inspect --format '{{.State.Pid}}' backup)
CG=$(awk -F: '/^0::/{print $3}' "/proc/$PID/cgroup")
cat /sys/fs/cgroup"$CG"/cgroup.controllers
findmnt -no FSTYPE,SOURCE /var/lib/docker

`--blkio-weight` may write a value that nothing reads

--blkio-weight is a proportional share, not a cap: under contention, cgroups get device time in proportion to their weights. runc converts the Docker range 10–1000 into the cgroup v2 range 1–10000 with 1 + (weight - 10) * 9999 / 990, so --blkio-weight 500 becomes io.weight 4950.

The catch is that io.weight has to be implemented by something in the block layer, and on many hosts it is not:

BFQ implements proportional weighting directly. If /sys/block/<dev>/queue/scheduler shows [bfq], weights work.
blk-iocost (CONFIG_BLK_CGROUP_IOCOST) implements io.weight independently of the scheduler, but only once io.cost.qos and io.cost.model have been configured on the root cgroup. Out of the box they usually have not been.
none, the default for NVMe, does no scheduling at all. With iocost unconfigured, io.weight is a number in a file that nothing consults.

Read-only / Safewill weights do anything on this host— Scheduler in brackets is the active one. An empty io.cost.qos with a 'none' scheduler means io.weight is inert.

for d in /sys/block/sd* /sys/block/nvme*; do
  [ -e "$d/queue/scheduler" ] || continue
  printf '%-14s %s\n' "$(basename "$d")" "$(cat "$d/queue/scheduler")"
done
cat /sys/fs/cgroup/io.cost.qos 2>/dev/null || echo 'iocost not configured'

If weights are inert on your hardware, use --device-write-bps and --device-write-iops instead. A hard cap is less elegant than a weight — it wastes capacity when the device is idle — but it demonstrably does something, which the weight may not.

When a limit is the right answer, and when it is not

Helps:

A backup, export or analytics job on the same spindle as a latency-sensitive database. Cap the batch job and the database’s await stops spiking.
Multi-tenant hosts on shared storage, where one tenant’s rsync can saturate the array for everyone.
Containing the blast radius of a runaway loop that writes in a tight cycle.

Hurts:

The primary workload. A database wants every IOP the device can give it; capping it converts spare capacity into latency.
Fast NVMe with abundant headroom. If the device is not the bottleneck, the limit is the bottleneck.
Anything bursty with a hard deadline. A cap smooths the burst into a long tail, which is exactly wrong for a job that must finish inside a window.

The default should be no limit on the primary workload, a limit on the noisy one. Limiting everything is a common and expensive overcorrection.

Diagnosing an I/O bottleneck

Read-only / Safeper-cgroup I/O accounting— io.stat is the container's own byte and operation counters, keyed by device. This is attribution, which iostat cannot give you.

PID=$(docker inspect --format '{{.State.Pid}}' backup)
CG=$(awk -F: '/^0::/{print $3}' "/proc/$PID/cgroup")
cat /sys/fs/cgroup"$CG"/io.stat

Read-only / Safedevice saturation and stall time— iostat shows the device's view; /proc/pressure/io shows how much time was lost waiting. Read both.

iostat -x 1 3
cat /proc/pressure/io
docker stats --no-stream --format 'table {{.Name}}\t{{.BlockIO}}'

%util near 100 with rising await means the device is saturated. For an SSD, await above about 10 ms is a warning; for a spinning disk, above 50 ms. /proc/pressure/io gives the number that actually correlates with user-visible pain: some avg10 is the percentage of the last ten seconds during which at least one task was stalled on I/O.

Note that iostat tells you the device is busy but not who made it busy. io.stat per container is the attribution step, and it is the one that ends the argument about which workload to limit.

io.max, and where the throttle sits in the stack

io.max is a nested-keyed cgroup file. Each line is keyed by $MAJ:$MIN with up to four nested keys:

8:16 rbps=2097152 wbps=max riops=max wiops=120

rbps and wbps are bytes per second, riops and wiops are operations per second, and max removes a specific limit. Docker’s --device-read-bps /dev/sda:20mb becomes rbps=20971520 on the line for sda’s major:minor.

The throttle is implemented in the block layer, at submission time. A bio that would exceed the cgroup’s allowance is queued and released when the token bucket refills. The submitting task blocks inside submit_bio(), which is why a throttled synchronous read shows up as D state in ps and as I/O pressure in /proc/pressure/io.

That position in the stack explains both of this lesson’s traps:

It is below page cache, so buffered writes are only throttled when writeback flushes them, and only if cgroup writeback can attribute the dirty pages to the right cgroup.
It is at the block device, so the limit follows major:minor and not any path, mount, or container-visible device node.

There is a knock-on worth knowing: throttling writeback slows the rate at which dirty pages are cleaned, so a container writing hard against a tight wbps accumulates dirty pages and comes under memory pressure. A wbps limit set aggressively on a write-heavy container can therefore show up as a memory problem — which is why memory.events is worth a glance when an I/O limit change is followed by odd behaviour.

What happens if the limit is too low, and how do you back it out?

The application looks slow and the CPU looks idle. Requests pile up in D state, load average climbs while %CPU does not, and /proc/pressure/io shows sustained stall. From the application’s side it is indistinguishable from a slow disk — which is precisely what you have built.

The dangerous version is a queue that never drains: a limit below the workload’s steady-state write rate means the backlog grows without bound, so the container is fine for twenty minutes and then degrades permanently. Compare the limit against the sustained rate from io.stat, not against a peak.

Backing it out is live, with no restart:

docker update --blkio-weight 500 backup

docker update covers --blkio-weight, but not the --device-*-bps and --device-*-iops flags — those are set at create time. To change a bandwidth cap you either recreate the container or, for an emergency, write the cgroup file directly:

PID=$(docker inspect --format '{{.State.Pid}}' backup)
CG=$(awk -F: '/^0::/{print $3}' "/proc/$PID/cgroup")
echo '8:0 wbps=max' | sudo tee /sys/fs/cgroup"$CG"/io.max

That takes effect immediately and is lost on the next container restart, so treat it as an incident action with a follow-up commit — the same pattern as memory.high.

Setting the limits

The device you name is the host device

Why the write limit “doesn’t work”

--blkio-weight may write a value that nothing reads

When a limit is the right answer, and when it is not

Diagnosing an I/O bottleneck

Knowledge check

`--blkio-weight` may write a value that nothing reads