CPU shares, quotas, and pinning · Docker & Containers

CPU control is more nuanced than memory, because CPU is not consumable in the same way. Memory a container holds is memory nobody else can have. CPU time a container does not use is simply given to somebody else, and reappears the instant it is wanted.

That single difference is why the most common CPU-limit bug is not “the limit was wrong” but “the limit did nothing until the day it did everything”.

There are three controls, and they are not interchangeable.

Quota — a hard ceiling

Configuration changecpu quota— Limit the container to the equivalent of 1.5 cores. Enforced by the kernel every 100 ms regardless of how idle the host is.

docker run -d --name api --cpus 1.5 myorg/api:1.0.0

--cpus 1.5 is shorthand for --cpu-quota=150000 --cpu-period=100000 and lands in the cgroup as cpu.max = "150000 100000": 150 ms of CPU time per 100 ms wall-clock period, which across multiple threads is 1.5 cores’ worth.

A quota is absolute. It applies on a completely idle host. This is the control you want for a hard cap, for capacity planning, and for making a workload’s performance reproducible.

Shares — a relative weight, and only under contention

Configuration changecpu shares— Relative priority. On an idle host these two containers behave identically.

docker run -d --name priority-app --cpu-shares 2048 myorg/api:1.0.0
docker run -d --name batch --cpu-shares 512 myorg/batch:1.0.0

--cpu-shares sets a weight. If both containers want CPU at the same instant and there is not enough to go round, the scheduler divides the contested time in proportion to the weights. If only one wants CPU, it gets all of it, share value irrelevant.

There is no --cpu-weight flag on docker run. The Docker flag is --cpu-shares (or -c); cpu.weight is the cgroup v2 file it writes to, and the two numbers are not the same value.

A share limit does nothing in testing and everything in production

This is the defining failure of CPU shares, and it is entirely predictable once stated.

You set --cpu-shares 512 on the batch job, deploy to staging, run the load test, and see no difference at all — because staging has one workload on it and nothing to contend with. The change looks inert. It ships.

Then production has a busy hour, three workloads want CPU simultaneously, and the batch job that ran in four minutes for six months takes forty. Nothing changed on the day it broke. The configuration that caused it landed weeks earlier and was verified as having no effect.

The corollary matters for testing: you cannot validate a share setting on an idle host. You have to generate contention. Run a CPU burner alongside the workload and measure both, or accept that the setting is untested. And when a workload mysteriously slows down, check whether a neighbour’s share value changed, not just its own.

`--cpu-shares 1024` makes a container lower priority, not default

1024 is documented as the default share value, and people therefore set it explicitly to mean “normal”. On a cgroup v2 host that is backwards.

runc converts shares to cpu.weight with 1 + ((shares - 2) * 9999) / 262142. So:

`--cpu-shares`	resulting `cpu.weight`
(not set)	`100` — the cgroup default, inherited
`512`	`20`
`1024`	`39`
`2048`	`79`
`10240`	`391`

A container with no --cpu-shares at all sits at weight 100. A container that explicitly asks for the “default” 1024 gets weight 39 — under contention it receives roughly 40% of the CPU share of the container that said nothing.

So a Compose file that sets cpu_shares: 1024 on some services and omits it on others has quietly created two tiers, in the opposite order from the one intended. Either set shares on every service in a contention group, or set them on none. Never set the “default” value explicitly.

Pinning — which cores, not how many

Configuration changecpu pinning— Restrict the container to CPUs 0 and 1. Also constrains which NUMA node its memory allocations are local to.

docker run -d --name db --cpuset-cpus="0,1" postgres:16

--cpuset-cpus is orthogonal to the other two: it chooses which cores, while quota chooses how much time. Use it for:

Latency-sensitive workloads that suffer from cache eviction when the scheduler migrates them between cores.
NUMA locality. Pin the container to the cores on the socket whose memory it is using; pair with --cpuset-mems for the matching NUMA node. Check the topology with lscpu | grep NUMA before choosing numbers.
Segregating a noisy workload onto cores that nothing latency-sensitive uses.

Pinning is easy to get wrong at scale. Two containers pinned to 0,1 contend with each other while cores 2–15 sit idle, and no quota is being exceeded, so nothing in the metrics says “throttled”. Treat a cpuset as an allocation that needs a registry, the same way static port assignments do.

Diagnosing throttling

A throttled container is slow in a way that looks like an application problem, a database problem, or a network problem. The counter that settles it is in the cgroup.

Read-only / Safecpu.stat— nr_periods is how many enforcement windows have passed; nr_throttled is how many ended with the container cut off.

$ CID=api; PID=$(docker inspect --format '{{.State.Pid}}' "$CID"); CG=$(awk -F: '/^0::/{print $3}' "/proc/$PID/cgroup"); cat /sys/fs/cgroup"$CG"/cpu.stat

usage_usec 918273645
user_usec 701122334
system_usec 217151311
nr_periods 864000
nr_throttled 212904
throttled_usec 4471928000

Illustrative output

The number that matters is the ratio:

nr_throttled / nr_periods — here 212904/864000 ≈ 25%. One period in four ended with the container stopped. That is severe.
throttled_usec / nr_throttled gives the average stall length — here about 21 ms per throttled period, which is a latency budget destroyed.

Under 1% is noise. Over 5% sustained means the quota is too low for the workload’s shape, which is not the same as too low for its average usage.

Read-only / Safethrottle ratio across every container— One pass over the host. Anything above a few percent deserves a look.

for c in $(docker ps -q); do
  name=$(docker inspect --format '{{.Name}}' "$c" | tr -d /)
  pid=$(docker inspect --format '{{.State.Pid}}' "$c")
  cg=$(awk -F: '/^0::/{print $3}' "/proc/$pid/cgroup")
  awk -v n="$name" '/nr_periods/{p=$2} /nr_throttled/{t=$2}
    END {if (p>0) printf "%-24s %6.2f%% throttled\n", n, 100*t/p}' \
    /sys/fs/cgroup"$cg"/cpu.stat
done

The multi-threaded quota trap: 40% average CPU, 100 ms stalls

This is the throttling case that wastes the most time, because every average-based metric says the container is fine.

A quota is spent across all threads together. Give a sixteen-thread application --cpus 1 and it has 100 ms of CPU per 100 ms period. Sixteen threads running in parallel consume that budget in roughly 6 ms of wall-clock time, and then every thread is frozen for the remaining 94 ms until the period rolls over.

The result:

Average CPU utilisation reads about 40%. Nothing looks saturated.
docker stats shows well under the limit.
p50 latency is normal.
p99 latency has a hard floor near the period length, because a request unlucky enough to arrive after the budget is gone waits for the next window.

Engineers see “40% CPU” and rule out CPU. The diagnosis is always nr_throttled, never a utilisation graph.

Three ways out, in order of preference:

Reduce the parallelism to match the quota. A container with --cpus 2 should be running about two worker threads, not sixteen. This is the real fix.
Raise the quota to match the concurrency the application actually needs.
Shorten the period. --cpu-period 10000 --cpu-quota 10000 is also “1 CPU”, but enforced every 10 ms, so the worst-case stall drops from 100 ms to 10 ms. Same throughput, an order of magnitude better tail latency. This is a genuine and underused technique.

Runtimes that read the limit — and the ones that do not

Point 1 above only works if the application knows what its limit is.

JVM: container-aware since JDK 10. availableProcessors() reflects cpu.max, and thread pools sized from it come out right. -XX:ActiveProcessorCount=N overrides it if you need to.
Go: container-aware since Go 1.25, which reads cpu.max and sets GOMAXPROCS from it, re-checking periodically in case the limit changes. Anything built with Go 1.24 or earlier defaults GOMAXPROCS to the host’s core count — 64 threads in a one-core container on a large host — unless the program uses a library such as automaxprocs or sets the variable explicitly.
Node.js: the libuv thread pool defaults to 4 regardless, but cluster-mode workers sized from os.cpus().length see the host.
Python / gunicorn: multiprocessing.cpu_count() sees the host. A workers = 2 * cpu_count() + 1 line in a config file is a very common cause of exactly the trap above.

Read-only / Safewhat the application thinks it has— nproc reads the CPU affinity mask, so it reflects --cpuset-cpus but NOT --cpus. That gap is the bug.

CID=api
docker exec "$CID" nproc
docker exec "$CID" cat /sys/fs/cgroup/cpu.max

nproc inside a container with --cpus 1 on a 32-core host prints 32. It is not lying — the affinity mask really does include all 32 cores; the quota just limits how much of them the container may use. Any code that sizes a pool from nproc is reading the wrong number.

cpu.max, cpu.weight, and what the scheduler does at the end of a period

Three files, three mechanisms:

cpu.max holds "$QUOTA $PERIOD" in microseconds, or "max 100000" for unlimited. The CFS bandwidth controller gives the cgroup a budget of QUOTA microseconds at the start of each PERIOD. Tasks consume it as they run; when it reaches zero every task in the cgroup is dequeued — moved off the run queue, still runnable, simply not scheduled — until the timer refills the budget. They are not sleeping and not blocked on anything, which is why they show as running-but-idle in most tooling.

cpu.weight is the proportional-share control, range 1–10000, default 100. It only ever matters when siblings under the same parent are simultaneously runnable and the parent’s CPU is oversubscribed.

cpu.stat is the evidence: nr_periods, nr_throttled, throttled_usec, plus nr_bursts and burst_usec if cpu.max.burst is in use. The counters are cumulative from cgroup creation, so sample twice and subtract if you want a rate rather than a lifetime average.

Confirm the translation yourself:

docker run -d --name t --cpus 1.5 --cpu-shares 2048 alpine:3.20 sleep 600
PID=$(docker inspect --format '{{.State.Pid}}' t)
CG=$(awk -F: '/^0::/{print $3}' "/proc/$PID/cgroup")
cat /sys/fs/cgroup"$CG"/cpu.max      # 150000 100000
cat /sys/fs/cgroup"$CG"/cpu.weight   # 79
docker rm -f t

79 rather than 2048 is the conversion formula at work, and seeing it once makes the shares table above stick.

Choosing between them

Goal	Control	Why
Cap a workload so capacity planning is predictable	`--cpus`	absolute, applies on an idle host
Stop a batch job disturbing an interactive one	`--cpu-shares` on both	relative; only acts when it needs to
Guarantee tail latency for one service	`--cpuset-cpus` plus a quota	removes migration and contention
Keep memory access NUMA-local	`--cpuset-cpus` + `--cpuset-mems`	pin cores and memory to one socket
Reduce throttle stall length without changing throughput	`--cpu-period` + `--cpu-quota`	shorter enforcement window

A useful default for a mixed production host is --cpus on everything (so nothing can run away) plus --cpu-shares within a tier (so the important thing wins when they collide). Pinning is a specialist tool; reach for it when you have measured a problem it solves.

What happens if the quota is too low — and how do you change it safely?

Symptoms, in the order they usually appear: p99 latency rises while p50 is unchanged; the service starts timing out under load it handled last week; CPU utilisation graphs look fine, which sends everyone to the database. nr_throttled climbing is the confirmation.

Unlike memory, CPU limits can be changed on a running container:

docker update --cpus 2.5 api
docker inspect api --format '{{.HostConfig.NanoCpus}}'

docker update accepts --cpus, --cpu-shares, --cpu-period, --cpu-quota, --cpuset-cpus, --cpuset-mems, --memory, --memory-reservation, --memory-swap, --blkio-weight, --pids-limit and --restart. It writes straight to the cgroup, so the effect is immediate and there is no restart.

Two cautions. docker update changes the running container only — recreate it from an unchanged Compose file and the old limit is back, so the emergency change must be followed by a commit. And raising one container’s quota on a host that is already saturated moves the problem rather than fixing it; check the host’s total allocation before you do it, not after.

The correct sizing method is not a rule of thumb: measure the workload’s CPU consumption at peak, set the quota above it, then watch nr_throttled for a week. A limit is right when the ratio sits near zero during normal operation and the container still cannot run away.