medium riskservice affecting~30 min

Runbook: Sustained high CPU — attribute it before you act

1 · Prerequisites

Confirm every item is in place before any state change.

Shell access on the Docker host; sudo is needed only for reading another user process detail and for perf
The host runs cgroup v2 (confirm with stat -fc %T /sys/fs/cgroup, which should print cgroup2fs)
You know the normal CPU baseline for this host, because high is a comparison and without a baseline it is a guess
You can state whether the complaint is CPU usage on a dashboard or latency reported by users; they lead to different steps
A second terminal is available, because most of these measurements are deltas taken over an interval

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· uptime
· nproc
· docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemPerc}}\t{{.PIDs}}'
· docker ps --format '{{.Names}} {{.Status}}'
· CONTAINER=web
· docker inspect -f '{{.HostConfig.NanoCpus}} {{.HostConfig.CpuQuota}} {{.HostConfig.CpuPeriod}} {{.HostConfig.CpuShares}} {{.HostConfig.CpusetCpus}}' "$CONTAINER"
· CID=$(docker inspect -f '{{.Id}}' "$CONTAINER"); cat /sys/fs/cgroup/system.slice/docker-"$CID".scope/cpu.stat

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Record the host baseline with uptime and nproc; expect the 1-minute load average and the CPU count. A load average below the CPU count is not a saturated host, whatever a percentage graph shows.
2Run docker stats --no-stream and sort by CPUPerc; expect one container to stand out. If every container is modest and the host is still busy, the consumer is on the host outside Docker and this runbook is the wrong one.
3Take a second docker stats --no-stream sample 30 seconds later; expect the same container to lead. A single sample catches startup spikes and health checks and attributes them as though they were sustained.
4Confirm the attribution from the cgroup rather than the summary - read usage_usec in /sys/fs/cgroup/system.slice/docker-"$CID".scope/cpu.stat twice, 10 seconds apart; expect the delta divided by 10 million to equal the number of CPUs consumed.
5Break the container down per process with docker top "$CONTAINER" -eo pid,pcpu,etime,comm --sort=-pcpu; expect one process or thread group to dominate. These are host PIDs, which is what makes the next step possible.
6For an unattributed host process, map its PID to a container with cat /proc/PID/cgroup; expect a line containing docker- followed by the full 64-character container ID, or no docker path at all if the process is not in a container.
7Read cpu.stat for nr_throttled and throttled_usec; expect both to be near zero. A throttled_usec that climbs while CPUPerc sits below the limit is throttling, and that is a latency incident rather than a capacity one.
8Compare the limit with the demand - read cpu.max in the same cgroup; expect quota then period, for example 100000 100000, which is one CPU. Then decide whether the limit is too low or the workload is too high.
9Apply one change - raise the quota with docker update --cpus, pin with --cpuset-cpus, or fix the workload - and record the before and after values of nr_throttled and the application latency.
10Re-measure over a full traffic cycle rather than for 30 seconds, because CPU problems are periodic far more often than they are constant.

4 · Verification

Confirm the procedure actually fixed the problem.

✓A 10-second delta of usage_usec in the container cgroup corresponds to the expected number of CPUs and no longer to the whole host
✓nr_throttled in /sys/fs/cgroup/system.slice/docker-"$CID".scope/cpu.stat stops increasing over a five-minute observation window under representative load
✓throttled_usec is flat across two samples taken ten minutes apart
✓The 1-minute load average from uptime is below the CPU count reported by nproc
✓Application p99 latency has returned to its recorded baseline - the number that made this an incident, not the CPU graph
✓docker stats --no-stream shows no single container above its intended share across three samples

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If docker update --cpus made throughput worse, restore the previous value with docker update --cpus and the number recorded in step 8; the change takes effect without a restart
↶If --cpuset-cpus was used to pin a container, remove the pinning with docker update --cpuset-cpus and the full CPU range, for example 0-7
↶If the limit was changed in a Compose file, revert the cpus or deploy.resources.limits.cpus value and run docker compose up -d for that service only
↶If a container was stopped to relieve the host, start it again and confirm docker ps shows it running before closing the incident
↶Reading measurements changes nothing, so steps 1 to 7 need no rollback. Only steps 8 and 9 alter state

6 · Escalation

When the runbook isn't enough, contact:

· CPU is consumed by kernel time rather than user time, or by a process outside every container: escalate to the platform team with the /proc/PID/cgroup output showing no docker path
· One container saturates its quota with no change in request volume: escalate to the application team as a suspected hot loop or a regression, attaching the per-process docker top output
· Throttling persists after the quota is raised to a value well above measured demand: escalate to the platform team, because the container may be contending for a specific CPU set or fighting an interrupt-heavy neighbour
· The host is a database or any latency-critical service: escalate before pinning CPUs, because cpuset changes interact with NUMA layout and can make latency worse

There are two CPU incidents that look alike on a dashboard and need opposite responses. In the first, a container is genuinely consuming the host. In the second, a container is being throttled — held below its quota by the scheduler — and reports modest CPU usage while every request it serves is slow. The second is the one that gets misdiagnosed, because the graph that would show it is not the CPU graph.

Attribute first. Every action in this runbook is wrong if applied to the wrong container.

Symptoms

Host load average above the CPU count, sustained.
One container at a high percentage in docker stats.
Or: normal CPU everywhere, and latency that has doubled.
Requests time out in bursts rather than continuously.

Step 1: Establish the baseline

Read-only / Safehost baseline— Reads load and CPU count. No side effects.

uptime
nproc
cat /proc/loadavg

# Pressure stall information: how much time was lost waiting for CPU
cat /proc/pressure/cpu

Load average is a count of runnable tasks, not a percentage. On a host with 8 CPUs, a load of 4 is half idle. Comparing load to nproc is the fastest way to find out whether there is an incident at all — and /proc/pressure/cpu is the honest signal for whether anything is actually waiting.

Step 2: Attribute to a container

Read-only / Safedocker stats— Samples container resource usage. No state change.

docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemPerc}}\t{{.PIDs}}'

# Take a second sample. One sample is an anecdote.
sleep 30
docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.PIDs}}'

docker stats reads the container’s cgroup CPU accounting over a short interval and reports the result as a percentage in which 100% is one full CPU. On an 8-CPU host a container using four cores reads roughly 400%, and a container pegged at 100% is using one core, not the whole machine. Reading it as a share of the host is the most common misreading of this output.

Read-only / Safecgroup CPU accounting— Reads cgroup v2 CPU counters. No state change.

CONTAINER=web
CID=$(docker inspect -f '{{.Id}}' "$CONTAINER")
CG=/sys/fs/cgroup/system.slice/docker-"$CID".scope

cat "$CG"/cpu.stat
cat "$CG"/cpu.max       # quota and period, or the word max for unlimited
cat "$CG"/cpu.weight    # relative weight, the cgroup v2 form of cpu-shares
cat "$CG"/cpu.pressure  # time tasks in this cgroup spent waiting for CPU

# Delta over ten seconds
grep usage_usec "$CG"/cpu.stat; sleep 10; grep usage_usec "$CG"/cpu.stat

Step 3: Map a host PID to a container

Sometimes the evidence arrives the other way round: top on the host shows a PID burning a core and no obvious owner. Every containerised process carries its cgroup path in /proc.

Read-only / SafePID to container— Reads /proc and container metadata. No state change.

# From the host's process table
ps -eo pid,pcpu,etime,comm --sort=-pcpu | head -10

# Which cgroup, and therefore which container, owns PID 2192?
cat /proc/2192/cgroup
# prints: 0::/system.slice/docker-CONTAINERID.scope, where CONTAINERID
# is the full 64-character container ID

# Turn that ID back into a name
docker inspect -f '{{.Name}} {{.Config.Image}}' 25fb9c8b983b

# The reverse direction: a container's main process on the host
docker inspect -f '{{.State.Pid}}' "$CONTAINER"

What this rules in or out. A /proc/PID/cgroup line with no docker- component means the process is not in a container at all — a host agent, a backup job, a kernel thread — and no amount of container tuning will touch it. That single check ends a surprising number of these incidents.

Step 4: Break it down inside the namespace

Read-only / Safeper-process inside the container— Runs ps on the host against the container's processes. No state change.

# docker top accepts ps options and runs on the HOST, so it works
# even for a distroless image with no shell and no ps of its own
docker top "$CONTAINER" -eo pid,pcpu,etime,comm --sort=-pcpu

# Thread-level, when one thread of a multi-threaded process is hot
docker top "$CONTAINER" -eLo pid,tid,pcpu,comm --sort=-pcpu | head -15

# If the image does have a shell and ps
docker exec "$CONTAINER" top -b -n 1 2>/dev/null | head -20

The PIDs docker top prints are host PIDs, which is exactly what you want: you can feed them straight to /proc, to ps, or to a profiler on the host without entering the namespace.

Step 5: Throttling — the diagnosis people miss

Read-only / Safethrottling counters— Reads CFS throttling statistics. No state change.

cat "$CG"/cpu.stat
# nr_periods      scheduling periods elapsed
# nr_throttled    periods in which the cgroup was throttled
# throttled_usec  total time tasks were held off the CPU

# Sample twice; only the delta means anything
grep -E 'nr_periods|nr_throttled|throttled_usec' "$CG"/cpu.stat
sleep 60
grep -E 'nr_periods|nr_throttled|throttled_usec' "$CG"/cpu.stat

Throttling looks like an application problem and reads as normal CPU

A container with --cpus 1 gets 100000 microseconds of CPU in every 100000 microsecond period. If it uses that budget in the first 40 milliseconds, the kernel stops scheduling it for the remaining 60 — every period, all day.

Averaged over a minute, docker stats reports something like 100%, which is precisely the limit and looks entirely healthy. Meanwhile every request unlucky enough to land in a stalled window waits tens of milliseconds for nothing, p99 latency triples, and the application team spends a week profiling code that is not slow.

The tell is nr_throttled climbing while CPU usage sits at or just under the limit. Divide nr_throttled by nr_periods: if a meaningful fraction of periods are throttled, that is your incident, and the fix is a higher quota or less work per request — not a faster algorithm.

Multi-threaded runtimes hit this hardest. A process with 16 threads on a host with 16 CPUs, limited to --cpus 2, can consume its entire period budget in an eighth of a period and then stall for the rest of it. Sizing the runtime’s thread pool to the CPU limit rather than to the host’s core count usually helps more than raising the quota.

Step 6: Understand which limit you actually set

This distinction decides whether your limit does anything at all on an otherwise idle host.

Flag	cgroup v2 file	Enforced	Behaviour
`--cpus 1.5`	`cpu.max`	Always	Hard quota. Throttles at the limit even when the host is idle
`--cpu-quota` and `--cpu-period`	`cpu.max`	Always	The same mechanism, expressed in raw microseconds
`--cpu-shares 512`	`cpu.weight`	Only under contention	A relative weight, not a cap
`--cpuset-cpus 0-3`	`cpuset.cpus`	Always	Restricts which CPUs may be used at all

Docker’s documentation says of --cpu-shares: “This is only enforced when CPU cycles are constrained. When plenty of CPU cycles are available, all containers use as much CPU as they need. In that way, this is a soft limit.” And it “doesn’t guarantee or reserve any specific CPU access.”

Docker documents the equivalence directly: --cpus 1.5 is “the equivalent of setting --cpu-period 100000 and --cpu-quota 150000” — which is what you will see in cpu.max.

Step 7: Change one thing

Configuration changeadjust CPU allocation— Changes the running container's CPU limits. Takes effect without a restart.

# Record the current values first
docker inspect -f '{{.HostConfig.NanoCpus}} {{.HostConfig.CpuShares}} {{.HostConfig.CpusetCpus}}' "$CONTAINER"

# Raise the quota
docker update --cpus 2 "$CONTAINER"

# Or restrict which CPUs it may use
docker update --cpuset-cpus 0-3 "$CONTAINER"

# Confirm the kernel agrees with the daemon
cat "$CG"/cpu.max

docker update changes a running container’s limits without recreating it, which is what makes it safe to try during an incident and easy to revert. For a Compose-managed service, make the same change in the Compose file afterwards, or the next docker compose up will quietly restore the old value.

What if raising the quota does not help?

Then the quota was not the constraint, and there are three usual alternatives.

It is not CPU at all. Check /proc/pressure/io and /proc/pressure/memory alongside cpu.pressure. A container blocked on disk shows high latency and unremarkable CPU, and looks identical from the outside.

It is one thread, not the process. A single-threaded hot loop cannot use more than one CPU no matter what quota you grant. docker top with -L shows per-thread usage; if one thread is at 100% and the rest idle, more quota changes nothing.

It is contention for specific CPUs. If --cpuset-cpus pins this container to cores that another container or a host interrupt handler is also using, the quota is available in theory and not in practice. Compare cpuset.cpus across containers before adding more.

Common patterns

Evidence	Diagnosis	Action
One container far above the rest, load above `nproc`	Genuine saturation	Find the process, then raise the limit or reduce the work
CPU at the limit, `nr_throttled` climbing, latency up	CFS throttling	Raise `--cpus`, or size the runtime’s threads to the limit
Low CPU everywhere, high latency, `cpu.pressure` low	Not a CPU incident	Check IO and memory pressure instead
Host busy, every container quiet	Consumer is outside Docker	Check `/proc/PID/cgroup` for the top host PIDs
Container uses every core despite `--cpu-shares`	Shares are a weight, not a cap	Use `--cpus` if a ceiling is required
Throttled on an idle host	Quota is absolute by design	Expected. Raise the quota if the limit is wrong
One thread at 100%, others idle	Single-threaded bottleneck	More quota will not help; escalate to the application team
Spikes every few minutes	Health check, cron, or GC	Correlate the interval with the health check interval first