Performance problems — slow application, container, host · Docker & Containers

The application is slow. The question is “where” — application, container, or host. The right tool depends on the layer.

Two of the layers lie to you if you use the obvious tool. docker stats reports CPU as a percentage averaged over a sampling interval, which hides the throttling that is the single most common containerised performance fault. And container memory in docker stats is a CLI-computed figure that subtracts cache — a reasonable choice that nonetheless means the number you see and the number the OOM killer uses are different numbers.

The methodology

flowchart TD
  S[App is slow] --> Q1{Application metrics?}
  Q1 -- latency high --> A[App-level issue]
  Q1 -- latency normal --> Q2{Container metrics?}
  Q2 -- CPU throttled --> C1[Increase quota or reduce usage]
  Q2 -- memory high --> C2[Memory leak, OOM risk]
  Q2 -- network saturated --> C3[Network config]
  Q2 -- normal --> Q3{Host metrics?}
  Q3 -- host CPU high --> H1[Host saturation]
  Q3 -- host memory high --> H2[Sum of containers > RAM]
  Q3 -- host disk I/O high --> H3[Disk contention]
  Q3 -- normal --> Q4{External dependency?}
  Q4 -- yes --> D[Database, cache, downstream]

The first measurement: is it throttled?

Before anything else, ask the cgroup whether the kernel has been stopping your process. This is a counter, not a sample, so it cannot be missed by bad timing, and it answers a question no other tool answers.

Read-only / SafeCFS throttling— cpu.stat is cumulative since the container started. Sample it twice.

CONTAINER=api

PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")
CGROUP=$(awk -F: '!/^0:/{print $3}' /proc/"$PID"/cgroup | head -1)
CG=/sys/fs/cgroup"$CGROUP"

echo '--- limit ---'
cat "$CG"/cpu.max

echo '--- throttling, sample 1 ---'
cat "$CG"/cpu.stat
sleep 60
echo '--- throttling, sample 2 ---'
cat "$CG"/cpu.stat

Illustrative output:

--- limit ---
50000 100000

--- throttling, sample 1 ---
usage_usec 184203118
user_usec 141002331
system_usec 43200787
nr_periods 421866
nr_throttled 98412
throttled_usec 2914003221

cpu.max of 50000 100000 is half a core: 50 ms of CPU time allowed per 100 ms period. nr_throttled divided by nr_periods is 23% — nearly a quarter of all scheduling periods ended with the container frozen until the next one began. throttled_usec is the total wall-clock time spent stopped.

Now look at what docker stats reports for the same container:

CONTAINER   CPU %    MEM USAGE / LIMIT   MEM %   NET I/O       BLOCK I/O   PIDS
api         38.42%   412MiB / 1GiB       40.2%   1.2GB / 890MB  0B / 4.1MB  34

38% CPU. Plenty of headroom, apparently. Both numbers are correct and they describe different things: 38% is the average over the sampling interval, and the throttling happens inside each 100 ms period. The container burns its 50 ms in the first 20 ms of the period and then does nothing for 80 ms. Averaged, that is modest utilisation. Experienced by a request in flight, it is an 80 ms stall.

Low CPU percentage, high latency, and a quota that is too small

This is the defining containerised performance bug and it is counter-intuitive enough that experienced Linux operators walk past it.

The symptom set: p99 latency several times p50, latency spikes clustering around a fixed value, docker stats showing comfortable CPU, and the application’s own profiler showing time going somewhere untraceable — because from the process’s point of view the CPU simply stopped being given to it, which no application-level profiler can see.

It is worst for multi-threaded runtimes. A JVM or a Go program that sees 32 host CPUs will create 32 threads and try to use them in parallel. With --cpus 0.5 they collectively get 50 ms per 100 ms period, so 32 threads consume the whole quota in under 2 ms of wall clock and then every one of them is frozen for the remaining 98 ms. Garbage collection, which is exactly the parallel phase, is where this hurts most.

The three fixes, in order of preference:

Tell the runtime the truth. GOMAXPROCS set from the quota, -XX:ActiveProcessorCount, or a library that reads cpu.max. Modern JVMs are container-aware and read the cgroup themselves; older ones and most language runtimes are not.
Raise the quota to match real demand, having first checked the host has the capacity — see the callout further down about oversubscription.
Remove the quota and use --cpu-shares (cpu.weight) instead, so the container is prioritised under contention rather than capped absolutely. This is the right shape for latency-sensitive services that are bursty, and the wrong shape for anything you need a hard ceiling on.

nr_throttled is the discriminator, it is free to read, and nothing else reports it.

Memory: two numbers, and only one kills you

Read-only / Safethe memory numbers that matter— What the cgroup accounts, what the kernel enforces, and what has already happened.

CONTAINER=api

PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")
CGROUP=$(awk -F: '!/^0:/{print $3}' /proc/"$PID"/cgroup | head -1)
CG=/sys/fs/cgroup"$CGROUP"

echo 'limit:      ' "$(cat "$CG"/memory.max)"
echo 'current:    ' "$(cat "$CG"/memory.current)"
echo 'peak:       ' "$(cat "$CG"/memory.peak 2>/dev/null || echo n/a)"
echo '--- events (oom_kill is the one that matters) ---'
cat "$CG"/memory.events
echo '--- breakdown: anon is unreclaimable, file is page cache ---'
grep -E '^(anon|file|slab|sock) ' "$CG"/memory.stat

memory.current includes page cache. docker stats deliberately does not: the CLI computes memory by subtracting cache from the total — inactive_file on cgroup v2 — while the API returns the raw figure. So the two disagree by design, and the disagreement is the page cache.

Which one predicts an OOM kill? Neither directly. The kernel enforces memory.max against memory.current, but page cache is reclaimable — it will be dropped under pressure rather than triggering a kill. The number that actually predicts trouble is anon from memory.stat: anonymous memory cannot be reclaimed, only swapped or killed.

Reading	Meaning
`memory.current` near `memory.max`, `anon` low, `file` high	Healthy. The cgroup is using its allowance as cache and will give it back.
`anon` climbing steadily toward `memory.max`	A leak or an undersized limit. This one ends in a kill.
`memory.events` `oom_kill` non-zero	It has already happened, at least once, whether or not you noticed.
`memory.events` `high` non-zero	Reclaim pressure is being applied — the application is being slowed to stay under the limit.

memory.events deserves a place in monitoring precisely because it is cumulative. A container that was OOM-killed at 04:00 and restarted looks completely normal at 09:00 in every live metric; the counter still says it happened.

Why `docker stats` costs more than you think, and what to use instead

docker stats is GET /containers/<id>/stats on the Engine API. For each container the daemon reads the full cgroup hierarchy, the network counters from the container’s netns, and the blkio statistics, then assembles a JSON document. Streaming mode repeats all of that once per second per container.

On a host with 80 containers this is measurable daemon CPU, and — worse during an incident — docker stats without --no-stream never returns, so it cannot be used in a script or a capture. Always pass --no-stream in automation.

For continuous measurement, read the cgroup files directly. They are the same source the daemon reads, they cost a read(2) each, and they expose fields the API does not surface at all: nr_throttled, memory.events, pids.events, and io.stat per device.

For the “is anything on this host starved” question, the pressure stall information files are better than any of them:

cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/cpu.pressure

PSI reports the percentage of wall-clock time tasks were stalled waiting for a resource. Unlike utilisation, it is directly comparable across resources and directly meaningful: some avg10=41.2 for io means that over the last ten seconds, tasks were blocked on I/O 41% of the time. A host at 100% CPU utilisation with zero CPU pressure is fully used and perfectly healthy; a host at 55% with high pressure is in trouble. That distinction is invisible to top.

The tools per layer

Application

# Profiling
go tool pprof http://app:6060/debug/pprof/profile  # 30s CPU profile
py-spy dump --pid PID                              # Python
node --prof app.js                                 # Node.js

For compiled languages, pprof / async-profiler / perf show function-level hotspots. For interpreted, the runtime’s profiler.

The container-specific catch: these tools need to see the process, and the process is in another PID namespace. py-spy and perf run from the host need --pid with the host-side PID, which is docker inspect --format '{{.State.Pid}}', not the PID the container reports. Run from inside the container they need SYS_PTRACE, which the default seccomp and capability set does not grant.

Read-only / Safeprofile a containerised process— Two approaches. Prefer the first; it changes nothing about the running container.

CONTAINER=api
PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")

# From the host, using the host-side PID. No container change needed.
sudo py-spy dump --pid "$PID"
sudo perf top -p "$PID"

# Map host PIDs to in-container PIDs so the two views line up
docker top "$CONTAINER" -eo pid,ppid,lwp,pcpu,rss,args

# If you must attach from inside, add the capability to a NEW debug
# container sharing the target's namespaces rather than to the target
docker run --rm -it \
--pid "container:$CONTAINER" \
--network "container:$CONTAINER" \
--cap-add SYS_PTRACE \
nicolaka/netshoot

The second form is the pattern worth learning. A sidecar debug container joined to the target’s PID and network namespaces gets full visibility without restarting the target, without changing its security profile, and without leaving SYS_PTRACE granted to a production workload afterwards. It is also the only way to get tooling into a distroless image.

Container

docker stats --no-stream api
docker top api
docker exec api cat /proc/loadavg

docker stats for a quick look, the cgroup files for anything you intend to act on. docker top is ps executed on the host against the container’s processes, so it works even when the image contains no ps.

Host

top
vmstat 1
iostat -xz 1
ss -tnp
cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory

Standard Linux performance toolset, with PSI added because it answers “is anything waiting?” which utilisation does not.

A worked example

Application latency is 800 ms (target: 200 ms). Walk the layers, and record what each one rules out.

# 1. Application metrics
#    Request duration: 800 ms p99, 190 ms p50
#    Database query duration: 750 ms p99, 12 ms p50
#    → The tail is in the database call. p50 is fine, so this is not
#      a uniformly slow system; something is intermittent.

# 2. Container: is the API being throttled?
#    nr_throttled/nr_periods = 0.4%  → no. Rules out the quota.

# 3. Host CPU and pressure
#    /proc/pressure/cpu:  some avg10=2.1   → not CPU-starved
#    /proc/pressure/io:   some avg10=44.7  → tasks blocked on I/O 45%
#      of the time. This is the finding.

# 4. Which device, and which container
#    iostat -xz 1: vda %util 96, await 38 ms
#    io.stat per cgroup: the db container owns most of the rbytes/wbytes

# 5. Inside the database
#    Long-running queries with sequential scans on a large table
#    → the disk saturation is a symptom of a missing index

The bottleneck is the database’s disk, driven by unindexed queries. Note what the walk bought: step 2 eliminated the most commonly-blamed cause in one command, and step 3 named the resource before anyone had to guess.

The intermediate wrong answers this avoids: adding CPU to the API container (step 2 said no), adding memory to the host (nothing pointed there), and moving to host networking (network was never implicated).

Raising the limit is a decision about the whole host

The obvious response to throttling is a bigger quota, and it is often right. But --cpus and --memory are not reservations — they are ceilings. Nothing stops you setting ceilings that sum to more than the host has.

Before raising a limit, check the sum:

docker ps -q | xargs -r docker inspect \
  --format '{{.Name}} cpus={{divf .HostConfig.NanoCpus 1000000000}} mem={{.HostConfig.Memory}}'

nproc
free -g

A host with 8 cores and containers whose quotas sum to 22 is fine while they are idle and becomes a queue the moment they are not. Memory is harsher: limits summing above physical RAM means the first simultaneous peak triggers the host OOM killer, which chooses its victim by its own heuristics and may well pick a container that was behaving.

If the sum is already over, the honest answer is that this host is oversubscribed and the fix is capacity or placement, not another increase. That is a capacity finding, and it belongs in a ticket rather than in a 03:00 docker update.

Disk and network, per container

Read-only / Safeper-container I/O attribution— Which container is driving the device, from the cgroup rather than by inference.

CONTAINER=db
PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")
CGROUP=$(awk -F: '!/^0:/{print $3}' /proc/"$PID"/cgroup | head -1)
CG=/sys/fs/cgroup"$CGROUP"

# Bytes and IOPS per block device, cumulative. Sample twice and subtract.
cat "$CG"/io.stat

# Is this cgroup stalling on I/O specifically?
cat "$CG"/io.pressure 2>/dev/null

# Map the major:minor numbers in io.stat to device names
lsblk -o NAME,MAJ:MIN,SIZE,MOUNTPOINT

# Network counters, from inside the container's netns
sudo nsenter -t "$PID" -n cat /proc/net/dev

io.stat keys are major:minor device numbers, which is why lsblk is in the block. Attributing device saturation to a container by reading iostat on the host and guessing is the step this replaces.

What happens if you diagnose the wrong layer?

You add CPU quotas to a container whose CPU is fine. You add memory to a host that doesn’t have a memory problem. You switch from bridge to host networking to “fix” latency that is actually database-driven.

Each wrong-direction fix costs time and may introduce new bugs. The systematic methodology prevents this.

There is a second cost that is easy to miss: a wrong fix that appears to work is worse than one that obviously does not. Raising a memory limit on a leaking container makes the OOM kills stop for a week, which reads as a successful fix and buries the leak. The next occurrence is at a larger scale with the original evidence long gone.

The guard is to require that a fix explains the measurement, not just the symptom. “We raised the limit and the kills stopped” is not an explanation. “anon was growing 40 MB/hour with no plateau, which is a leak, and here is the allocation site” is.

Verification

Read-only / Safeverify the performance fix— Compares throttling and pressure before and after, and fails on a threshold.

#!/usr/bin/env bash
set -euo pipefail
CONTAINER=api
WINDOW=120

PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")
CGROUP=$(awk -F: '!/^0:/{print $3}' /proc/"$PID"/cgroup | head -1)
CG=/sys/fs/cgroup"$CGROUP"

read_stat() { awk -v k="$1" '$1==k {print $2}' "$CG"/cpu.stat; }

p0=$(read_stat nr_periods);   t0=$(read_stat nr_throttled)
sleep "$WINDOW"
p1=$(read_stat nr_periods);   t1=$(read_stat nr_throttled)

periods=$(( p1 - p0 )); throttled=$(( t1 - t0 ))
[ "$periods" -gt 0 ] || { echo 'FAIL: no scheduling periods observed' >&2; exit 1; }

pct=$(( throttled * 100 / periods ))
echo "throttled $throttled of $periods periods = $pct percent"
[ "$pct" -lt 1 ] || { echo "FAIL: still throttling at $pct percent" >&2; exit 1; }

oom=$(awk '$1=="oom_kill"{print $2}' "$CG"/memory.events)
[ "$oom" -eq 0 ] || { echo "FAIL: $oom OOM kills recorded" >&2; exit 1; }

echo OK

Sampling a delta rather than reading the cumulative value is what makes this a verification. nr_throttled never decreases, so a container that was throttled badly before the fix will still show a large total afterwards; only the rate over a fresh window tells you whether the fix worked.