Linux performance tools for containers

A container is a normal Linux process with restricted views and restricted resources. Every performance tool you already know works on it. The difficulty is that most of them, run the way you would run them on a host, silently answer a different question than the one you asked — and return a plausible number rather than an error.

Start here: top inside a container is lying to you

$ docker run --rm -m 256m --cpus 0.5 alpine:3.20 sh -c 'nproc; free -m | head -2'

32
             total        used        free      shared  buff/cache   available
Mem:          128737        9821       94013         512       24902      117204

Thirty-two CPUs. One hundred and twenty-eight gigabytes of RAM. The container has half a core and 256 MB.

The numbers that are actually the container

CONTAINER=api

docker exec "$CONTAINER" sh -c '
echo "memory.max     $(cat /sys/fs/cgroup/memory.max)"
echo "memory.current $(cat /sys/fs/cgroup/memory.current)"
echo "cpu.max        $(cat /sys/fs/cgroup/cpu.max)"
echo "--- memory.events ---"
cat /sys/fs/cgroup/memory.events
echo "--- cpu.stat ---"
cat /sys/fs/cgroup/cpu.stat
'

$ docker exec api sh -c 'cat /sys/fs/cgroup/cpu.stat /sys/fs/cgroup/memory.events'

usage_usec 184920331
user_usec 141002118
system_usec 43918213
nr_periods 402118
nr_throttled 291044
throttled_usec 812004119
low 0
high 0
max 0
oom 0
oom_kill 0

Four fields carry almost all the diagnostic value:

• memory.events oom_kill — nonzero means the kernel has killed something in this cgroup for exceeding memory.max. This is the definitive answer to “did we get OOM-killed”, and it is a counter, so it records kills that happened before you arrived.
• memory.events high — nonzero means the cgroup hit memory.high and was throttled into reclaim. The container is not dead but is spending time in the reclaim path instead of doing work.
• cpu.stat nr_throttled / nr_periods — the ratio is the fraction of scheduling periods in which the cgroup exhausted its quota and was stopped. Above 291044 of 402118 here: 72% of periods throttled.
• cpu.stat throttled_usec — total microseconds spent stopped. 812 seconds in this sample.

Scoping host tools to one container

Two mechanisms, and they answer different questions. Get them the right way round and the whole toolkit becomes container-aware.

CONTAINER=api

# Handle 1: the host PID of the container's PID 1. Use for namespaces.
PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")

# Handle 2: the cgroup path, read from the process itself rather than guessed.
CGROUP=$(sed -n 's/^0:://p' /proc/"$PID"/cgroup)

echo "pid=$PID cgroup=$CGROUP"

Reading /proc/<pid>/cgroup rather than constructing the path is deliberate. The layout depends on the daemon’s cgroup-driver: with the systemd driver it is /system.slice/docker-<id>.scope, with cgroupfs it is /docker/<id>, and rootless Docker puts it under a user slice. The process knows; you do not have to.

By cgroup — for resource accounting

CONTAINER=api
PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")
CGROUP=$(sed -n 's/^0:://p' /proc/"$PID"/cgroup)
CGDIR="/sys/fs/cgroup$CGROUP"

# Per-container block I/O, which iostat cannot break out at all.
sudo cat "$CGDIR/io.stat"

# Memory broken down by kind. This is where you find page cache versus anon.
sudo grep -E '^(anon|file|slab|sock) ' "$CGDIR/memory.stat"

# Hardware counters for just this cgroup.
sudo perf stat --cgroup "$CGROUP" -e cycles,instructions,cache-misses -- sleep 10

# Live per-cgroup ranking, systemd hosts.
systemd-cgtop --order=cpu --iterations=3

perf stat --cgroup is the tool people do not know exists. It attaches the hardware counters to a cgroup rather than a process, so it follows every process in the container including ones that fork after you started measuring — which is exactly what a process-scoped perf misses on a worker pool.

io.stat is the only reliable per-container block I/O accounting. iostat reports per device, so on a host with fifteen containers sharing one NVMe it can tell you the device is saturated and nothing at all about who saturated it.

By namespace — for the container’s own view

CONTAINER=api
PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")

# Network namespace only: the container's sockets, with host tooling.
sudo nsenter -t "$PID" -n ss -tanp
sudo nsenter -t "$PID" -n ip -s link

# Mount + PID namespace: the container's filesystem and process tree.
sudo nsenter -t "$PID" -m -p ls -l /proc/1/exe

# Everything, for an interactive shell in a distroless image.
sudo nsenter -t "$PID" -m -u -i -n -p -- /bin/sh

This is the technique worth internalising, because it solves the problem that minimal images create. A distroless or scratch image has no ss, no ps, no shell — docker exec gives you nothing. nsenter runs the host’s binaries with the container’s namespaces attached, so you get the full sysadmin toolkit inside a container that contains one static binary and nothing else.

Note the flag selection. -n alone enters the network namespace but keeps the host’s mount namespace, so ss is the host’s ss reading the container’s sockets. That combination is usually what you want and it is much less disruptive than entering everything.

Which tool for which symptom

Verification that can fail

A performance check should end in an assertion, not a screenshot.

FAIL=0
for C in $(docker ps -q); do
NAME=$(docker inspect --format '{{.Name}}' "$C")
PID=$(docker inspect --format '{{.State.Pid}}' "$C")
CG="/sys/fs/cgroup$(sed -n 's/^0:://p' /proc/"$PID"/cgroup)"
[ -r "$CG/cpu.stat" ] || continue

PERIODS=$(awk '/^nr_periods/{print $2+0}' "$CG/cpu.stat")
THROT=$(awk '/^nr_throttled/{print $2+0}' "$CG/cpu.stat")
OOM=$(awk '/^oom_kill/{print $2+0}' "$CG/memory.events" 2>/dev/null || echo 0)

if [ "$PERIODS" -gt 100 ]; then
  PCT=$(( THROT * 100 / PERIODS ))
  if [ "$PCT" -gt 5 ]; then
    echo "THROTTLED $NAME at $PCT percent of periods"
    FAIL=1
  fi
fi
if [ "$OOM" -gt 0 ]; then
  echo "OOM-KILLED $NAME count=$OOM"
  FAIL=1
fi
done
[ "$FAIL" -eq 0 ] && echo 'PASS: no throttling above 5% and no OOM kills'
exit "$FAIL"

Five percent is a starting threshold, not a law. A batch job that bursts and gets clipped is fine; an interactive service throttled at 5% of periods is adding tail latency on one request in twenty.

Knowledge check