Lab: CPU saturation investigation

This lab applies USE to CPU on a live host. By the end you will have the discipline for CPU saturation diagnosis.

Tasks

Task 1: Capture baseline

nproc                                 # number of CPUs
uptime                                # load average
mpstat 1 5 > /tmp/baseline-mpstat.txt
mpstat -P ALL 1 5                     # per-CPU, including %steal

Load average only means something next to nproc. A load average of 8 is idle on a 32-core host and a four-times overload on a 2-core one. Write both numbers down now:

echo "nproc=$(nproc) load=$(cut -d' ' -f1-3 /proc/loadavg)"

On a VM, also record %steal from mpstat -P ALL. Steal is time the hypervisor gave to somebody else. A guest at 40% steal is not slow because of anything happening inside the guest, and no amount of in-guest tuning will fix it.

Task 2: Apply stress, one clean signal at a time

Each scenario must produce one signal. A command that mixes CPU workers with I/O workers produces a reading you cannot attribute to either.

# Install stress-ng
sudo apt install stress-ng

# Scenario A - single-thread bottleneck
stress-ng --cpu 1 --timeout 60s

# Scenario B - all CPUs saturated
stress-ng --cpu $(nproc) --timeout 60s

# Scenario C - I/O bound: no CPU workers, real disk I/O, bounded and scoped
mkdir -p /var/tmp/stress-io
stress-ng --hdd 2 --hdd-bytes 512M --temp-path /var/tmp/stress-io --timeout 60s
rm -rf /var/tmp/stress-io          # cleanup

# Scenario D - lock contention
stress-ng --futex 4 --timeout 60s

--io does not generate I/O, and --cpu drowns it out anyway

A natural-looking third scenario is stress-ng --cpu $(nproc) --io 4. It cannot demonstrate an I/O-bound system, for two independent reasons.

--io workers call sync(2) in a loop. They flush dirty pages; they do not generate a sustained read or write workload, so they produce little or no iowait. And even if they did, --cpu $(nproc) on the same command line puts one busy worker on every core, driving %us to roughly 100%. The reading you get is a CPU-bound system, which is the opposite of what the scenario is meant to show.

Use --hdd with --hdd-bytes and --temp-path instead: real file writes, a bounded size, and confined to a directory you chose. The sync(2) loop is also antisocial on a shared host — it repeatedly flushes the whole page cache, including everyone else’s.

Task 3: Observe during stress (in another terminal)

mpstat -P ALL 1 5
# Look for: %us, %wa, %sys, %steal, per-CPU spread

pidstat 1 5
# Look for: which process, %CPU, threads

pidstat -w 1 5
# Look for: cswch/s (voluntary) vs nvcswch/s (involuntary)

vmstat 1 5
# Look for: r (run queue), b (blocked), cs (context switches)

iostat -x 1 5
# Look for: await, %util on the device under test

Task 4: Document the patterns

Scenario A - single-thread stress:
- mpstat -P ALL: 1 CPU at ~100% %us, others idle
- pidstat: stress-ng-cpu at ~100% of one CPU
- vmstat: r=1, b=0, no wait

Scenario B - all-CPU stress:
- mpstat -P ALL: every CPU at ~100% %us
- pidstat: stress-ng-cpu processes spread across CPUs
- vmstat: r >= NCPUs, b=0, no wait

Scenario C - disk I/O stress (--hdd):
- mpstat -P ALL: %wa elevated, %us low (CPUs idle waiting on disk)
- pidstat: stress-ng-hdd processes, modest %CPU
- vmstat: b (blocked) > 0
- iostat -x: await elevated, %util near 100 on the target device

Scenario D - lock contention (--futex):
- mpstat -P ALL: %sys elevated, %us low
- pidstat -w: cswch/s very high (voluntary - threads blocking on the lock)
- vmstat: cs (context switches) very high, r low relative to the
  number of runnable threads

Record the actual numbers you measured next to each line. Where your reading disagrees with the expected pattern, that disagreement is the finding — investigate it rather than overwriting it.

Task 5: Recognise each pattern

For each stress scenario, identify:

CPU bound: high %us, low %wa, r > CPUs. (Scenario B)
I/O bound: high %wa, low %us, b > 0, iostat await elevated. (Scenario C)
Single-thread bottleneck: 1 CPU at 100%, others idle, load average pinned near 1 regardless of nproc. (Scenario A)
Multi-thread efficient: CPUs all near saturation, balanced. (Scenario B)
Lock contention: high cswch/s, high %sys, low %us. (Scenario D)
Stolen CPU: %steal high, %us and %wa both low, and the guest feels slow with no in-guest cause. Not reproducible from inside the guest — check it on every VM-hosted investigation.

The CPU-bound versus I/O-bound branch is the most consequential decision in a real slow-server incident, because the two lead to completely different remediations: more cores or a faster algorithm on one side, faster storage or fewer synchronous writes on the other. Scenarios B and C are the two halves of that branch, which is why they must be run separately.

Objectives

Prerequisites