Lab: USE investigation on a live host

This lab applies the USE methodology to a live host. By the end you will have a performance baseline and the discipline for performance investigations.

Tasks

Task 1: Apply USE to every resource

For each resource (CPU, memory, disk, network):

Resource	Utilisation	Saturation	Errors
CPU	top/mpstat	load average, run queue	mcelog, dmesg
Memory	free	page scan rate, swap	OOM events
Disk	iostat %util	await, aqu-sz	iostat errors
Network	ifutil	dropped packets	CRC errors

Record each value.

Task 2: Identify the bottleneck

If any utilisation > 80% or saturation > 0, that is the candidate bottleneck. Cross-check:

# CPU
mpstat 1 5

# Memory
free -h
vmstat 1 5

# Disk
iostat -x 1 5

# Network - sar ships with sysstat; nicstat is not packaged for
# Debian/Ubuntu/RHEL and will not be present unless you built it
sar -n DEV 1 5
sar -n EDEV 1 5

Task 3: Drill into the bottleneck

If CPU:

mpstat -P ALL 1 5    # per-CPU
pidstat 1 5          # per-process
top -c               # full command line

If memory:

pidstat -r 1 5
slabtop
cat /proc/<pid>/status

If disk:

# Substitute your own values before running:
DEVICE=/dev/sdb

iotop             # per-process I/O
biosnoop "$DEVICE" # per-I/O
pidstat -d 1 5

If network:

sar -n DEV 1 5                      # throughput per interface
sar -n TCP,ETCP 1 5                 # TCP actives/passives, retransmits
cat /sys/class/net/eth0/speed       # Mb/s - divide throughput by this for %util
sudo tcpdump -i eth0 -c 100
ss -tinp

Task 4: Capture the baseline

For the host, capture the baseline metrics:

# Run each tool and capture
top -bn 1 > /tmp/baseline-top.txt
mpstat 1 5 > /tmp/baseline-mpstat.txt
iostat -x 1 5 > /tmp/baseline-iostat.txt
free -h > /tmp/baseline-free.txt
ss -s > /tmp/baseline-ss.txt

Save for future comparison.

Task 5: Document

USE INVESTIGATION
=================
Host: <host>
Date: 2026-08-09
Reported symptom: "the application is slow"

USE checklist:
| Resource | Util | Saturation | Errors | Verdict |
|----------|------|------------|--------|---------|
| CPU      | 39%  | vmstat r=1 | 0      | see note|
| Memory   | 60%  | swap 0     | 0      | OK      |
| Disk     | 5%   | await 1ms  | 0      | OK      |
| Network  | 10%  | 0 drops    | 0      | OK      |

Note: load average is 4.5 on 4 CPUs, which looks like CPU
saturation and is not. vmstat shows r=1: only one task is
runnable. The rest of the load is D-state.

Drill-down:
- mpstat -P ALL: CPU 0 at 95%, the other three at 20%
  -> ~39% aggregate, not a saturated host
- vmstat 1: r=1, b=0
- pidstat: PID 1234 (java) at 90% CPU, single thread
- pidstat -t: one thread carries all of it
- top -c: full command "java -jar /opt/app/server.jar"

Bottleneck: one saturated CPU, not the host. No amount of
extra cores helps a workload that cannot use them.

Hypothesis: single-threaded hot path in the Java application
- GC on a single collector thread, or a serialisation lock.
Next step is a thread dump, not a bigger instance.

The point of this table is the note. Filling in “CPU 85%, load 4.5/4, BOTTLE” from a load average is the most common way a USE investigation goes wrong: load counts D-state tasks, so it is not a CPU measurement, and the drill-down here contradicts it outright. Make the utilisation column agree with mpstat, and put the run-queue length in the saturation column instead of the load average.

Objectives

Prerequisites