top, htop, and vmstat - the basic performance tools · Linux

top, htop, and vmstat are the first tools to reach for when investigating performance. This lesson covers what each shows and how to read the output.

top

top

Output (top section):

top - 14:30:00 up 30 days, 1 user, load average: 4.50, 4.20, 4.00
Tasks: 350 total,   2 running, 348 sleeping
%Cpu(s): 80.0 us,  5.0 sy,  0.0 ni,  5.0 id, 10.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  64000.0 total,  10000.0 free,  40000.0 used,  14000.0 buff/cache
MiB Swap:   8192.0 total,   1000.0 free,   7192.0 used.

Reading:

load average: 4.50, 4.20, 4.00. Three numbers: 1-min, 5-min, 15-min averages. If > number of CPUs, the system is overloaded.
%Cpu: 80% user (applications), 5% system (kernel), 10% iowait (waiting for I/O). High iowait = I/O bound.
Mem: 40 GB used of 64 GB. Plenty of headroom.
Swap: 7 GB used. Some swapping; could be a concern.

Press 1 to show per-CPU stats. Press P to sort by CPU. Press M for memory.

htop

htop

Interactive, colourful, easier to read. Same data as top but with:

Per-CPU bars.
Per-process tree view.
Easy scrolling, sorting, filtering (F4).
Mouse support.

vmstat

vmstat 1

Output:

procs ---------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs  us sy id wa st
 4  0      0 10000  5000 40000    0    0     5    20  500 1000  80  5  5 10  0
 4  0      0 10000  5000 40000    0    0     0     0  500 1000  82  4  4 10  0

Reading:

r (runnable): 4. Number of runnable processes. Compare to CPU count. If > CPUs, the system is overloaded.
b (blocked): 0. Processes blocked on I/O.
si/so: swap in / out. Both 0 = no swapping.
bi/bo: block I/O in / out per second.
in/cs: interrupts / context switches per second. High context switches may indicate lock contention.
us/sy/id/wa/st: user, system, idle, iowait, steal (CPU time).

Common patterns

Pattern	Meaning
High `%us`, low `%wa`	CPU bound by application code
High `%wa`	I/O bound (disk or network)
High `%sy`	Kernel overhead (drivers, syscalls)
High `r` (run queue)	CPU saturated
High `b` (blocked)	I/O saturated
High `si/so`	Memory pressure, swapping
High `in`	Hardware interrupts (driver issue?)

Drill down from USE

USE found the bottleneck. The tools above show the details:

CPU bottleneck: top → which processes. pidstat → per-CPU usage. perf → which functions.
Memory: free, /proc/meminfo. pmap for a process.
Disk: iostat, iotop, biosnoop.
Network: sar -n DEV / sar -n EDEV, ip -s link, tcpdump.