CPU utilisation and load average - measuring the workload · Linux

CPU utilisation and load average are the two most common CPU metrics. Understanding what they mean and do not mean is essential.

CPU utilisation

top -bn 1 | head -5

Output:

top - 14:30:00 up 30 days, 1 user, load average: 4.50, 4.20, 4.00
Tasks: 350 total,   2 running, 348 sleeping
%Cpu(s): 80.0 us,  5.0 sy,  0.0 ni,  5.0 id, 10.0 wa,  0.0 hi,  0.0 si,  0.0 st

The percentages:

%us (user): time spent in application code. Most CPU work shows here.
%sy (system): time in kernel code. Drivers, syscalls, scheduling.
%ni (nice): time spent on niced (low priority) processes.
%id (idle): time with nothing to do. Healthy system.
%wa (iowait): time waiting for I/O. Indicates storage bottleneck.
%hi / %si (hardware / software interrupts): time in interrupt handlers.
%st (steal): time stolen by hypervisor (VM only). High steal = hypervisor is overcommitted.

Interpret percentages

%us high, %wa low      : CPU bound by application
%wa high, %us low      : I/O bound; the CPU is idle waiting
%sy high               : Kernel overhead (drivers, syscalls)
%hi + %si high         : Network or storage interrupts
%st high               : Hypervisor overcommitted (VM)
100% across all        : CPU saturated

A healthy system has a mix of %us and %id, with low %wa and %st.

Load average

Load average is the running average of runnable + uninterruptible processes, displayed as three numbers: 1-minute, 5-minute, 15-minute.

uptime
14:30:00 up 30 days, 1 user, load average: 4.50, 4.20, 4.00

Interpretation:

Number of CPUs: nproc.
Load < CPUs: little is waiting for anything.
Load = CPUs: as many tasks want to run or are blocked as you have cores.
Load > CPUs: tasks are waiting - but read the next paragraph before concluding they are waiting for CPU.

Load average includes uninterruptible sleep (D state), which includes processes waiting for I/O. So a load of 10 on a 4-CPU host might mean 4 running + 6 waiting for I/O - the I/O is the bottleneck, not the CPU.

That is why the comparison above is a starting point and not a verdict. To decide whether the CPU is actually saturated, use the run queue and pressure signals instead:

nproc
cat /proc/loadavg              # R + D tasks, not CPU utilisation
cat /proc/pressure/cpu         # "some avg10" - real CPU saturation
vmstat 1 5                     # r = runnable, b = blocked in D state
ps -eo state,comm | awk '$1 ~ /D/' | head   # who is in D state

A load of 10 with r near 10 is CPU saturation. The same load with r near 1 and b near 9 is storage, and adding CPUs will not move it.

Load average trends

Watch the trend:

1-min rising, 5-min rising, 15-min steady: spike. Will pass.
All three rising: sustained load. Investigate.
All three falling: load is decreasing. The problem has passed.
15-min high, 1-min low: was high, now low. Resolved.

Common pitfalls

100% CPU is not always bad: a CPU-bound application using 100% CPU is doing what it should.
Load average is not a CPU metric: it counts tasks that are runnable (R) plus tasks in uninterruptible sleep (D), and the latter are usually blocked on I/O and burning no CPU at all. “Load > CPUs means the CPU is saturated” is the inference the D-state rule forbids. High load with low %us and high %wa is a storage problem wearing a CPU costume.
0% CPU is not always good: idle CPU with high latency means the workload is blocked on something else (I/O, lock).
Load average on VMs includes steal time: a high load on a VM may be hypervisor overcommit, not the workload.