Utilisation is not saturation - what the queue tells you that the percentage cannot · Linux

A capacity ceiling expressed as a percentage carries a hidden assumption: that the resource performs the same at 80% as it did at 40%, only with less room left. That assumption is false for every resource that has a queue in front of it, which is all of them.

This is why “we still have 20% headroom” and “the service is slow” are routinely true at the same time, and why the engineer holding the dashboard and the engineer holding the pager cannot agree.

Three measurements, not one

The USE method names them, and the value of the naming is that it stops one number being asked to do three jobs.

Utilisation: the fraction of time the resource was busy. A percentage. Bounded at 100%.
Saturation: how much work is waiting because the resource was busy. A queue length or a wait time. Unbounded.
Errors: work the resource refused or dropped. A count.

Utilisation tells you how full the road is. Saturation tells you how long the traffic jam is. Only the second one correlates with what a user experiences, and only the second one keeps rising after utilisation has flattened at 100%.

Two hosts both reporting 100% CPU are not in the same state if one has a run queue of 2 and the other a run queue of 60. The percentage cannot tell them apart. The queue can.

The queueing multiplier

For a single server with random arrivals, the classic result is that the average time in the system - queueing plus service - grows as:

response time = service time / (1 - u)

where u is utilisation expressed as a fraction. The multiplier is worth memorising, because it explains almost every “it was fine yesterday” incident:

Utilisation	Multiplier `1 / (1 - u)`	10 ms of work takes
50%	2.0	20 ms
70%	3.3	33 ms
80%	5.0	50 ms
90%	10.0	100 ms
95%	20.0	200 ms
99%	100.0	1000 ms

Going from 50% to 70% costs you 13 ms. Going from 90% to 95% costs you 100 ms. The same twenty-point move costs eight times as much at the top of the range, which is the whole reason capacity ceilings are set well below 100%.

Measuring CPU saturation

Utilisation comes from %idle. Saturation comes from the run queue: the count of threads that are runnable but not running.

Read-only / Saferun queue against core count— runq-sz is the number of runnable threads waiting for a CPU. On this 12-core host a queue of 31 means roughly three threads competing for every core - so every thread spends about two thirds of its wall-clock time waiting. Utilisation would report 100% and stop there.

$ nproc; sar -q LOAD 1 3

12
03:41:02 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
03:41:03 PM        31       902     18.44      9.71      4.02         0
03:41:04 PM        27       903     18.51      9.88      4.09         0
03:41:05 PM        34       901     18.66     10.02      4.17         0
Average:           31       902     18.54      9.87      4.09         0

Illustrative output

The useful derived figure is runq-sz / nproc. Below 1 there is no meaningful contention. At 3, wall-clock time for CPU-bound work is roughly three times its CPU time, and no amount of profiling the application will change that.

The blocked column is the second saturation signal: threads in uninterruptible sleep, which almost always means waiting on I/O. A high blocked count with a low runq-sz says the CPU is not your problem at all.

Measuring disk saturation

%util is the field everyone reads and the one that lies hardest on modern hardware.

Read-only / Safeawait and queue depth, not %util— Columns abridged - iostat -xz prints many more. Both devices read 99% util and are in completely different states: the NVMe serves 6.8k writes at 6.9 ms, busy and healthy; the SATA disk serves 91 writes at 88 ms, saturated. %util cannot distinguish them; w_await and aqu-sz can.

$ iostat -xz 1 3 | tail -8

Device   r/s     w/s    rkB/s    wkB/s  r_await  w_await  aqu-sz  %util
nvme0n1 41.0  6820.0   1312.0 872960.0     0.28     6.91   47.62   99.90
sda      3.0    91.0     48.0   1108.0     1.11    88.40    8.13   99.20

Illustrative output

%util is the percentage of wall-clock time during which the device had at least one request in flight. For a single-queue rotational disk that was a good proxy for saturation, because one request in flight meant the one mechanism was occupied.

An NVMe device has many hardware queues and serves requests concurrently. It reaches 100% %util with one request outstanding while running at a fraction of its capability, and it stays at 100% all the way to genuine saturation. On such a device %util is a busy indicator with no capacity meaning whatsoever.

Do not set a capacity alert on disk %util

This is the single most common false alert in Linux monitoring. A fleet of NVMe-backed hosts will sit at 95-100% %util under perfectly ordinary load, and the team learns within a month that the alert means nothing. Then the one host that really is saturated fires the same alert and is ignored with everything else.

Alert on the saturation signals instead:

# latency now: the number a user can feel
iostat -xz 1 5

# await and queue depth over time, from the sysstat history
sar -d -p -f /var/log/sysstat/sa11 | grep -E 'DEV|nvme0n1|sda'

Thresholds worth starting from: r_await or w_await above ten times the device’s idle latency, or aqu-sz persistently above the device’s useful queue depth. Both are workload-specific, and both are numbers you take from your own baseline rather than from an article.

Pressure stall information

Run queue and await are per-resource proxies. Since kernel 4.20, /proc/pressure reports the thing you actually wanted: the percentage of wall-clock time during which work was delayed by a shortage of CPU, memory or I/O.

Read-only / Safedirect saturation measurement— some is the share of time at least one task was stalled; full is the share of time every task was stalled, which means the machine was doing no useful work at all. I/O some at 14.7% over ten seconds is a real finding; CPU and memory are clean.

$ cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory

some avg10=0.00 avg60=0.01 avg300=0.00 total=811140006
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
some avg10=14.72 avg60=11.08 avg300=7.41 total=94822013301
full avg10=9.30 avg60=7.02 avg300=4.66 total=52117884204
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

Illustrative output

Read it as follows.

some above a few percent means the resource is already costing you latency, whatever the utilisation percentage says.
full above zero for CPU or I/O means wall-clock time in which nothing progressed. Any sustained non-zero full is a capacity finding, not a tuning opportunity.
The avg10, avg60 and avg300 windows let you separate a momentary burst from a sustained condition without storing high-resolution samples.

sar -q PSI records the same counters into the sysstat history, so you can go back and ask what the pressure was during last Tuesday’s incident rather than only what it is now.

What this changes about the ceiling

The survivable ceiling from the N+1 arithmetic - N x u / (N-1) reaching 100% - assumes a node can serve at 100% utilisation. The queueing multiplier says it cannot serve acceptably there.

Take three nodes at the ceiling of 66.7%. Lose one, and the two survivors sit at 100% by the arithmetic. By the multiplier, a request that took 33 ms at 66.7% does not take 50 ms at 100% - it takes as long as the queue is deep, and the queue grows for as long as arrivals exceed service. The failover does not degrade the service; it stops it.

That is why every capacity lesson in this course says the same thing in a different way: the ceiling is where failure becomes outage, and the operating target sits well below it. Saturation metrics are how you find out where “well below” actually is for your workload, rather than guessing at a percentage.

Take the utilisation baseline: median, p95 and max per role
Take the saturation baseline alongside it: runq-sz/nproc, await, PSI some and full
Find the utilisation at which your saturation metric starts to climb - that is your knee, measured rather than assumed
Set the operating target below the knee, and the alert threshold on the saturation metric rather than on utilisation
Re-measure the knee after any change to hardware, kernel or workload shape

Three measurements, not one

The queueing multiplier

Measuring CPU saturation

Measuring disk saturation

Pressure stall information

What this changes about the ceiling

Knowledge check