iostat and sar - storage and historical performance

iostat shows real-time disk I/O statistics. sar collects historical performance data over time. Together they give visibility into storage performance and trends.

iostat

# -x: extended statistics. Without it you get tps and kB/s only - none of
#     the await, aqu-sz or %util columns this lesson is about.
# -z: skip devices with no activity, so loop devices do not fill the screen.
#  1: repeat every second. The FIRST report is an average since boot and
#     tells you nothing about now; read the second and later ones.
iostat -xz 1

Output (columns elided — the real -x report is about 22 columns wide and also carries rrqm/s, %rrqm, rareq-sz, the discard d/* set and f/s):

Linux 6.8.0-51-generic (host)  2026-08-09  _x86_64_  (12 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          20.00    0.00    5.00    5.00    0.00   70.00

Device     r/s    rkB/s  r_await     w/s    wkB/s  w_await   aqu-sz  %util
sda       5.00   200.00     1.00   10.00   400.00     4.00     0.50   5.00
nvme0n1 100.00 10000.00     0.30   50.00  5000.00     0.50     1.50  30.00

Note that modern iostat reports r_await and w_await separately and has no combined await column. Read them apart: reads and writes have different service-time characteristics, and a write-back cache can hide write latency while reads queue.

Reading:

• r/s, w/s: read/write IOPS per second.
• rkB/s, wkB/s: read/write throughput.
• r_await, w_await: average time (ms) for a read, or a write, to complete — queue time included. These are the latency columns. Older iostat had a single combined await; current versions report only the split form, and older documentation that refers to “await” means these.
• aqu-sz: the time-averaged number of requests in flight. It has no fixed saturation threshold - not 1, not any number. Read it against the device’s queue capability and alongside await.
• %util: the percentage of elapsed time during which at least one I/O request was in flight. It measures busy, not full.

aqu-sz has no threshold

Older guidance - including an earlier version of this lesson - said aqu-sz > 1 means saturation. That is the same mistake as reading %util as a fixed number: it turns a capability-relative measurement into a constant.

aqu-sz follows Little’s Law, aqu-sz ≈ IOPS × await, so it rises when the workload submits more concurrent I/O or when the device gets slower. A SATA disk advertises NCQ depth 32; an NVMe controller has many hardware queues, each thousands deep. On such a device an aqu-sz of 8 or 32 can be perfectly normal concurrency with flat latency. Conversely aqu-sz of 0.3 with an await of 40 ms is a slow device, not a saturated one.

Saturation is the knee of the latency curve: offered load keeps rising, IOPS stop rising, and await climbs instead. The lesson linux-queue-depth-and-utilisation works through this in full.

Read %util the right way round

%util counts time, not work. A device with only one request outstanding for the whole interval reads 100% just as surely as one with a queue of 64.

On a device that serves requests serially - a single spinning disk - one request at a time is genuinely all it can do, so 100% does mean saturated. On a device that serves requests in parallel - NVMe, RAID arrays, SAN LUNs, cloud block volumes - 100% only means the device was never completely idle. An NVMe drive can sit at %util 100 while delivering 2% of the IOPS it is capable of.

So the widely repeated rule “on SSDs, %util reads low even at high IOPS” is backwards. Parallel devices read high, and early. man iostat says so directly:

%util  Percentage of elapsed time during which I/O requests were
       issued to the device (bandwidth utilization for the
       device). Device saturation occurs when this value is close
       to 100% for devices serving requests serially. But for
       devices serving requests in parallel, such as RAID arrays
       and modern SSDs, this number does not reflect their
       performance limits.

On a parallel device, ignore %util and judge saturation from await (and latency percentiles if you have them) against IOPS compared with the device’s known rated capability.

Identify the bottleneck

iostat -xz 1

-x shows extended stats; -z hides idle devices. Look for:

• High r_await / w_await (>10ms typical for spinning disks,

  1ms for SSDs). Current iostat reports the two directions separately and has no combined await column - read them apart, because a write-back cache can keep w_await low while reads queue behind it.

• aqu-sz high relative to that device’s queue capability, and rising while await rises with it. aqu-sz alone, against any constant, proves nothing.
• Errors in the device.

For NVMe, also check:

nvme smart-log /dev/nvme0n1

SMART data shows media errors, temperature, wear.

sar

# CPU history
sar -u 1 5         # 5 samples at 1-second interval

# Memory history
sar -r 1 5

# Disk history
sar -d 1 5

# Network history
sar -n DEV 1 5

# Load average history
sar -q 1 5

sar reads from /var/log/sa/ (or /var/log/sysstat/).

Configure sysstat data collection

# /etc/default/sysstat
ENABLED="true"

The sa1 and sa2 cron jobs collect data every 10 minutes. Historical data is kept for 28 days by default.

# View historical CPU usage
sar -u -f /var/log/sa/sa15 -s 14:00 -e 16:00

# Compare current to yesterday
sar -u -s 14:00 -e 16:00

Common patterns

Trends over time

sar lets you compare current vs. historical:

# What was the disk I/O at 14:00 yesterday?
sar -d -f /var/log/sa/sa15 -s 14:00 -e 15:00

# What is it today?
sar -d -s 14:00 -e 15:00

A sudden change (e.g. disk I/O doubled) is a signal of something new running or a problem.

Knowledge check