I/O latency, throughput, and IOPS - the three storage metrics

Storage performance is measured by three primary metrics: latency, throughput, and IOPS. Each tells a different story.

The three metrics

Latency: time for a single I/O to complete. Includes queue time + service time.

Throughput: bytes per second. For large I/Os (sequential reads, large writes), throughput is the bottleneck.

IOPS: operations per second. For small I/Os (random database reads, metadata), IOPS is the bottleneck.

Latency

iostat -x 1

Output:

Device  r/s  w/s  rkB/s  wkB/s  await  r_await  w_await  aqu-sz
sda     5    10   200    400    2.50   1.00     4.00    0.50

await is the average latency (ms) for I/O completion. Includes queue time.

• Spinning disk: <10ms is healthy, >50ms is a problem.
• SSD/NVMe: <1ms is healthy, >5ms is a problem.

Throughput

# -x for the per-direction columns, -m to report in MB/s, -z to skip
# idle devices. `iostat -m 1` alone gives only tps and MB_read/s - no
# r/s, no w/s, and none of the latency columns.
iostat -xzm 1

Output (columns elided):

Device     r/s    rMB/s  r_await     w/s    wMB/s  w_await  aqu-sz  %util
sda       5.00     0.20     1.00   10.00     0.40     4.00    0.50    5.00

The unit flag renames the columns: -m gives rMB/s/wMB/s, and the default (no flag) gives rkB/s/wkB/s. Total throughput is the sum of the two directions.

The three numbers are tied together by one identity:

throughput = IOPS x block size

Work it through. A 1 GB sequential read at 1000 MB/s takes about a second and needs roughly 1000 operations of 1 MiB each. The same 1 GB moved in 4 KiB random I/Os at 250,000 IOPS is 250,000 x 4 KiB = 1,024,000 KiB/s, which is also about 1000 MB/s - and 1 GiB / 4 KiB is 262,144 operations, so it too finishes in about a second.

Same throughput, same elapsed time, 250 times as many operations. That is why the two workloads stress different limits: the random workload runs into the device’s IOPS ceiling, the sequential one runs into its bandwidth ceiling.

IOPS

The same iostat output shows IOPS as r/s + w/s (5 + 10 = 15 in the example).

For random 4 KB reads:

• HDD: 100-200 IOPS.
• SSD: 10,000-100,000 IOPS.
• NVMe: 100,000-1,000,000+ IOPS.

The trade-off

Latency, throughput, and IOPS are related:

• Higher IOPS often = higher latency (queue depth).
• At a fixed bandwidth, larger transfers mean fewer operations per second. Block size and IOPS trade against each other for the same MB/s.
• Sequential workloads use large blocks, so they are usually bandwidth-limited. Random small-block workloads use tiny blocks, so they are usually IOPS-limited.

A device rated at both 1000 MB/s and 200,000 IOPS will deliver 1000 MB/s on 1 MiB sequential reads (1000 ops/s, well under the IOPS cap) and about 800 MB/s on 4 KiB random reads (200,000 ops/s, right at the IOPS cap). Neither number is wrong; the block size decides which one binds.

The right metric for the workload:

• OLTP database: IOPS and latency (small random I/Os).
• Backup / streaming: throughput (large sequential I/Os).
• Web server: latency (small random reads).
• Log server: throughput and IOPS (sequential writes, many small).

Read iostat output

Device  r/s  w/s  rkB/s  wkB/s  await  aqu-sz  %util
sda     5    10   200    400    2.50   0.50   5.00

• r/s, w/s: IOPS for read and write.
• rkB/s, wkB/s: throughput.
• await: average latency.
• aqu-sz: average number of requests in flight. This is concurrency, not a verdict - there is no fixed threshold, and “greater than 1 means saturated” is wrong on any device built for parallelism.
• %util: the fraction of elapsed time during which at least one request was in flight. It measures busy, not full.

Knowledge check