Queue depth and utilisation - storage saturation signals

Queue depth is the number of in-flight I/O operations. It is a measure of concurrency, not of health. A deep queue on a device built for parallelism is normal; the same depth on a single spinning disk is a latency problem. The number alone never tells you which.

Queue depth explained

When an application submits an I/O, the kernel queues it on the device’s request queue. The device’s driver processes the queue, dispatching requests to the device.

Queue depth:

• 1: only one I/O at a time. Low throughput, low latency.
• 32: typical default for SATA.
• 256+: high queue depth, used for high-throughput workloads (databases, batch jobs).

Higher queue depth = more requests in flight = potentially higher throughput. But also potentially higher latency (requests wait longer).

Read aqu-sz

iostat -x 1

Output:

Device  r/s  w/s  await  aqu-sz  %util
sda     5    10   2.50   0.50    5.00
nvme0n1 100 50  0.50   1.50   30.00

aqu-sz is the time-averaged number of requests in flight. It follows Little’s Law:

aqu-sz  ≈  IOPS  ×  await

So aqu-sz rises whenever either the workload submits more concurrent I/O or the device gets slower. It is one number covering two very different causes, which is exactly why a fixed threshold cannot interpret it.

Interpret aqu-sz against the device’s queue capability, and always alongside await:

• aqu-sz 20 on a single SATA disk (NCQ 32): the queue is nearly full. await will already be climbing. Real queueing.
• aqu-sz 20 on an NVMe with many hardware queues: ordinary. Check whether await moved at all before calling it anything.
• aqu-sz 0.3 with await of 40 ms: barely any concurrency and terrible latency. The device is slow, not saturated - a failing disk, a throttled cloud volume, or a noisy neighbour.

Saturation is the knee of the latency curve: the point where offered load keeps rising, IOPS and throughput stop rising, and await climbs instead. You find it by watching IOPS and await together over time, not by comparing aqu-sz with a constant.

Distinguish from utilisation

%util is the percentage of elapsed time during which at least one request was in flight. It measures busy, not full. aqu-sz is how many requests were in flight on average.

• High %util, low aqu-sz: one request at a time, and the device was rarely idle. On a serial device that is genuinely all it can do.
• Low %util, high aqu-sz: near-impossible by definition, because requests in flight are what make the device count as busy. Treat it as a sampling artefact, not a finding.
• High both: read await before concluding anything. On a serial device it is saturation; on a parallel device it is ordinary concurrent load.

The block layer: scheduler, nr_requests, read-ahead

Before you touch any knob, know which layer it belongs to. There are three distinct queues between an application and the platter or flash:

Which scheduler is active

# Brackets mark the current scheduler
cat /sys/block/nvme0n1/queue/scheduler    # [none] mq-deadline kyber bfq
cat /sys/block/sda/queue/scheduler        # none [mq-deadline] kyber bfq

# Is this device rotational?
cat /sys/block/sda/queue/rotational       # 1 = spinning disk, 0 = SSD/NVMe

All current kernels use the multi-queue block layer (blk-mq). The single-queue schedulers cfq and deadline are gone. What you will see is:

• none - no reordering at all. The right answer for NVMe and for anything behind a capable hardware RAID controller or a cloud hypervisor. Scheduling costs CPU and adds latency, and the device already reorders internally.
• mq-deadline - deadline-based, with a read preference and an expiry for starved requests. The sensible default for SATA SSDs and spinning disks, and the usual choice under a database.
• kyber - targets a latency budget by throttling submission. Useful on fast devices with mixed read/write.
• bfq - fair-share proportional scheduling per process. Good for interactive desktops and for isolating a noisy tenant; its overhead and its latency variance make it a poor fit under a throughput-sensitive database.

Read-ahead

cat /sys/block/sda/read_ahead_kb          # default 128

Read-ahead is how far the kernel reads past a detected sequential pattern. 128 KB is a compromise. A large sequential scan - a backup read, an analytics table scan - benefits from raising it; a random-access OLTP workload gains nothing and wastes page cache and bandwidth on data it will never read. Measure with iostat -x before and after: sequential gains show up as higher rkB/s at the same or lower await.

Persist the choice with udev, not a boot-time echo

A value written to /sys is lost at reboot, and a device added later never gets it. Set it as a device property instead:

# /etc/udev/rules.d/60-ioscheduler.rules
ACTION=="add|change", KERNEL=="nvme[0-9]n[0-9]", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="mq-deadline", ATTR{queue/read_ahead_kb}="1024"

Reload with sudo udevadm control --reload and re-trigger with sudo udevadm trigger --subsystem-match=block.

nr_requests is not the device queue depth

# View current
cat /sys/block/sda/queue/nr_requests

# Change it (runtime only; use udev to persist)
echo 256 | sudo tee /sys/block/sda/queue/nr_requests

Detect saturation

iostat -x 1 5

Saturation is a trend across samples, never a single reading. Look for all three together:

• aqu-sz rising, and
• await rising with it, and
• IOPS (r/s + w/s) flat or falling while the first two rise.

That combination is the knee: more work is being offered, none of it is completing any faster, so it queues. If aqu-sz rises and await stays flat, the workload simply became more concurrent and the device absorbed it - that is capacity being used, not capacity being exhausted.

Common patterns

Knowledge check