Queue depth and where latency accumulates

Why this matters in production

Queue depth is the setting that most changes a benchmark result and is most often left unstated. Understanding it prevents both bad benchmarks and bad conclusions from good ones.

The queues

flowchart TD
  A[Application queue] --> B[Client library / kernel]
  B --> C[Network]
  C --> D[OSD shard queue]
  D --> E[BlueStore]
  E --> F[Block layer queue]
  F --> G[Device queue]

The trade

low queue depth  → low latency per operation, low throughput
high queue depth → high throughput, higher latency per operation

# the same device, different depths
for qd in 1 4 16 64; do
  fio --name=qd$qd --filename=/dev/sdX --direct=1 --rw=randread \
      --bs=4k --iodepth=$qd --runtime=20 --time_based --readonly \
      --output-format=json | python3 -c '
import sys,json; d=json.load(sys.stdin)["jobs"][0]["read"]
print("qd", d["iodepth_level"] if "iodepth_level" in d else "",
      int(d["iops"]), round(d["lat_ns"]["mean"]/1e6,2), "ms")'
done

Typical HDD result:

qd1    150 IOPS   6.6 ms
qd4    170 IOPS  23.5 ms
qd16   180 IOPS  88.0 ms
qd64   182 IOPS 351.0 ms

Throughput barely improves past qd4 while latency rises linearly — the device is saturated and the queue is just accumulating.

NVMe behaves differently:

qd1    18k IOPS   0.055 ms
qd16  240k IOPS   0.067 ms
qd64  610k IOPS   0.105 ms

Here depth buys real throughput because the device has internal parallelism to use.

Choosing a depth

A benchmark at qd=1 and one at qd=256 measure different properties of the same hardware, and quoting one as “the performance” is the most common benchmarking error.

Recognising a queue depth problem

iostat -x 1 5

Quiz

Production discipline

State the queue depth with every latency and IOPS figure; the same device produces wildly different numbers at different depths and both are true. Benchmark at the application’s actual concurrency rather than at the depth that produces the best headline number.

Cross-course references

• Kubernetes: request concurrency changes latency and throughput identically
• Linux: nr_requests and scheduler depth present the same trade at the block layer