Queue depth: the parameter that decides RBD throughput

Why this matters in production

Ceph’s per-operation latency is what it is: two network round trips plus a device commit. You cannot tune it below that. Throughput, however, scales with how many operations are in flight — so the single most important performance question about an RBD workload is its queue depth, and it is the one least often asked.

The arithmetic

IOPS = queue_depth / latency

Latency stays roughly flat until the cluster approaches saturation, at which point queueing begins and it rises. Until then, concurrency is free throughput.

The consequence for single-threaded workloads

A single-threaded synchronous writer has queue depth 1, permanently. At 1 ms latency it achieves 1,000 IOPS and no amount of cluster tuning changes that — the cluster is idle between its operations.

# queue depth 1: latency-bound
fio --name=qd1 --ioengine=rbd --pool=rbd-vms --rbdname=test \
    --rw=randwrite --bs=4k --iodepth=1 --runtime=60 --time_based

# queue depth 64: concurrency-bound
fio --name=qd64 --ioengine=rbd --pool=rbd-vms --rbdname=test \
    --rw=randwrite --bs=4k --iodepth=64 --runtime=60 --time_based

The second will show far higher IOPS at similar per-operation latency. That gap is the entire performance story for most RBD workloads.

Measuring what you have

# in the guest — avgqu-sz is the effective queue depth
iostat -x 5

# at the image level
rbd perf image iostat --pool rbd-vms

An avgqu-sz near 1 means the workload is serialised and Ceph tuning will not help it. A value in the tens means concurrency is available and cluster capacity is what matters.

Configuring it

# kernel RBD: the block device queue
cat /sys/block/rbd0/queue/nr_requests
echo 512 > /sys/block/rbd0/queue/nr_requests

# librbd: in-flight operation limits
ceph config set client rbd_concurrent_management_ops 20

# QEMU: use multiple virtio queues
# <driver name='qemu' queues='8'/>

But raising limits does nothing if the application does not generate concurrency. The lever is usually in the application: more threads, async I/O, batching, or removing an unnecessary fsync.

Quiz

Production discipline

Measure the workload’s queue depth before investigating the cluster for any RBD performance complaint; it distinguishes the two most common causes in one command and determines who owns the fix. Plan capacity against a utilisation ceiling rather than peak benchmark throughput, and say so when quoting benchmark figures.

Cross-course references

• Kubernetes: application concurrency determines throughput on PVC-backed storage identically
• Linux: Little’s Law applies to every queueing system, and this is a direct application of it