Queue depth and block size — the two knobs that set the cost

Why this matters in production

Two benchmarks of the same cluster can differ by a factor of fifty and both be correct. The difference is nearly always queue depth and block size, and an operator who cannot reason about them cannot tell a meaningful measurement from a marketing one — or design a test that resembles the workload they actually run.

Queue depth: how much is in flight

Queue depth is the number of I/O requests submitted but not yet completed. It is a property of the client, not the device.

• Queue depth 1 is a synchronous workload: submit, wait, submit. A database committing a transaction, fsync() on a log file, a single-threaded copy. Latency is the entire story here — the device is idle between requests.
• Queue depth 32 or 128 is a concurrent workload: many threads or an async engine keeping the device busy. Throughput and IOPS look much better; individual operations take longer.

The device’s own internal parallelism decides how much this helps. A spinning disk has one head and gains little beyond a small depth. A SATA SSD has a queue of 32. An NVMe drive has many queues of great depth and scales until something else is the bottleneck.

Block size: how much per request

Block size is how many bytes one request carries. Small blocks make the per-operation overhead dominant; large blocks amortise it.

4 KiB    database pages, metadata, small random I/O
64 KiB   filesystem readahead, mixed workloads
1-4 MiB  backups, recovery, sequential streaming
4 MiB    Ceph's default RADOS object size for RBD

The last line matters. RBD stripes an image into 4 MiB objects by default, so a 4 KiB write from a VM becomes a partial write to one 4 MiB RADOS object, and a 16 MiB sequential write touches four objects that may live on entirely different OSDs — which is exactly how Ceph gets parallelism on large transfers.

Where Ceph exposes these

# RBD stripe geometry, set at image creation
rbd create pool/image --size 100G --object-size 4M \
  --stripe-unit 4M --stripe-count 1

rbd info pool/image        # shows object size and striping

For most workloads the defaults are right. Smaller objects spread a single image across more PGs, which raises parallelism and also raises the number of objects the cluster tracks. Larger objects reduce metadata and concentrate a client’s I/O on fewer OSDs.

On the OSD side, the queue is governed by settings such as osd_op_num_threads_per_shard and osd_op_num_shards, and by the recovery throttles that decide how much of the device’s queue recovery is allowed to occupy. Those are covered in the recovery tuning part; the relevant point here is that recovery and client I/O compete for the same finite queue.

Reading the two together

The useful mental model is a two-by-two:

Each quadrant has a different bottleneck and a different fix. Knowing which quadrant a workload sits in tells you whether to look at device latency, at parallelism, or at network bandwidth.

Quiz

Production discipline

Record block size and queue depth with every performance number you produce or receive; a figure without them is not comparable to anything. Benchmark the quadrant the workload actually occupies, not the one the hardware is best at. And carry the erasure-coding consequence into design conversations: partial-stripe writes are a read-modify-write across k+m OSDs, which makes EC excellent for large sequential objects and unsuitable for small random writes, whatever a 4 MiB benchmark says.

Cross-course references

• Linux: Part XLI (Disk Performance) for fio and the block layer queue.
• Ceph: Part XXV (Erasure Coding Fundamentals) develops the stripe-write cost in detail.
• Ceph: Part LX (Recovery Tuning) covers the queue that recovery and clients share.