Concurrency and where throughput comes from

Why this matters in production

A cluster capable of 200,000 IOPS delivering 8,000 to an application is usually not a cluster problem. Concurrency on the client side is the most common explanation and the least often checked.

The relationship

throughput = concurrency / latency

one client, qd=1, 0.5 ms latency  →   2,000 IOPS
one client, qd=32, 0.5 ms latency →  64,000 IOPS
20 clients, qd=32, 0.5 ms latency → 1,280,000 IOPS (if the cluster can)

The cluster’s capability is a ceiling; the client’s concurrency determines how much of it is reached.

Where concurrency is limited

# what the client is achieving
rbd perf image iostat --pool rbd-vms

# what the cluster is doing overall
ceph osd pool stats rbd-vms

A cluster at 5% device utilisation while an application reports poor performance is a concurrency problem essentially every time.

iostat -x 1 5     # on the OSD hosts

Raising concurrency

# client-side threads
rbd config image set rbd-vms/vm-disk-1 rbd_op_threads 8

# more images rather than one large one
rbd create --size 500G rbd-vms/data-{1..4}
# stripe the application across them

# in the application or benchmark
fio --iodepth=64 --numjobs=4 ...

Multiple images is frequently the most effective change: each image has its own client-side state and its own path through the cluster, so four images at qd=32 behave very differently from one image at qd=128.

When more concurrency will not help

# the cluster is already saturated
ceph osd perf | sort -k2 -rn | head -5
iostat -x 1 5   # on OSD hosts: %util near 100

# latency rising with no throughput gain
for qd in 8 16 32 64 128; do
  fio --name=q$qd --ioengine=rbd --pool=rbd-vms --rbdname=bench \
      --rw=randread --bs=4k --iodepth=$qd --runtime=20 --time_based \
      --output-format=json | python3 -c '
import sys,json; d=json.load(sys.stdin)["jobs"][0]["read"]
print(int(d["iops"]), round(d["lat_ns"]["mean"]/1e6,2))'
done

When IOPS stops rising and latency rises proportionally, the saturation point has been passed and more concurrency only adds queueing.

Quiz

Production discipline

Check OSD device utilisation before investigating cluster-side causes of poor application throughput — single-digit utilisation means the constraint is client concurrency and no cluster tuning will help. Shard across several images rather than using one large one; each gives an independent client-side path.

Cross-course references

• Kubernetes: a single-replica workload cannot use a cluster sized for many
• Linux: single-threaded I/O against a fast device is the classic underutilisation case