Client-side queueing and threading

Why this matters in production

At high concurrency the client itself becomes the bottleneck, and the symptom — high client CPU with the cluster idle — is distinctive once recognised.

The client-side path

flowchart TD
  A[Application I/O] --> B[librbd: striping to objects]
  B --> C[librbd cache]
  C --> D[librados objecter]
  D --> E[messenger threads]
  E --> F[TCP connections to OSDs]

The settings

rbd config global ls | grep -E 'op_threads|concurrent'
ceph config get client ms_async_op_threads
ceph config get client objecter_inflight_ops
ceph config get client objecter_inflight_op_bytes

The objecter limits are throttles: reaching them blocks new operations until outstanding ones complete, which caps concurrency regardless of what the application requests.

Recognising a client-side bottleneck

# on the client
top -H -p $(pidof -s qemu-system-x86_64)
mpstat -P ALL 1 5

# on the cluster
ceph osd perf | sort -k2 -rn | head -3

# objecter state on a running client
ceph --admin-daemon /var/run/ceph/ceph-client.*.asok perf dump objecter | \
  python3 -c '
import sys,json; d=json.load(sys.stdin)["objecter"]
for k in ("op_active","op_laggy","osdop_read","osdop_write"):
    print(k, d.get(k))'

op_active sitting at the objecter_inflight_ops value means the throttle is binding.

Tuning

ceph config set client objecter_inflight_ops 4096
ceph config set client objecter_inflight_op_bytes 1073741824
ceph config set client ms_async_op_threads 5
rbd config image set rbd-vms/busy-image rbd_op_threads 8

Every increase costs client CPU and memory. The balance point is where client CPU is comfortably used but not saturated:

mpstat -P ALL 1 10 | tail -3

Quiz

Production discipline

Run top -H on the client and check OSD utilisation together before tuning anything — client CPU saturated with an idle cluster and the reverse call for opposite fixes. When neither is saturated and throughput is flat, check op_active against objecter_inflight_ops; the throttle is likely binding.

Cross-course references

• Kubernetes: client-side connection pool limits cap throughput identically
• Linux: application thread pool sizing bounds what any fast backend can deliver