Slow ops originating at the client

Why this matters in production

A substantial share of “Ceph is slow” reports originate above Ceph. The cluster’s own metrics look healthy, which makes the investigation frustrating until the client is checked.

What the cluster sees

A slow client produces no SLOW_OPS at all: operations arrive at the OSD and complete normally. The cluster is healthy because it is.

ceph -s
ceph osd perf | sort -k2 -rn | head -3
ceph health detail

All clean, while the client reports poor performance. That combination is the signal.

Establishing the client is the source

# from a second, independent client host
fio --name=probe --ioengine=rbd --pool=rbd-vms --rbdname=probe \
    --rw=randread --bs=4k --iodepth=16 --runtime=60 --time_based

One test, ten minutes, and half the possibilities are eliminated.

The client-side causes

# librbd client state
ceph --admin-daemon /var/run/ceph/ceph-client.*.asok perf dump objecter | \
  python3 -c '
import sys,json; d=json.load(sys.stdin)["objecter"]
print("op_active:", d.get("op_active"), "op_laggy:", d.get("op_laggy"))'

ceph config get client objecter_inflight_ops

op_active sitting at the configured limit means the client is throttling itself.

The definitive distinction

# 1. cluster-side latency, from the cluster's own view
ceph osd perf | awk 'NR>1 {s+=$2; n++} END {print "avg", s/n, "ms"}'

# 2. client-observed latency
fio ... --percentile_list=50:99

# 3. the gap

A large gap between the two is time spent above the OSDs — in the client, its queueing, or the network. A small gap means the cluster is the delay.

# and check the client is not simply idle
mpstat -P ALL 1 5
iostat -x 1 5      # inside the guest, for RBD

Quiz

Production discipline

Run the same test from a second client host before investigating the cluster — a healthy cluster with a slow client is the signature of a client-side cause, and the second test confirms it in one step. Check CPU steal explicitly on virtualised clients; it produces storage-like symptoms and appears nowhere in storage metrics.

Cross-course references

• Kubernetes: a slow pod on a contended node presents as a slow backend
• Linux: application-side scheduling delay is routinely misattributed to I/O