Isolating the slow layer

Why this matters in production

Each layer’s test must exclude the layers above it, or the result is a combined measurement that proves nothing about any single layer.

Device layer

# on the OSD host, against the raw device — NOT the OSD's filesystem
fio --name=devtest --filename=/dev/sdX --direct=1 --rw=randread \
    --bs=4k --iodepth=1 --runtime=30 --time_based --readonly

--readonly and a read workload are essential: writing to a device an OSD is using destroys data.

Above the range for its class, the device is the problem and nothing above it needs investigating yet.

Network layer

HOST=stor-04
iperf3 -s                        # on one host
iperf3 -c ${HOST} -t 30 -P 4      # on the other

ping -c 1000 -i 0.01 ${HOST} | tail -3
ping -M do -s 8972 -c 5 ${HOST}   # MTU verification for a 9000 MTU

# loss and errors on the interface
IFACE=bond0
ip -s link show ${IFACE}
ethtool -S ${IFACE} | grep -iE 'drop|err|discard'

Any non-zero loss on a cluster network is significant; Ceph’s behaviour degrades sharply with even fractional-percent loss.

OSD layer

ceph osd perf | sort -k2 -n | tail -10
ceph daemon osd.12 perf dump | python3 -c '
import sys,json; d=json.load(sys.stdin)["osd"]
for k in ("op_latency","op_w_latency","op_r_latency","subop_latency"):
    v=d.get(k,{})
    if v.get("avgcount"): print(k, round(v["sum"]/v["avgcount"]*1000,2), "ms")'

Comparing the OSD’s reported latency against the device’s measured latency is the key step: a large gap is queueing or processing inside the OSD.

RADOS layer

rados bench -p bench 30 write --no-cleanup
rados bench -p bench 30 rand
rados -p bench cleanup

This includes the network and the OSDs but excludes RBD and the client library.

Client layer

rbd perf image iostat --pool rbd-vms
fio --name=clienttest --ioengine=rbd --pool=rbd-vms --rbdname=bench \
    --rw=randread --bs=4k --iodepth=1 --runtime=30 --time_based

Working bottom-up

device OK → network OK → OSD OK → RADOS OK → client is the problem
device OK → network OK → OSD slow → the OSD's queue or threading
device OK → network lossy → the network
device slow → the device

Stopping at the first layer that fails avoids the common error of tuning the OSD when the device is at 40 ms.

Quiz

Production discipline

Test each layer in a way that excludes the layers above it and work bottom-up, stopping at the first failure — a combined measurement proves nothing about any single layer. Compare ceph osd perf against raw device latency early; the gap decides whether to tune the OSD or replace the device.

Cross-course references

• Kubernetes: isolating node, network, and container layers uses the same exclusion principle
• Linux: measuring the block layer separately from the filesystem is the same technique