Slow ops originating at an OSD

Why this matters in production

An OSD holding operations may be waiting on something else or may be the delay itself. The flag point distinguishes them and determines everything that follows.

Confirming the OSD is the source

ceph daemon osd.12 dump_ops_in_flight | python3 -c '
import sys,json
d = json.load(sys.stdin)
from collections import Counter
c = Counter(op.get("type_data",{}).get("flag_point") for op in d.get("ops",[]))
for k, v in c.most_common(): print("%4d  %s" % (v, k))'

Only the last three implicate this OSD.

The OSD-internal causes

ceph daemon osd.12 perf dump | python3 -c '
import sys,json
d = json.load(sys.stdin)
o, b = d["osd"], d.get("bluestore", {})
for name, src in (("op_latency", o), ("op_w_process_latency", o),
                  ("kv_sync_lat", b), ("kv_commit_lat", b),
                  ("state_kv_queued_lat", b), ("submit_lat", b)):
    v = src.get(name, {})
    if v.get("avgcount"):
        print("%-24s %8.2f ms" % (name, v["sum"]/v["avgcount"]*1000))'

ceph osd df | awk -v o=12 'NR==1 || $1==o'
ceph pg ls-by-osd 12 | grep -c scrub

Gathering the evidence

# host side, on the OSD's host
ceph osd find 12
iostat -x 1 5
vmstat 1 5
free -g

# the fsid names the systemd unit
FSID=$(ceph fsid)
journalctl -u "ceph-$FSID@osd.12" --since '30 min ago' | grep -iE 'slow|stall|compact'

The daemon log is frequently decisive: BlueStore logs compaction events and RocksDB stalls explicitly.

The remedies

Quiz

Production discipline

Read the flag point distribution before investigating an OSD — only queued for pg, started, and the local commit stages implicate it, and waiting for subops names a different OSD entirely. Check ceph osd df for PG count before concluding hardware; an over-assigned OSD produces identical symptoms.

Cross-course references

• Kubernetes: a pod slow because it waits on a dependency versus one slow itself
• Linux: distinguishing local I/O latency from remote wait in a distributed system