Finding the slow OSD

Why this matters in production

One slow OSD out of a hundred produces a cluster-wide tail latency problem. Finding it takes one command; confirming it takes two more.

Ranking by latency

ceph osd perf

osd  commit_latency(ms)  apply_latency(ms)
 44                  87                 87
 12                   3                  3
 31                   2                  2

ceph osd perf | sort -k2 -rn | head -10

With BlueStore these are effectively the same figure, because a write is durable and visible at the same point — the separate columns are a legacy of FileStore, where the journal made them genuinely distinct.

Deeper per-OSD detail

ceph daemon osd.44 perf dump | python3 -c '
import sys,json
d = json.load(sys.stdin)["osd"]
for k in ("op_latency","op_r_latency","op_w_latency","op_w_process_latency",
          "subop_latency","op_before_queue_op_lat","op_before_dequeue_op_lat"):
    v = d.get(k, {})
    if v.get("avgcount"):
        print("%-28s %8.2f ms  (n=%d)" % (k, v["sum"]/v["avgcount"]*1000, v["avgcount"]))'

# BlueStore internals
ceph daemon osd.44 perf dump bluestore | python3 -c '
import sys,json
d = json.load(sys.stdin)["bluestore"]
for k in ("kv_sync_lat","kv_commit_lat","state_kv_queued_lat","submit_lat"):
    v = d.get(k, {})
    if v.get("avgcount"):
        print("%-24s %8.2f ms" % (k, v["sum"]/v["avgcount"]*1000))'

kv_sync_lat rising specifically points at the DB device rather than the data device, which narrows a replacement decision considerably.

Confirming the outlier is real

# 1. is it consistent, or a single sample?
for i in 1 2 3; do ceph osd perf | awk '$1==44'; sleep 30; done

# 2. does the device agree?
ceph osd find 44
# on that host:
iostat -x 1 5 | grep -E '^Device|sdX'

# 3. is the whole host affected, or one device?
HOST=$(ceph osd find 44 -f json | jq -r '.crush_location.host')
ceph osd df tree | grep -A15 "$HOST"
ceph osd perf | sort -k2 -rn | head -20

Three OSDs on the same host being slow together points at the host — CPU, memory, HBA — rather than at three simultaneously failing devices.

Mapping to hardware

ceph osd find 44
ceph osd metadata 44 | python3 -c '
import sys,json; d=json.load(sys.stdin)
for k in ("hostname","devices","bluestore_bdev_dev_node","bluefs_db_devices",
          "bluestore_bdev_type","osd_objectstore"):
    print("%-28s %s" % (k, d.get(k)))'

That output gives the host, the data device, and the DB device — the three facts needed before touching anything.

Quiz

Production discipline

Confirm a latency outlier across three samples before acting; the figure is a rolling average that a single burst can skew. When several OSDs on one host are slow together, investigate the host — CPU, HBA, or a shared DB device — rather than the individual drives.

Cross-course references

• Kubernetes: several pods degraded on one node points at the node
• Linux: multiple devices behind one controller failing together implicates the controller