Working a slow-ops report

Why this matters in production

SLOW_OPS is the health check that most often accompanies a user-visible performance problem, and it names the OSDs involved — which is most of the diagnosis if the output is read properly.

What the report says

ceph health detail

[WRN] SLOW_OPS: 47 slow ops, oldest one blocked for 92 sec, daemons
      [osd.12,osd.31] have slow ops.

Three facts: how many, how long the oldest has waited, and which daemons. The daemon list is the lead.

ceph daemon osd.12 dump_ops_in_flight | python3 -c '
import sys,json; d=json.load(sys.stdin)
for op in d.get("ops",[])[:5]:
    print(op.get("description","")[:80])
    print("   ", op.get("type_data",{}).get("flag_point"))'

The flag_point is the stage the operation is stuck at, which names the subsystem directly.

Reading the flag points

waiting for subops from [31] is the most informative: it names the OSD that is actually slow, which may not be the one reporting the slow op.

Applying the method

# 1. which OSDs, and is one of them named in the subops?
IFACE=bond0
THE_IMPLICATED_HOST=stor-04
ceph health detail | grep -o 'osd\.[0-9]*'
ceph daemon osd.12 dump_ops_in_flight | grep -o 'waiting for subops from.*'

# 2. device layer on the implicated host
ceph osd find 31
# on that host:
iostat -x 1 5
dmesg -T | tail -30
smartctl -a /dev/sdX | grep -iE 'pending|reallocated'

# 3. host layer
uptime
vmstat 1 5
free -g

# 4. network layer
ethtool -S ${IFACE} | grep -iE 'drop|err'
ping -c 100 ${THE_IMPLICATED_HOST}

The common cases

# rule several out quickly
ceph osd perf | sort -k2 -n | tail -5
ceph -s | grep -E 'recovery|degraded'
ceph health detail | grep -E 'FULL|NEARFULL'

Quiz

Production discipline

Read dump_ops_in_flight and its flag_point before investigating the OSDs named in a SLOW_OPS warning; “waiting for subops from” names the OSD actually responsible, which is often not in the warning. Check the device, host, and network layers on that host rather than on the reporting one.

Cross-course references

• Kubernetes: a failing dependency surfaces as errors in every service that calls it
• Linux: a slow NFS server produces symptoms on every client, not on itself