Investigate disk latency on a Ceph host

This runbook covers Investigate disk latency on a Ceph host on a Ceph Tentacle cluster.

When to use

• Active production scenario matching the impact (service-affecting)
• The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

• An OSD host reports elevated I/O latency
• ceph daemonperf confirms the disk is slow

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph daemonperf `<osd>` shows the OSDs latency percentiles
# iostat -x shows the per-device latency
# smartctl -a /dev/<device> reports device health
# The host kernel log has I/O errors

Procedure

Step 1

Identify the slow device

Step 2

Check device health (SMART for HDD; nvme smart-log for NVMe)

Step 3

Check the I/O scheduler

Step 4

Check the multi-queue depth

Step 5

If the device is healthy: check the host

Step 6

If the device is failing: replace

Step 7

Verify the OSD recovers

Verification

After completing the procedure, verify the result:

# The slow device is identified
# OSD latency returns to baseline
# Cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

• If the device is suspect: replace it
• If the device is healthy: investigate downstream
• If the OSD daemon is the issue: restart it

Escalation

If you cannot roll back or the rollback fails, escalate:

• Storage team
• Hardware vendor if the device is failing