Investigate slow ops across the cluster

This runbook covers Investigate slow ops across the cluster on a Ceph Tentacle cluster.

When to use

• Active production scenario matching the impact (service-affecting)
• The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

• ceph -s reports HEALTH_WARN with slow ops
• Clients experience storage latency

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# The number of slow ops in ceph -s
# The clusters IOPS / recovery rate
# Per-OSD latency from ceph daemonperf
# Per-pool IOPS from ceph osd pool stats
# Host-side latency from iostat

Procedure

Step 1

Identify whether the slowness is global or scoped

Step 2

If scoped: trace the affected OSDs

Step 3

If global: check network, recovery, capacity

Step 4

Apply the smallest mitigation

Step 5

Verify client latency returns to baseline

Verification

After completing the procedure, verify the result:

# Client latency p99 returns to baseline
# Slow ops count drops to zero
# Cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

• If the slow ops persist: engage the storage team
• If recovery is the cause: tune recovery settings
• If hardware is the cause: replace the disk

Escalation

If you cannot roll back or the rollback fails, escalate:

• Storage team
• Hardware vendor if a device is suspect