Investigate network latency on a Ceph cluster

This runbook covers Investigate network latency on a Ceph cluster on a Ceph Tentacle cluster.

When to use

• Active production scenario matching the impact (service-affecting)
• The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

• Clients see I/O latency
• ceph health detail reports slow ops
• The cluster is otherwise HEALTH_OK

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph daemonperf `<osd>` shows per-OSD latency
# iperf3 between OSD hosts shows throughput / latency
# mtr shows loss / latency on the path
# Switch port counters are stable

Procedure

Step 1

Identify the scope of the slowness: all OSDs, a subset, a single host

Step 2

If all: network is suspect; check switches, MTU, congestion

Step 3

If a subset: check the affected hosts and switches

Step 4

If a single host: check the hosts NIC and bond

Step 5

Apply the smallest mitigation

Step 6

Verify: iperf3 shows the expected throughput

Verification

After completing the procedure, verify the result:

# iperf3 shows the expected throughput
# ceph daemonperf shows latency back to baseline
# Cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

• If the network stays slow: involve the network team
• If recovery is interacting: throttle recovery

Escalation

If you cannot roll back or the rollback fails, escalate:

• Network team
• Storage team