Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-network~22 min

Network latency spikes - cluster responds slowly

Reported symptoms

  • ●PGs report slow ops during the latency window
  • ●Clients see I/O p99 spike
  • ●The cluster returns to normal after the network recovers

Evidence

  • · iperf3 between OSD hosts shows elevated RTT
  • · mtr shows a high-loss / high-latency hop
  • · A switch port on the cluster network reports errors
Diagnosis and resolutionclick to reveal

Root cause

A network event caused transient latency.

Remediation

Once the network recovers, verify the cluster returns to normal. Identify the failure; address the root cause.

Verification

ceph -s returns to HEALTH_OK; ceph daemonperf <osd> shows latency back to baseline; slow ops clear.

Prevention

Network observability; redundant switch topology; cluster network on dedicated VLANs; alerting on key links.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • PGs report slow ops during the latency window
  • Clients see I/O p99 spike
  • The cluster returns to normal after the network recovers

Evidence

  • iperf3 between OSD hosts shows elevated RTT
  • mtr shows a high-loss / high-latency hop
  • A switch port on the cluster network reports errors

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.