Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateStorage~25 min

Proxmox VM storage latency - RBD write p99 spikes

Reported symptoms

  • ●A Proxmox VM reports I/O latency above 50ms
  • ●Other VMs on the same RBD pool show similar latency
  • ●Ceph reports slow ops in ceph health detail

Evidence

  • · ceph daemonperf <osd> identifies the slow OSD
  • · ceph pg <pgid> query shows the affected PGs
  • · pvesm status confirms the RBD storage is the path
Diagnosis and resolutionclick to reveal

Root cause

Recovery or backfill is consuming disk and network; the RBD write path is contending with recovery I/O.

Remediation

Throttle recovery (osd_recovery_max_active 1; osd_recovery_sleep 1); wait for latency to return to baseline.

Verification

ceph daemonperf returns to baseline latency; the VM I/O latency normalises; ceph health detail shows no slow ops.

Prevention

Pin Proxmox VMs to a CRUSH rule on NVMe; keep backfill tuned; alert on slow ops.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • A Proxmox VM reports I/O latency above 50ms
  • Other VMs on the same RBD pool show similar latency
  • Ceph reports slow ops in ceph health detail

Evidence

  • ceph daemonperf &lt;osd&gt; identifies the slow OSD
  • ceph pg &lt;pgid&gt; query shows the affected PGs
  • pvesm status confirms the RBD storage is the path

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.