Skip to main content
RunBook Academy

← All runbooks in Ceph

medium riskservice affecting~30 min

Troubleshoot Proxmox / Ceph storage performance

1 · Prerequisites

Confirm every item is in place before any state change.

  • A Proxmox VM reports storage latency
  • The RBD image shows slow ops
  • Ceph reports slow ops

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Check Proxmox storage status: pvesm status
  • · Check the RBD image: rbd info
  • · Check Ceph: ceph -s, ceph health detail
  • · Check the Proxmox hosts bridge
  • · Check the Proxmox hosts CPU / memory / disk
  • · Check the Ceph OSD hosts resources

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Identify the layer: RBD client, network, Ceph cluster, OSD
  2. 2If RBD client: tune krbd / librbd / queue depth
  3. 3If network: verify bonding, MTU, congestion
  4. 4If Ceph cluster: tune recovery, identify slow OSDs
  5. 5Verify: client latency returns to baseline

4 · Verification

Confirm the procedure actually fixed the problem.

  • The Proxmox VMs latency returns to baseline
  • Ceph reports no slow ops
  • Cluster is HEALTH_OK

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • If the slow OSD is the cause: replace it
  • If the network is the cause: engage the network team
  • If recovery is the cause: tune recovery settings

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team
  • · Network team if the network is the cause

Troubleshoot Proxmox / Ceph storage performance

This runbook covers Troubleshoot Proxmox / Ceph storage performance on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (service-affecting)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • A Proxmox VM reports storage latency
  • The RBD image shows slow ops
  • Ceph reports slow ops

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# Check Proxmox storage status: pvesm status
# Check the RBD image: rbd info
# Check Ceph: ceph -s, ceph health detail
# Check the Proxmox hosts bridge
# Check the Proxmox hosts CPU / memory / disk
# Check the Ceph OSD hosts resources

Procedure

Step 1

Identify the layer: RBD client, network, Ceph cluster, OSD

Step 2

If RBD client: tune krbd / librbd / queue depth

Step 3

If network: verify bonding, MTU, congestion

Step 4

If Ceph cluster: tune recovery, identify slow OSDs

Step 5

Verify: client latency returns to baseline

Verification

After completing the procedure, verify the result:

# The Proxmox VMs latency returns to baseline
# Ceph reports no slow ops
# Cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

  • If the slow OSD is the cause: replace it
  • If the network is the cause: engage the network team
  • If recovery is the cause: tune recovery settings

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team
  • Network team if the network is the cause

References

  1. Ceph documentation