Tune recovery and backfill safely

This runbook covers Tune recovery and backfill safely on a Ceph Tentacle cluster.

When to use

• Active production scenario matching the impact (service-affecting)
• The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

• Recovery is taking too long
• Recovery is impacting client I/O

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph -s reports the current recovery rate
# ceph progress reports the recovery percentage
# ceph daemonperf `<osd>` shows per-OSD latency

Procedure

Step 1

Capture the baseline latency and IOPS

Step 2

Decide the target recovery rate

Step 3

ceph config set osd osd_recovery_max_active 3

Step 4

ceph config set osd osd_recovery_max_operations 5

Step 5

ceph config set osd osd_recovery_sleep 0.1

Step 6

ceph config set osd osd_max_backfills 1

Step 7

Apply per-pool priorities: ceph osd pool set <pool> recovery_priority 5

Step 8

Verify: client latency is acceptable; recovery completes

Verification

After completing the procedure, verify the result:

# Recovery completes within the planned window
# Client latency p99 is acceptable
# Cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

• If latency increases: reduce the recovery rate
• If recovery stalls: identify the slow OSD
• If pool priorities are wrong: revert

Escalation

If you cannot roll back or the rollback fails, escalate:

• Storage team