Running a recovery tuning cycle

Why this matters in production

Tuning without a baseline produces a configuration that someone believes is better. Tuning with one produces a number. Only the second survives a staff change.

Establishing a baseline

# client latency, from a client host
fio --name=base --ioengine=rbd --pool=rbd-vms --rbdname=probe \
    --rw=randwrite --bs=4k --iodepth=16 --runtime=300 --time_based \
    --percentile_list=50:95:99 > baseline.txt

# recovery rate, over a window
a=$(ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects (degraded|misplaced)' | cut -d/ -f1)
sleep 900
b=$(ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects (degraded|misplaced)' | cut -d/ -f1)
echo "baseline rate: $(( (a-b)/900 )) objects/s"

Both numbers, recorded together with the configuration that produced them.

The cycle

1. record the current configuration and both measurements
2. change exactly one setting
3. wait for the effect to stabilise — at least 15 minutes
4. measure both again
5. record the pair
6. if the trade is favourable, keep it and repeat from 2
   if not, revert and try a different setting

ceph config dump | grep osd > config-step-3.txt
ceph config set osd osd_max_backfills 4
sleep 900
# re-measure

Recognising when to stop

The last is the goal and it is easy to overshoot past it in pursuit of a better recovery number that nobody asked for.

Recording the outcome

Cluster: prod-ceph-01
Date: 2026-08-18
Hardware: 12x 16TB HDD per host, NVMe DB, 25GbE cluster network
Scheduler: mclock, balanced profile

baseline:  240 obj/s recovery, p99 write 12 ms
max_backfills 2: 460 obj/s, p99 14 ms   ← kept
max_backfills 4: 610 obj/s, p99 19 ms   ← kept
max_backfills 8: 640 obj/s, p99 34 ms   ← reverted, poor trade
sleep_hdd 0:     700 obj/s, p99 21 ms   ← kept

final: max_backfills 4, sleep_hdd 0

That record answers next year’s question without repeating the work, and it makes the reasoning auditable when someone proposes changing it.

Quiz

Production discipline

Establish a baseline of both recovery rate and client latency before changing anything, then change one setting at a time with a fifteen-minute settling period. Record every measurement pair with the hardware and scheduler — the record answers the next proposed change without repeating the experiment in production.

Cross-course references

• Kubernetes: resource tuning follows the same baseline-change-measure discipline
• Linux: any performance work without a baseline produces an opinion rather than a result