Bounding recovery's cost to client latency

Why this matters in production

Recovery competes with clients at the OSD, and the competition is continuous during any failure. Setting a bound in advance means the trade-off is a policy rather than an argument during an incident.

Quantifying the cost

# baseline, no recovery
fio --name=base --ioengine=rbd --pool=rbd-vms --rbdname=probe \
    --rw=randwrite --bs=4k --iodepth=32 --runtime=180 --time_based \
    --percentile_list=50:95:99

# during recovery, same test
ceph -s | grep recovery

A typical measurement on an HDD cluster:

no recovery:              p50 4 ms   p95 9 ms   p99 16 ms
recovery, balanced:       p50 5 ms   p95 14 ms  p99 38 ms
recovery, high_recovery:  p50 7 ms   p95 24 ms  p99 71 ms
recovery, high_client:    p50 4 ms   p95 10 ms  p99 21 ms

Those numbers are the input to the policy. Without them the choice of profile is a guess.

Setting the bound

Policy: during recovery, client p99 write latency may reach 2× baseline.
        Recovery is throttled to hold that bound.
        Exception: when a pool is at min_size, recovery takes priority
        and the bound is suspended until redundancy is restored.

The exception is what makes the policy usable — a blanket latency bound would prevent the cluster protecting itself when that matters most.

Enforcing it

# the primary lever
ceph config set osd osd_mclock_profile high_client_ops

# supplementary, if mClock is not in use
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_sleep_hdd 0.1

# the exception path
ceph config set osd osd_mclock_profile high_recovery_ops

# make the exception explicit and observable
ceph health detail | grep -E 'PG_DEGRADED|undersized'
ceph pg dump pgs | awk '$10 ~ /undersized/ {print $1, $10}' | head

Verifying the bound holds

# during any recovery
watch -n 60 'ceph -s | grep -E "recovery|degraded"'

# Prometheus, over the recovery window
histogram_quantile(0.99, rate(ceph_osd_op_w_latency_bucket[5m]))

An alert on the bound being exceeded turns the policy into something enforced rather than intended:

- alert: RecoveryLatencyBudgetExceeded
  expr: |
    histogram_quantile(0.99, rate(ceph_osd_op_w_latency_bucket[5m])) > 0.032
    and on() ceph_pg_recovering > 0
  for: 15m
  annotations:
    summary: "Client p99 exceeds 2× baseline during recovery"
    runbook: "Lower mclock profile to high_client_ops, or accept and document"

Quiz

Production discipline

Measure client p99 under each mClock profile during a controlled recovery before writing a latency policy — without the numbers the profile choice is a guess made during an incident. Express the bound on the p99, not the median, and state explicitly that it is suspended below min_size.

Cross-course references

• Kubernetes: background job resource budgets need the same tail-latency framing
• Linux: I/O scheduler policies for background work face the identical trade