The settings that bound recovery speed

Advanced⏱ ~17 minceph

What you'll learn

Identify each recovery throttle and its effect
Understand the interaction with mClock
Adjust recovery rate deliberately
Measure the effect of a change

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Recovery speed is entirely a configuration choice, and several settings influence it with overlapping effects. Knowing which one actually applies on your cluster prevents adjusting a setting that mClock is overriding.

The classic throttles

ceph config get osd osd_recovery_max_active
ceph config get osd osd_recovery_max_single_start
ceph config get osd osd_recovery_sleep_hdd
ceph config get osd osd_recovery_sleep_ssd
ceph config get osd osd_recovery_op_priority

Setting	Controls
`osd_recovery_max_active`	concurrent recovery operations per OSD
`osd_recovery_max_single_start`	how many start per PG at once
`osd_recovery_sleep_*`	a delay between operations, per device class
`osd_recovery_op_priority`	priority relative to client operations

osd_recovery_sleep is the blunt instrument: a non-zero value inserts a pause between operations, capping the rate regardless of available capacity.

mClock

Recent Ceph uses the mClock scheduler by default, which allocates OSD I/O capacity between classes of work rather than capping concurrency:

ceph config get osd osd_op_queue                 # mclock_scheduler
ceph config get osd osd_mclock_profile

ceph config set osd osd_mclock_profile high_client_ops
ceph config set osd osd_mclock_profile balanced
ceph config set osd osd_mclock_profile high_recovery_ops

With mClock active, several classic throttles are overridden. Setting osd_recovery_max_active on an mClock cluster may have no effect.

ceph config show osd.0 | grep -E 'mclock|recovery_max_active|recovery_sleep'

Check which are actually in force before adjusting anything.

Which to reach for

Situation	Lever
mClock active, need faster recovery	`osd_mclock_profile high_recovery_ops`
mClock active, protect clients	`osd_mclock_profile high_client_ops`
mClock active, custom allocation	the mClock reservation and weight settings
mClock disabled	the classic throttles
Emergency stop	`ceph osd set norecover`

The profile is the intended interface on a modern cluster and it degrades gracefully — recovery expands into spare capacity rather than being capped.

Measuring a change

ceph -s | grep recovery
# recovery: 412 MiB/s, 103 objects/s

ceph config set osd osd_mclock_profile high_recovery_ops
sleep 120
ceph -s | grep recovery

Measure client latency alongside, since the whole point is the trade between them:

ceph osd pool stats rbd-vms

Quiz

Knowledge check · 4 questions

Q1. You set `osd_recovery_max_active` higher and recovery speed does not change. Why?

The setting requires an OSD restart before it takes effect on running daemons The value was set at the wrong scope and applies to only one OSD Recovery is already at the maximum the source devices can supply The mClock scheduler is active and overrides several classic throttles
Q2. mClock adapts recovery rate to available capacity without a setting change.

True False
Q3. Speed up a slow recovery.

A recovery is projected to take 40 hours. Client latency is well within the SLO. The cluster uses the default mClock configuration.
Q4. What does mClock need to know about the device, and what goes wrong if the value is bad?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Check which throttles are actually in force before adjusting anything — on an mClock cluster several classic settings are ignored and adjusting them wastes effort. Use the mClock profile as the primary lever and measure both recovery rate and client latency, since the trade between them is the whole point.

Cross-course references

Kubernetes: QoS classes allocating proportionally rather than capping is the same design
Linux: proportional I/O schedulers replaced fixed limits for the same reasons