Recovery traffic and the throttles that shape it

Advanced⏱ ~18 minceph

What you'll learn

Estimate recovery traffic volume for a failure scenario
Apply the throttles that pace recovery
Balance recovery speed against client impact
Use the mClock scheduler profiles appropriately

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Recovery is unbounded by client demand — it is bounded only by the throttles. Left at defaults it may take days on a large cluster; opened up it can consume the entire network and make the cluster unusable. Both extremes are failures of configuration, not of Ceph.

The volume

Failure	Replicated size 3	EC 8+3
One 8 TB OSD	8 TB	64 TB
One host, 12 × 8 TB	96 TB	768 TB
One rack, 5 hosts	480 TB	3.8 PB

Divide by realistically available bandwidth to get the window. Those numbers are why recovery pacing is a design concern rather than a runtime detail.

The classic throttles

# concurrent backfills per OSD — the primary lever
ceph config set osd osd_max_backfills 1

# concurrent recovery operations per OSD
ceph config set osd osd_recovery_max_active 3

# artificial delay between recovery operations
ceph config set osd osd_recovery_sleep_hdd 0.1
ceph config set osd osd_recovery_sleep_ssd 0

# how many PGs may be in recovery at once, cluster-wide
ceph config set osd osd_recovery_max_single_start 1

osd_max_backfills is the one to reach for first. It bounds how many PGs an OSD will backfill concurrently, which bounds both its network usage and its device queue depth.

mClock

Recent Ceph uses the mClock scheduler by default, which allocates OSD I/O capacity between client operations, recovery, and scrubbing by proportional share rather than by concurrency caps.

ceph config get osd osd_mclock_profile

ceph config set osd osd_mclock_profile high_client_ops   # protect clients
ceph config set osd osd_mclock_profile balanced          # default
ceph config set osd osd_mclock_profile high_recovery_ops # prioritise repair

This is a better tool than the classic throttles because it degrades gracefully: rather than a hard cap that either starves recovery or floods the network, it allocates whatever capacity is spare.

With mClock active, several classic throttles are overridden — check which are in effect before tuning:

ceph config show osd.0 | grep -E 'mclock|max_backfills|recovery_sleep'

Choosing a posture

Situation	Profile
Normal single-OSD failure, healthy cluster	`balanced`
Client SLO under pressure during recovery	`high_client_ops`
PGs below min_size, or durability at risk	`high_recovery_ops`
Planned migration in a maintenance window	`high_recovery_ops`

The reasoning is the same as everywhere else in Ceph: while redundancy is intact, client latency wins; once durability is genuinely at risk, recovery wins.

Why mClock beats fixed concurrency caps

A fixed osd_max_backfills value is a static bound that has to be chosen for the worst case — busy cluster, peak client load — and therefore wastes capacity whenever clients are idle. mClock instead assigns weights, reservations, and limits to each class of work and schedules against the OSD’s measured I/O capacity, so recovery expands into spare capacity at night and contracts under daytime load without anyone touching a setting. The cost is that the scheduler must know the device’s real capacity, which is why mClock benchmarks the OSD at startup and why a badly-measured capacity produces either sluggish recovery or client impact — checking osd_mclock_max_capacity_iops_hdd/_ssd for an implausible value is the first step when mClock behaves oddly.

Quiz

Knowledge check · 4 questions

Q1. A cluster is recovering with several PGs below min_size and clients blocked. Which mClock profile is appropriate?

high_client_ops, to protect the clients whose I/O is still completing balanced, as a compromise while the cluster works through the backlog high_recovery_ops, because durability and availability are at risk Disable mClock and set fixed throttles so recovery has a hard ceiling
Q2. mClock allocates OSD I/O capacity by proportional share rather than through fixed concurrency caps.

True False
Q3. Tune recovery for a large EC rebuild.

A host with twelve 16 TB OSDs failed on an EC 8+3 pool. Estimated recovery traffic is around 1.5 PB. At current rates the rebuild will take nine days. Clients are meeting their SLO comfortably, and the cluster has a 100 Gb network.
Q4. Why does mClock benchmark the OSD device at startup, and what goes wrong if the measurement is bad?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Record the mClock profile in your incident runbook alongside the criteria for changing it, and make reverting it an explicit closing step — a cluster silently left on a recovery-priority profile serves clients worse for months. Estimate recovery traffic for your largest failure domain during design and check it against the network you actually have.

Cross-course references

Kubernetes: priority classes and preemption serve the same role of ordering competing work
Linux: ionice and cgroup I/O weights implement the same proportional-share idea