Controlling how fast recovery runs

Advanced⏱ ~18 minceph

What you'll learn

Identify the active OSD scheduler
Change recovery aggressiveness through the right lever
Estimate when recovery will finish
Detect an OSD whose measured capacity is wrong

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Recovery competes with client I/O for the same devices, and the settings most operators reach for have had no effect since the scheduler changed.

Which scheduler is running

ceph config get osd osd_op_queue          # mclock_scheduler
ceph config get osd osd_mclock_profile    # balanced
ceph config get osd osd_mclock_override_recovery_settings   # false

Profile	Client I/O	Recovery
`high_client_ops`	prioritised	slowest
`balanced`	default split	default
`high_recovery_ops`	reduced	fastest
`custom`	operator-defined reservations and limits	operator-defined

ceph config set osd osd_mclock_profile high_recovery_ops
ceph config set osd osd_mclock_profile balanced      # back again

The settings that no longer apply

# these are overridden by the profile unless the override is enabled
ceph config get osd osd_max_backfills
ceph config get osd osd_recovery_max_active
ceph config get osd osd_recovery_sleep

ceph config set osd osd_mclock_override_recovery_settings true
ceph config set osd osd_max_backfills 4

Setting `osd_max_backfills` under the default scheduler and observing no
change is the single most common wasted hour in a recovery. Switch the
profile instead.

Estimating completion

ceph -s | grep -A2 'io:'

  io:
    client:   412 MiB/s rd, 88 MiB/s wr, 3.10k op/s rd, 1.90k op/s wr
    recovery: 1.9 GiB/s, 640 objects/s

ceph -s --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)["pgmap"]
deg, rate = d.get("degraded_objects",0), d.get("recovering_objects_per_sec",0)
print("degraded: %d  rate: %d/s  eta: %s" %
      (deg, rate, "%.1f h" % (deg/rate/3600) if rate else "stalled"))'

ceph progress

Removing the competition

ceph osd set noscrub
ceph osd set nodeep-scrub

Deep scrub reads every object on the OSD and competes for exactly the
device time recovery needs. Suspending it for the duration is usually
worth more than any profile change, and both flags must be cleared
afterwards or the cluster silently stops verifying itself.

Finding the OSD that is holding it up

ceph osd perf | sort -k2 -n -r | head
ceph config show osd.7 osd_mclock_max_capacity_iops_ssd

Each OSD benchmarks its device at first start and stores the result. A
benchmark taken while the device was busy records a capacity far below
the truth, and mClock then throttles that OSD to match.

ceph config rm osd.7 osd_mclock_max_capacity_iops_ssd
ceph orch daemon restart osd.7

Why mClock allocates device time rather than counting concurrent backfills

mClock is a quality-of-service scheduler that gives each operation class — client, recovery, scrub — a reservation, a weight, and a limit expressed as a share of the OSD’s measured throughput. Counting concurrent backfills was always a poor proxy, because one backfill of large objects and one of 4 KiB objects consume wildly different amounts of device time, so the same number produced very different interference on different pools. Allocating measured capacity fixes that, at the cost of depending on the measurement: each OSD benchmarks its device at first start and records osd_mclock_max_capacity_iops_hdd or _ssd in the configuration database. If that benchmark ran while the device was under load, the OSD is throttled against a fictitious capacity for the rest of its life, and it becomes the OSD every recovery waits for. Removing the stored value and restarting the daemon forces a fresh measurement.

Quiz

Knowledge check · 4 questions

Q1. Recovery is too slow and `osd_max_backfills` has been raised with no observable change. What is happening?

The mClock profile supplies its own value and the setting is ignored unless the override is enabled The setting requires an OSD restart to apply, and the daemons have not been restarted since the change The value must be set on the monitors instead, since backfill concurrency is a cluster-wide limit The setting only applies to erasure-coded pools, and the degraded PGs here are replicated
Q2. Changing the mClock profile requires restarting the OSDs.

True False
Q3. Speed up a recovery that will otherwise run into business hours.

A host loss left 3.7 million objects degraded. Recovery is running at 180 objects per second and the cluster is idle overnight.
Q4. How can one OSD end up throttling an entire recovery under mClock?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Change osd_mclock_profile rather than osd_max_backfills — under the default scheduler the classic throttles are supplied by the profile and ignored. Check ceph osd perf and the stored mClock capacity for a single outlier before concluding the whole cluster is slow.

Cross-course references

Kubernetes: resource requests allocate shares of measured capacity, not counts of pods
Linux: an I/O scheduler calibrated against a bad benchmark throttles for the life of the device