Controlling backfill speed

Why this matters in production

Backfill is the movement that runs for hours or days after any topology change, and it is the one most likely to affect clients over an extended period. Its throttles are separate from the recovery ones.

The settings

ceph config get osd osd_max_backfills
ceph config get osd osd_backfill_scan_min
ceph config get osd osd_backfill_scan_max
ceph config get osd osd_backfill_retry_interval
ceph config get osd osd_recovery_sleep_hdd

osd_max_backfills is the primary lever. It bounds both how many backfills an OSD sources and how many it receives, so raising it increases concurrency at both ends.

Under mClock

ceph config get osd osd_op_queue
ceph config show osd.0 | grep -E 'mclock|max_backfills'

With mClock active, osd_max_backfills may be overridden along with the recovery throttles. The profile is again the applicable lever:

ceph config set osd osd_mclock_profile high_client_ops

Adjusting deliberately

# slow, for a busy period
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_sleep_hdd 0.1

# faster, for an off-peak window
ceph config set osd osd_max_backfills 8
ceph config set osd osd_recovery_sleep_hdd 0

Changes take effect without restarting daemons, so a backfill can be paced through a diurnal cycle:

# a simple schedule
0 8  * * * ceph config set osd osd_max_backfills 1
0 20 * * * ceph config set osd osd_max_backfills 8

Verifying the effect

ceph -s | grep recovery
sleep 300
ceph -s | grep recovery

If the rate does not change after a throttle adjustment, either mClock is overriding it or the bottleneck is elsewhere — the network, the source device, or a backfill_toofull condition.

ceph health detail | grep -i backfill
ceph osd df | sort -k17 -rn | head

Quiz

Production discipline

Pace long backfills through the diurnal cycle rather than choosing one conservative value; the overnight window is capacity a single setting cannot use and the changes take effect immediately. Verify a throttle change had an effect — if it did not, the bottleneck is elsewhere.

Cross-course references

• Kubernetes: scheduling background work by time of day is the same capacity-shaping technique
• Linux: scheduling maintenance jobs against a load profile is standard practice