The backfill settings and their interaction

Why this matters in production

Backfill has more settings than recovery and they interact. Changing them in the wrong order produces no effect and wastes a maintenance window.

The settings

ceph config get osd osd_max_backfills
ceph config get osd osd_backfill_scan_min
ceph config get osd osd_backfill_scan_max
ceph config get osd osd_backfill_retry_interval
ceph config get osd osd_min_recovery_priority

Why the scan settings exist

Backfill proceeds by scanning ranges of the PG’s object namespace and comparing the primary’s contents with the destination’s. Larger scan ranges mean fewer round trips and higher throughput; they also mean each scan holds locks longer and produces a larger burst.

ceph config set osd osd_backfill_scan_max 1024    # default is smaller

This helps on clusters with very many small objects, where the round-trip cost dominates. It rarely helps elsewhere, which is why it is not the first thing to change.

Order of changes by effect size

# 1. the profile, if mClock is active
ceph config set osd osd_mclock_profile high_recovery_ops

# 2. concurrency
ceph config set osd osd_max_backfills 4

# 3. the sleep, if non-zero
ceph config set osd osd_recovery_sleep_hdd 0

# 4. scan ranges, only for small-object workloads
ceph config set osd osd_backfill_scan_max 1024

Change one at a time and measure between changes; changing several together means the effect cannot be attributed.

When the setting is not the bottleneck

# a full destination
ceph health detail | grep -i toofull

# a saturated network
ceph osd perf | sort -k3 -rn | head

# a single slow OSD limiting every PG it holds
ceph osd perf | sort -k2 -n | tail -5

# operations blocked
ceph -s | grep -i 'slow ops'

If any of these is present, raising a throttle changes nothing because the limit is elsewhere.

Quiz

Production discipline

Change one backfill setting at a time and measure for fifteen minutes between changes; simultaneous changes give a result with no attribution. Before raising a throttle again, confirm it is the binding constraint — check backfill_toofull, ceph osd perf outliers, and slow ops.

Cross-course references

• Kubernetes: changing several resource settings at once makes regressions unattributable
• Linux: single-variable changes are the basis of any credible performance work