The client impact of backfill and how to bound it

Why this matters in production

Backfill runs for days after a large change and its impact accumulates over that period. A latency increase that is acceptable for an hour may not be acceptable for a week, which is a different calculation from the one applied to recovery.

The resource profile

Backfill’s operations are large, so a client request queued behind one waits longer than it would behind a client request.

Measuring the impact

# before, or during a pause
ceph osd pool stats rbd-vms
rbd perf image iostat --pool rbd-vms

# during
ceph -s | grep -E 'client|recovery'
ceph osd perf | sort -k2 -n | tail -5

The comparison that matters is the client-visible one:

# from a client host
fio --name=probe --ioengine=rbd --pool=rbd-vms --rbdname=probe \
    --rw=randwrite --bs=4k --iodepth=16 --runtime=60 --time_based \
    --percentile_list=50:95:99

Run it during a pause and during backfill, and the difference is the impact.

Setting an impact budget

during backfill: p99 write latency may rise to 25 ms from a 12 ms baseline
duration: acceptable for up to 5 days

Both terms. A budget expressed only as a latency limit permits an indefinite backfill at that limit, which is a different commitment.

# enforce it
ceph config set osd osd_max_backfills 2
ceph config set osd osd_mclock_profile high_client_ops

Distinguishing backfill impact from other causes

ceph osd set norebalance
sleep 300
# measure client latency with backfill paused
ceph osd unset norebalance

Pausing and measuring is the definitive test. If latency does not improve during the pause, backfill is not the cause and the investigation should move elsewhere — most commonly to a single slow OSD.

ceph osd perf | sort -k2 -n | tail -5

Quiz

Production discipline

Pause backfill briefly and re-measure before concluding it is the cause of a latency increase; a marginal OSD produces the same symptom at the same time. Express backfill impact budgets as both a latency limit and a duration, since the two commitments are different.

Cross-course references

• Kubernetes: pausing a rollout to isolate whether it is causing an observed regression is the same test
• Linux: stopping background I/O to confirm attribution is standard diagnostic practice