Pre-check: capacity and impact for the backfill

Why this matters in production

The backfill from an addition is predictable in size and duration, and planning it is what makes the expansion a scheduled operation.

Estimating the backfill

ceph df
ceph osd df | awk 'NR>1 && $1 ~ /^[0-9]+$/ {n++} END {print n, "OSDs currently"}'

current OSDs: 96, adding 12
data: 180 TB
moved ≈ 180 × 12/108 ≈ 20 TB

# and the achievable rate, from a previous expansion or a measurement
# 20 TB at 400 MB/s ≈ 14 hours

Verifying the cluster can absorb it

# no OSD approaches backfillfull during the movement
ceph osd df | sort -k17 -rn | head -5

# and there is headroom for the transient increase
ceph df | grep -A2 'RAW STORAGE'

ceph -s
ceph osd dump | grep flags

Planning the client impact

# measure the current baseline
# Substitute your own pool before running:
POOL=rbd-vms

fio --name=base --ioengine=rbd --pool="$POOL" --rbdname=probe \
    --rw=randwrite --bs=4k --iodepth=32 --runtime=120 --time_based \
    --percentile_list=50:95:99

The plan states:
  expected duration at the chosen throttle
  expected client latency during it
  the schedule if paced diurnally
  the abort condition

The abort condition is what makes the plan actionable: a stated client latency above which the backfill is paused with norebalance.

Setting throttles before starting

ceph config set osd osd_mclock_profile high_client_ops
ceph config set osd osd_max_backfills 2
ceph config get osd osd_max_backfills

# and a diurnal schedule if the duration warrants it
0 8  * * * ceph config set osd osd_max_backfills 1
0 20 * * * ceph config set osd osd_max_backfills 6

Setting the throttles before adding the OSDs means the movement starts at the intended rate rather than at the default and being corrected.

Quiz

Production discipline

Set the throttles and mClock profile before adding the OSDs so the movement starts at the intended rate rather than at the default. Define an abort condition on client latency before starting; deciding to pause during the impact is a decision made under pressure and made late.

Cross-course references

• Kubernetes: defining rollback criteria before a rollout is the same discipline
• Linux: any long-running operation benefits from a stated abort condition