Managing the rebalance an expansion causes

Why this matters in production

An expansion’s rebalance runs for hours or days and is entirely predictable in size, which makes it plannable rather than something to endure.

Estimating the movement

new capacity fraction ≈ new OSD count / total OSD count after
data moved      ≈ total data × that fraction

96 OSDs holding 180 TB, adding 36:
  new fraction = 36 / 132 = 27%
  data moved ≈ 180 × 0.27 ≈ 49 TB

ceph df
ceph osd df | awk 'NR>1 {n++} END {print n, "OSDs"}'

# after starting, the actual figure
ceph -s | grep misplaced

Pacing it

# conservative, during business hours
ceph config set osd osd_max_backfills 1
ceph config set osd osd_mclock_profile high_client_ops

# faster, overnight
ceph config set osd osd_max_backfills 6
ceph config set osd osd_mclock_profile balanced

# a diurnal schedule
0 8  * * * ceph config set osd osd_max_backfills 1
0 20 * * * ceph config set osd osd_max_backfills 6

Pacing an expansion through the diurnal cycle typically halves the total duration with no additional peak-hour impact.

Monitoring progress

ceph -s | grep -E 'misplaced|recovery'
ceph progress

# windowed rate, more reliable than the instantaneous figure
a=$(ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects misplaced' | cut -d/ -f1)
sleep 900
b=$(ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects misplaced' | cut -d/ -f1)
echo "$(( (a-b)/900 )) objects/s; $(( b / ((a-b)/900) / 3600 )) hours remaining"

# and the distribution converging
ceph osd df | awk 'NR>1 {if($17+0>m)m=$17+0; if(min==""||$17+0<min)min=$17+0}
  END {printf "spread %.1f\n", m-min}'

The spread narrowing is the outcome; the misplaced count falling is the mechanism.

When it runs long

ceph health detail | grep -i backfill
ceph osd df | sort -k17 -rn | head -3

Quiz

Production discipline

Compute expansion movement from the post-expansion OSD count, not the capacity increase — the figure is smaller and correct. Pace the rebalance through the diurnal cycle; a single conservative throttle wastes the overnight window and roughly doubles the duration.

Cross-course references

• Kubernetes: rebalancing after node addition follows the same proportional redistribution
• Linux: array expansion movement is likewise a function of the resulting member count