Every change that triggers backfill

Advanced⏱ ~17 minceph

What you'll learn

Enumerate the operations that trigger backfill
Estimate the data volume each produces
Sequence changes to minimise total movement
Recognise unintended backfill triggers

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Several routine operations move data, and some are not obviously placement changes. Knowing the catalogue means a change is planned as a data movement rather than discovered to be one.

The catalogue

Operation	Volume moved
Add an OSD	proportional to the new OSD’s share
Remove an OSD	that OSD’s contents
Purge an OSD	additionally, the bucket weight change
Add a host	proportional to the host’s share
Change a CRUSH rule	potentially the whole pool
Change the failure domain	potentially the whole pool
Change a device class	every pool using rules that select it
Increase `pg_num`	roughly half the pool
Decrease `pg_num`	roughly half the pool
Change `size`	one full copy of the pool
Reweight an OSD	proportional to the weight change
Enable the balancer	proportional to the existing imbalance

The pg_num and size rows are the ones that surprise people: both are single commands with pool-wide consequences.

Estimating the volume

# an OSD's contents
ceph osd df | grep '^ *13 '

# a pool's total
ceph df detail

# how much a change moved, after the fact
ceph -s | grep misplaced

For a CRUSH rule change, the movement is not easily predicted — it depends on how the new rule’s output differs from the old, which is pseudo-random. Testing on a copy of the CRUSH map is the reliable approach:

ceph osd getcrushmap -o /tmp/crush.bin
crushtool -d /tmp/crush.bin -o /tmp/crush.txt
# edit
crushtool -c /tmp/crush.txt -o /tmp/crush-new.bin
crushtool -i /tmp/crush-new.bin --test --show-mappings --rule 1 --num-rep 3

crushtool --compare between the old and new maps reports how many mappings change, which is the volume estimate.

Sequencing to minimise movement

Several changes applied together move less than the same changes applied individually, because intermediate states are skipped:

# individually: three rebalances
ceph osd purge 13 --yes-i-really-mean-it
ceph osd purge 14 --yes-i-really-mean-it
ceph osd purge 15 --yes-i-really-mean-it

# together: one
for i in 13 14 15; do ceph osd out $i; done
# wait for the drain
for i in 13 14 15; do ceph osd purge $i --yes-i-really-mean-it; done

The same applies to adding hosts: adding six hosts at once produces one rebalance rather than six.

Unintended triggers

Action	Unintended consequence
Enabling the autoscaler on a pool	`pg_num` change and its movement
Enabling the balancer	movement proportional to the imbalance
Changing a device class on an OSD	every pool with a class-scoped rule
Adding a host with the wrong label	services and OSDs deployed unintentionally

Quiz

Knowledge check · 4 questions

Q1. How much data does increasing a pool's pg_num typically move?

None, because splitting a PG happens locally on its OSDs The entire pool, since every object is rehashed onto new PGs An amount proportional to the number of OSDs in the pool Roughly half the pool, as the new PGs are placed by CRUSH
Q2. The volume a CRUSH rule change will move has to be computed rather than reasoned about.

True False
Q3. Plan a set of cluster changes to minimise movement.

A maintenance window includes: retiring four old OSDs, adding two new hosts, and changing one pool from failure_domain=host to failure_domain=rack. All are approved.
Q4. Why does purging several OSDs together cost less than purging them individually?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Batch OSD and host changes so intermediate placements are skipped; individual changes move data to states the final configuration does not require. Estimate CRUSH rule changes with crushtool --compare before committing — the volume cannot be reasoned about and can be computed.

Cross-course references

Kubernetes: batching node pool changes avoids repeated rescheduling waves
Linux: combining LVM operations into one commit avoids intermediate data movement