Rule and weight changes that move data — planning the whole event

Advanced⏱ ~17 mincephosdmaptool

What you'll learn

Enumerate the operations that cause data movement
Estimate movement volume before acting
Sequence multiple changes to minimise total movement
Monitor a movement event to completion

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Data movement is the most consequential thing an operator can trigger deliberately. It is also entirely predictable, and treating each of these operations as a planned event rather than a command is what separates smooth clusters from ones that are perpetually rebalancing.

What causes movement

Operation	Typical volume
Adding OSDs	proportional to capacity added
Removing OSDs	proportional to capacity removed
`crush reweight`	proportional to the weight delta
Changing a pool’s `crush_rule`	large — every PG recomputes
Changing failure domain	large
Changing device class of OSDs	large for those OSDs’ pools
Adding a bucket level	large — every walk changes
Changing tunables	large
Raising `pg_num`	metadata split, then movement as `pgp_num` follows
Enabling the balancer	small, ongoing
Setting `osds_per_device` (recreate)	per device, drain and refill

Estimating before acting

ceph osd getmap -o /tmp/before.bin
osdmaptool /tmp/before.bin --test-map-pgs-dump --pool 7 > /tmp/before.txt

# apply the change to a copy of the map, then
osdmaptool /tmp/after.bin --test-map-pgs-dump --pool 7 > /tmp/after.txt
diff /tmp/before.txt /tmp/after.txt | grep -c '^<'

The count of changed mappings against the total PG count is the fraction of the pool that will move. This takes minutes and turns “probably fine” into a number.

Sequence changes to avoid moving the same data twice

Three changes applied separately can move overlapping data three times. The general principles:

Batch related changes into one map update: all weight corrections in one rack, all class fixes for one tier.
Do topology before rules. Adding a rack level and then changing the failure domain in two steps moves data twice; the second step supersedes the first.
Complete one event before starting the next. Wait for active+clean.
Do elective work when the cluster is healthy, never during a degraded state — norebalance exists for the case where a failure interrupts you.

The exception is when control matters more than efficiency: a large change split into stages can be paused between them, which is worth the extra movement on a cluster serving latency-sensitive clients.

Pacing

ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 2
ceph config set osd osd_recovery_sleep 0.1
ceph config set osd osd_recovery_op_priority 1

Set these before applying the change. Setting them after complaints arrive means the first hour ran unthrottled.

Because a rebalance keeps full redundancy, throttling it hard is a safe trade — unlike throttling a recovery, which extends a period of reduced redundancy.

Monitoring to completion

watch -n 30 'ceph -s | grep -E "objects (misplaced|degraded)|recovery"'
ceph osd pool stats
ceph pg dump_stuck

Watch the counts of misplaced objects fall, not the percentage — the percentage moves as the denominator changes and can appear to stall or reverse.

Completion is active+clean for every PG:

ceph pg stat
# e.g. "4096 pgs: 4096 active+clean"

Why a rebalance can appear to stall near the end

The final few percent of a rebalance often takes disproportionately long, and it looks like a fault.

Two causes. Backfill reservations are per-OSD and bounded by osd_max_backfills, so the last PGs may be queued behind each other on a small number of OSDs rather than spread across many — parallelism collapses as the work concentrates.

And PGs that need backfill from a heavily loaded OSD wait for its reservation slot, which can be held for a long time by a large PG.

ceph pg dump | grep backfill_wait | wc -l
ceph pg dump | grep backfilling | wc -l

A large backfill_wait count with few backfilling confirms reservation queuing rather than a stall. Raising osd_max_backfills at that point speeds the tail, and it is safe precisely because the bulk of the work — and its client impact — is already done.

The event checklist

Estimate the movement.
Confirm the cluster is healthy and active+clean.
Set throttles.
Capture rollback state: CRUSH map, pool rule assignments.
Apply.
Monitor counts to zero.
Restore throttles.

Quiz

Knowledge check · 4 questions

Q1. A rebalance is 96% complete and has slowed dramatically. ceph pg dump shows many PGs in backfill_wait and few backfilling. What does this indicate?

A failing OSD is blocking the remaining backfills, holding its reservation until it is marked out Reservation queuing — the remaining work is concentrated on few OSDs, so parallelism has collapsed The cluster has run out of capacity, so the remaining PGs cannot be backfilled to their new OSDs The balancer is competing with the rebalance and moving PGs back towards their previous placement
Q2. Recovery throttles should be set before applying a change that causes data movement, not after client impact appears.

True False
Q3. A cluster needs three changes: correcting 12 wrong OSD weights, adding a rack bucket level, and changing the pool failure domain to rack. Sequence them.

96-OSD cluster, 4 racks, 500 TB stored. Twelve OSDs carry weights for the wrong device size. The CRUSH map has no rack level. The pool must end up with rack-level separation. Clients are active. The team wants minimal total data movement and a clear rollback at each stage.
Q4. Give the checklist for a planned data-movement event.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Treat every one of these operations as a planned event: estimate the movement with osdmaptool, confirm active+clean first, set throttles before applying rather than after complaints, capture rollback state, and watch the misplaced object count rather than the percentage. Sequence changes so topology precedes rules and related edits are batched, since separate applications move overlapping data repeatedly. And read a slow tail as reservation queuing rather than a stall — the backfill_wait count confirms it and raising osd_max_backfills is safe by then.

Cross-course references

Ceph: Part LIX (Backfill) for the mechanism.
Ceph: Part LX (Recovery Tuning) for the throttle settings.
Ceph: Part XCIII (Changing CRUSH Topology) for the live procedure.