What changes when weights change

Intermediate⏱ ~16 minceph

What you'll learn

Predict the volume of data movement from a weight change
Distinguish remapped from degraded during rebalancing
Control rebalancing pace with the appropriate settings
Sequence multiple weight changes safely

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Weight changes are the most common deliberate cause of data movement, and their impact is predictable enough to plan for — if you know what to expect.

What happens

1. Weight changes; osdmap epoch increments
2. CRUSH recomputes; affected PGs get new up sets
3. PGs go active+remapped: acting set still serving, up set differs
4. Backfill copies data to the new OSDs
5. When complete, acting catches up to up; PGs go active+clean

Crucially, the PGs remain active throughout. Data is not at reduced redundancy during a rebalance — the old copies stay in the acting set until the new ones are ready.

ceph -s
ceph pg dump_stuck
ceph osd pool stats

Remapped versus degraded

State	Meaning	Redundancy
`active+remapped+backfilling`	moving to new placement	full
`active+undersized+degraded`	fewer copies than `size`	reduced

A rebalance produces the first. A failure produces the second. Seeing degraded during a planned rebalance means something else has happened as well, and it deserves separate investigation.

Estimate the movement before making the change

Roughly, the fraction of data that moves is the fraction of total weight that changed:

adding 10% capacity     → roughly 10% of data moves
reweighting one OSD from 7.3 to 3.6 in a 96-OSD cluster
                        → roughly 3.6/700 ≈ 0.5% of data moves

For precision, simulate:

ceph osd getmap -o /tmp/before.bin
# apply the change to a copy, then
osdmaptool /tmp/after.bin --test-map-pgs-dump --pool 7 > /tmp/after.txt
diff <(...) <(...)

Knowing whether a change moves 0.5% or 30% of the cluster determines whether it needs a maintenance window, and it is knowable in advance.

Controlling the pace

ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 2
ceph config set osd osd_recovery_sleep 0.1
ceph config set osd osd_recovery_op_priority 1

Lower values mean less client impact and a longer rebalance. Since a rebalance does not reduce redundancy, extending it is a much safer trade than extending a recovery — which is the key operational difference between the two.

The norebalance flag stops rebalancing entirely while allowing degraded PGs to recover:

ceph osd set norebalance
ceph osd unset norebalance

Useful when a failure occurs during a planned rebalance: pause the elective movement and let redundancy restore first.

Why several small weight changes cost more than one large one

Each weight change recomputes CRUSH for the affected PGs. Making three sequential changes to the same subtree means some PGs are remapped three times, and data that moved for the first change may move again for the second.

Batching changes into a single map update avoids the intermediate states:

ceph osd getcrushmap -o /tmp/cm.bin
crushtool -d /tmp/cm.bin -o /tmp/cm.txt
# edit all weights at once
crushtool -c /tmp/cm.txt -o /tmp/cm.new
ceph osd setcrushmap -i /tmp/cm.new

The tension is with control: one large change is more efficient and harder to pause or reverse, while several small ones are slower overall and let you stop after each.

The reasonable compromise is to batch changes that belong to the same logical operation — correcting all wrong weights in one rack — and to sequence changes that are logically separate.

Sequencing

Batch changes that belong together into one map update.
Wait for active+clean between logically separate changes.
Throttle before starting, not after complaints arrive.
Use norebalance if a failure occurs mid-rebalance.

Quiz

Knowledge check · 4 questions

Q1. During a planned rebalance, what PG state should you expect to see?

active+remapped+backfilling, with full redundancy maintained throughout active+undersized+degraded, since data is temporarily missing from new OSDs peering, until the new acting sets are established incomplete, until backfill delivers the first copy to each new OSD
Q2. Batching several weight changes into one map update moves less data than applying the same changes one after another.

True False
Q3. An OSD fails during a planned rebalance that has hours remaining. Decide what to do.

A rebalance from correcting several CRUSH weights has been running for three hours with an estimated four remaining. During it, osd.34 failed. ceph -s now shows both remapped and degraded PGs. Pool size 3, min_size 2, so degraded PGs are still serving. Client latency is already elevated from the rebalance.
Q4. Explain why extending a rebalance is safer than extending a recovery.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Estimate movement before changing weights — roughly the fraction of total weight that changed, or precisely with osdmaptool — since that determines whether a maintenance window is needed. Batch changes belonging to one logical operation into a single map update, because sequential changes remap the same PGs repeatedly. Throttle before starting rather than after complaints. And when a failure interrupts a rebalance, set norebalance so redundancy restores before elective movement resumes.

Cross-course references

Ceph: Part LIX (Backfill) for the movement mechanism.
Ceph: Part LX (Recovery Tuning) for the throttle settings.
Ceph: Part XCV (Maintenance Flags) for norebalance and its relatives.