Changing a CRUSH rule

Advanced⏱ ~18 mincephcrushtool

What you'll learn

Classify rule changes by their impact
Compute the movement before applying
Apply a rule change safely
Roll back if needed

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A rule change can move every PG in a pool. Knowing which changes do and computing the volume beforehand is what makes it a planned operation.

Classifying the changes

Change	Impact
Failure domain `host` → `rack`	large; most PGs remap
Failure domain `rack` → `host`	large; most PGs remap
Adding a device class restriction	large if it changes the candidate set
Changing the root	complete; a different set of OSDs entirely
Changing `min_size` or `max_size` in the rule	none for existing valid mappings
Creating a new rule and switching a pool to it	as above, depending on the difference
Editing a rule in place	same, and riskier

RULE=replicated_rule
POOL=rbd-vms
ceph osd crush rule dump ${RULE}
ceph osd pool get ${POOL} crush_rule

Computing the movement

ceph osd getcrushmap -o /tmp/crush.bin
crushtool -d /tmp/crush.bin -o /tmp/crush.txt

# add the new rule to /tmp/crush.txt
rule rack-rule {
    id 2
    type replicated
    step take default
    step chooseleaf firstn 0 type rack
    step emit
}

crushtool -c /tmp/crush.txt -o /tmp/crush-new.bin
crushtool -i /tmp/crush-new.bin --test --rule 2 --num-rep 3 --show-bad-mappings
crushtool -i /tmp/crush.bin --compare /tmp/crush-new.bin --rule 1 --num-rep 3

Two checks: that the new rule can place every PG, and how many mappings differ from the current rule.

Applying safely

# POOL is the pool being moved onto the new rule; substitute your own:
POOL=rbd-vms

# create the new rule alongside the existing one
ceph osd crush rule create-replicated rack-rule default rack

# verify it exists and can place
ceph osd crush rule dump rack-rule

# hold the movement
ceph osd set norebalance

# switch the pool
ceph osd pool set "$POOL" crush_rule rack-rule

# see the volume
ceph -s | grep misplaced

# release when ready, with throttles set
ceph config set osd osd_max_backfills 2
ceph osd unset norebalance

Creating the new rule rather than editing the existing one means the old rule remains available for rollback.

Rolling back

POOL=rbd-vms
OLD_RULE=replicated_rule
ceph osd pool set ${POOL} crush_rule ${OLD_RULE}

Rollback moves the data back, so it costs the same volume again.
Rolling back partway through means the movement reverses from wherever it
reached, which is more total movement than either completing or not
starting.

Situation	Decision
Realised before releasing `norebalance`	switch back; nothing moved
Partway through, and the new rule is wrong	roll back; accept the cost
Partway through, and the new rule is right	complete it
Complete, and it was wrong	roll back as a planned operation

Quiz

Knowledge check · 4 questions

Q1. What does the misplaced count show after switching a pool's rule with `norebalance` set?

Nothing at all until the flag is cleared, because placement is recomputed only when movement starts A rough estimate that the cluster refines as the movement proceeds, so it may be well off at first Only the PGs already in flight when the rule changed, not the full set the new rule affects The exact volume the change will move, computed by the cluster, with no data having moved yet
Q2. Creating a new CRUSH rule and switching one pool to it lets the change be evaluated on that pool before any other pool follows.

True False
Q3. Change a pool from host to rack failure domain.

A pool holding 90 TB needs its failure domain changed from host to rack. Three racks exist.
Q4. Which CRUSH rule changes move little or no data?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Create a new CRUSH rule and switch pools to it rather than editing the existing rule — the old one remains for rollback and each pool’s movement is separate. Switch with norebalance set: the misplaced count is then the cluster’s own computation of the volume, with nothing committed.

Cross-course references

Kubernetes: blue-green configuration changes preserve the rollback path the same way
Linux: staging a change so its effect is visible before committing is general practice