Editing rules — which changes are safe and which are not

Advanced⏱ ~16 mincephcrushtool

What you'll learn

Classify rule edits by their data-movement cost
Apply low-risk edits confidently
Plan high-cost edits as data-movement events
Avoid the edits that cause silent misbehaviour

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The difference between a five-second change and a three-day rebalance is one line in a rule, and the cluster gives no warning before you apply it.

The classification

Edit	Cost	Notes
Create a new unused rule	none	nothing references it
Rename a rule	none	pools reference ids, not names
Change `choose_total_tries`	none to small	affects retries only
Assign a pool to a different rule	large	placement changes for every PG
Change failure domain in a rule in use	large	every PG remaps
Add a device class to a rule in use	large	different OSD set entirely
Change the `take` root	large	different subtree
Renumber rule ids	catastrophic	pools silently follow the id

The last deserves emphasis: renumbering does not fail, it silently repoints pools at different rules.

Never renumber rules, and verify ids after any map edit

Pools store a rule id. If a hand-edit reorders rules and the ids change, a pool that referenced id 2 now uses whatever rule holds id 2 — possibly a different root, class, or failure domain.

The result is a large unexplained rebalance and a pool with protection nobody chose.

# before any map edit
for p in $(ceph osd pool ls); do
  printf '%s %s\n' "$p" "$(ceph osd pool get $p crush_rule -f json | jq -r .crush_rule)"
done > /tmp/pool-rules.before

# after
# ... same command to /tmp/pool-rules.after
diff /tmp/pool-rules.before /tmp/pool-rules.after

An empty diff is the confirmation. Prefer the CLI rule commands, which cannot renumber.

Applying a high-cost edit

# 1. capture state
ceph osd getcrushmap -o /tmp/cm.before.bin
ceph osd pool get rbd-vms crush_rule > /tmp/rule.before

# 2. build and validate the new rule
ceph osd crush rule create-replicated rack_rule default rack
crushtool -i /tmp/cm.before.bin --test \
    --rule $(ceph osd crush rule dump rack_rule | jq .rule_id) \
    --num-rep 3 --show-bad-mappings

# 3. throttle
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_sleep 0.05

# 4. apply
ceph osd pool set rbd-vms crush_rule rack_rule

# 5. watch
ceph -s

Note that step two creates a new rule rather than editing the existing one. That is deliberate: the old rule remains available for rollback, and no pool changes until step four.

Why editing a rule in place is worse than creating a new one

Editing a rule that pools already use changes their placement the instant the map is applied, with no intermediate state and no way to revert other than editing it back — which moves the data again.

Creating a new rule and reassigning pools separates the two decisions. The rule can be created and validated with no effect. Pools can be moved one at a time, so a cluster with five pools on the old rule does not rebalance all five simultaneously. And rollback is a reassignment rather than another edit.

The only cost is a slightly longer rule list, which is trivial against the control gained. This pattern — create new, validate, migrate, retire old — applies to most Ceph configuration that carries data movement, and it is worth adopting as a habit rather than reasoning about each time.

The edits to avoid entirely

Renumbering rule ids. Use the CLI.
Editing a rule in place when pools use it. Create a new one.
Combining a rule change with a topology change. Do them separately so the cause of any movement is unambiguous.
Applying without crushtool --test. Five seconds against a production incident.

Quiz

Knowledge check · 4 questions

Q1. Which CRUSH rule edit is effectively free of data movement?

Renaming a rule Changing the failure domain of a rule that pools use Adding a device class to a rule in use Changing the take root of a rule in use
Q2. With five pools sharing one rule, creating a replacement rule lets four of them keep their current placement while the fifth is moved and checked.

True False
Q3. Five pools share one rule. The failure domain must change from host to rack for all of them. Plan the migration.

Five pools totalling 600 TB share host_rule. The requirement is rack-level separation. Cluster has 4 racks, roughly balanced. Pools range from 8 TB to 400 TB. Clients are active throughout. The team has a two-week window for the whole migration.
Q4. Explain why renumbering CRUSH rule ids is dangerous and how to detect it.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Adopt create-new, validate, migrate, retire-old as the pattern for every rule change: it makes validation free, lets pools move one at a time, and turns rollback into a reassignment rather than a second rebalance. Never renumber rule ids, and capture every pool’s crush_rule before and after any map edit so an empty diff confirms nothing shifted. And separate rule changes from topology changes, so any movement has an unambiguous cause.

Cross-course references

Ceph: Part XIV (CRUSH Failure Domains) for what the domains cost.
Ceph: Part XCIII (Changing CRUSH Topology) for topology alongside rules.
Ceph: Part LX (Recovery Tuning) for the throttles during migration.