CRUSH — placement without a lookup table

Intermediate⏱ ~16 mincephcrushtool

What you'll learn

Explain how CRUSH computes placement from object name and cluster map
Describe the role of weights and how they influence distribution
Predict what a topology or weight change does to data placement
Use CRUSH tooling to inspect and simulate placement

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

CRUSH is a function, not a directory. That single fact explains why Ceph scales without a metadata bottleneck, why topology changes move data, and why you can compute any object’s location from a laptop holding a copy of the map.

The computation

object name  ──hash──▶  pg_id  ──CRUSH(map, rule, pg_id)──▶  [osd list]

Two stages. The first is a simple hash into pg_num buckets. The second walks the CRUSH hierarchy according to the rule, selecting buckets pseudo-randomly but deterministically, weighted by capacity.

Everyone with the same map computes the same answer. No coordination, no lookup, no state.

ceph osd map rbd-vms myobject
# osdmap e41207 pool 'rbd-vms' (7) object 'myobject' ->
#   pg 7.b1f2c3d (7.3d) -> up ([12,47,83], p12) acting ([12,47,83], p12)

Weights

Two different weights exist and conflating them causes confusion:

CRUSH weight — the capacity share, conventionally the device size in TiB. Set at OSD creation and changed with ceph osd crush reweight. Changing it changes the CRUSH computation itself.

Reweight — a 0.0 to 1.0 override applied after CRUSH placement, set with ceph osd reweight. It is a correction mechanism, not a capacity statement, and it is what ceph osd reweight-by-utilization adjusts.

ceph osd df tree           # shows both, plus utilisation
ceph osd crush reweight osd.12 7.2
ceph osd reweight osd.12 0.9

Prefer the balancer to manual reweighting

Manual reweight was the standard way to fix uneven distribution before the balancer module existed. It still works and it is now usually the wrong tool: manual weights are static, so they become stale as the cluster changes, and they interact confusingly with the balancer if both are in use.

ceph balancer status
ceph balancer mode upmap
ceph balancer on

The balancer in upmap mode adjusts individual PG mappings rather than weights, which is more precise and self-maintaining. Reset manual reweights to 1.0 before enabling it, or the two will fight.

Why change moves data

Because placement is computed from the map, any change to the map changes some placements. Adding an OSD changes the weighted selection at every level it participates in; removing one does the same.

CRUSH uses “straw2” bucket selection specifically to minimise this: the theoretical ideal is that adding capacity moves only the data that should live on the new capacity, and straw2 approaches that closely. Older algorithms moved substantially more.

The practical consequence is that expansion is proportional: adding 10% capacity moves roughly 10% of data, not all of it.

Why CRUSH sometimes cannot satisfy a rule, and what it does then

CRUSH descends the hierarchy choosing distinct buckets of the required type. If it cannot find enough — because there are fewer buckets than replicas, or because chosen buckets have no available OSDs — it retries with a different pseudo-random path, bounded by choose_total_tries (50 by default).

If every retry fails, CRUSH returns a short list and the PG is marked undersized. There is no error message explaining why; the cluster simply reports fewer replicas than requested.

This is why undersized on a cluster with no failures nearly always means an unsatisfiable rule, and why crushtool --test --show-bad- mappings is the diagnostic: it simulates placement and reports exactly which PGs could not be fully mapped, before the map is applied.

Simulating before applying

ceph osd getcrushmap -o /tmp/cm.bin
crushtool -d /tmp/cm.bin -o /tmp/cm.txt
# edit /tmp/cm.txt
crushtool -c /tmp/cm.txt -o /tmp/cm.new
crushtool -i /tmp/cm.new --test --rule 1 --num-rep 3 --show-bad-mappings
ceph osd setcrushmap -i /tmp/cm.new

The --test step is the difference between discovering a broken rule at a desk and discovering it on a production pool.

Quiz

Knowledge check · 4 questions

Q1. What is the difference between ceph osd crush reweight and ceph osd reweight?

They are two names for one operation, differing only in the range of values each accepts and in the units crush reweight applies only to newly written data, while reweight also relocates data already stored crush reweight changes the capacity share used by CRUSH; reweight is a 0-1 override applied after placement crush reweight persists in the CRUSH map, while reweight is held in memory and resets when the OSD restarts
Q2. Adding 10% more capacity to a Ceph cluster typically moves roughly 10% of the data.

True False
Q3. A cluster reports undersized PGs after a CRUSH rule change, with no OSD failures. The team wants to know why CRUSH did not simply pick different OSDs.

The rule was changed from failure domain host to failure domain rack for a size 3 pool. The CRUSH map defines three racks, but rack-c contains only two hosts and both were recently drained for maintenance with ceph osd out. All OSDs report up. Roughly a third of PGs in the pool are undersized.
Q4. Explain why CRUSH being a function rather than a lookup table has both a scaling benefit and a diagnostic benefit.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Use the balancer in upmap mode rather than manual reweighting, and reset any existing manual reweights to 1.0 first so the two do not fight. Run crushtool --test --show-bad-mappings against every proposed map change, since an unsatisfiable rule produces undersized PGs with no explanation. And plan failure-domain rules together with maintenance procedures — draining a whole domain is precisely the operation a domain-level rule cannot absorb.

Cross-course references

Ceph: Part XIII (CRUSH Fundamentals) for the algorithm in depth.
Ceph: Part XV (CRUSH Maps and Rules) for authoring rules.
Ceph: Part X (Manager Daemons) for the balancer module.