The CRUSH map — buckets, items, and weights

Why this matters in production

The map is the cluster’s own description of its topology. Reading it directly is the only way to be certain what Ceph believes, and editing it correctly is a skill that pays off on every topology change.

Getting a readable map

ceph osd getcrushmap -o /tmp/cm.bin
crushtool -d /tmp/cm.bin -o /tmp/cm.txt

The sections

# begin crush map
tunable choose_local_tries 0
tunable chooseleaf_vary_r 1
tunable straw_calc_version 1

# devices
device 0 osd.0 class ssd
device 1 osd.1 class ssd
device 2 osd.2 class hdd

# types
type 0 osd
type 1 host
type 3 rack
type 11 root

# buckets
host ceph-01 {
  id -3
  alg straw2
  hash 0
  item osd.0 weight 3.638
  item osd.1 weight 3.638
}

rack rack-a {
  id -9
  alg straw2
  hash 0
  item ceph-01 weight 7.276
}

root default {
  id -1
  alg straw2
  hash 0
  item rack-a weight 7.276
}

# rules
rule replicated_rule {
  id 0
  type replicated
  step take default
  step chooseleaf firstn 0 type host
  step emit
}

Weights

Weights are conventionally the device capacity in TiB. A 4 TB drive carries roughly 3.638.

Crucially, a bucket’s weight is the sum of its children. A host with two 3.638 OSDs weighs 7.276. A rack with four such hosts weighs 29.104. This propagation is what makes CRUSH distribute proportionally to capacity at every level.

ceph osd df tree              # shows weight and utilisation together
ceph osd crush reweight osd.12 7.276

Editing safely

ceph osd getcrushmap -o /tmp/cm.bin
cp /tmp/cm.bin /tmp/cm.bin.backup
crushtool -d /tmp/cm.bin -o /tmp/cm.txt
# edit /tmp/cm.txt
crushtool -c /tmp/cm.txt -o /tmp/cm.new
crushtool -i /tmp/cm.new --test --rule 0 --num-rep 3 --show-bad-mappings
ceph osd setcrushmap -i /tmp/cm.new

The backup and the --test step are the two that matter. Everything else is mechanical.

Reading a map critically

• Do the bucket weights sum correctly from their children?
• Do device weights match the actual device sizes?
• Does every host appear under the intended parent bucket?
• Do the rules reference bucket types that exist and have enough instances?

Quiz

Production discipline

Compare CRUSH weights against actual device sizes as a routine check — a mismatch silently costs capacity and defeats the balancer, and it is a common artefact of OSDs created outside the normal spec. Back up the map and run crushtool --test before every edit. And set tunables to optimal at build time while the cluster is empty; on a populated cluster treat it as a planned data-movement event with a ceph features compatibility check first, because older clients may simply stop connecting.

Cross-course references

• Ceph: Part XV (CRUSH Maps and Rules) for rule syntax.
• Ceph: Part X (Manager Daemons) for what the balancer can and cannot fix.
• Ceph: Part XCIII (Changing CRUSH Topology) for live changes.