The select operation — choosing items at each level

Why this matters in production

The select step is where a CRUSH rule says what the cluster must survive. Getting its arguments wrong produces either undersized PGs or a rule that silently provides less protection than intended.

choose versus chooseleaf

step choose firstn 3 type rack        pick 3 racks, stop there
step chooseleaf firstn 3 type rack    pick 3 racks, then one OSD in each

choose selects buckets of the named type and leaves the result at that level. chooseleaf selects buckets and then descends to a leaf OSD within each.

Most replicated rules use chooseleaf, because the goal is one OSD per failure domain:

rule replicated_rule {
  step take default
  step chooseleaf firstn 0 type host
  step emit
}

choose is used when a rule needs multiple selections at different levels — for example, choosing racks and then several OSDs within each, which is common for erasure coding.

The numeric argument

firstn 0    "as many as the pool needs" — size for replicated, k+m for EC
firstn 3    exactly 3
firstn -1   pool size minus 1

0 is the usual choice because it adapts if size changes. A literal number is used when a rule deliberately splits selection across steps.

firstn versus indep

firstn    on failure, shift subsequent results left
indep     on failure, keep positions stable

For replicated pools, firstn is correct. The acting set is an ordered list where position matters only for choosing the primary, so compacting the list on failure is fine.

For erasure-coded pools, indep is correct and important. Each position in the list corresponds to a specific chunk — position 3 holds data chunk 3. If a failure shifted subsequent entries left, every chunk after the failure would be misidentified.

rule ec_rule {
  step take default
  step chooseleaf indep 0 type host
  step emit
}

Multi-level selection

A rule can select at more than one level, which is how you express “two racks, three OSDs in each”:

step take default
step choose firstn 2 type rack
step chooseleaf firstn 3 type host
step emit

This produces six OSDs across two racks, three hosts per rack. It is the shape used for erasure coding across a limited number of racks, and it needs care: losing one rack loses three chunks at once, so the EC profile’s m must exceed that.

Validating a select step

crushtool -i /tmp/cm.new --test --rule 1 --num-rep 6 \
    --show-mappings --show-bad-mappings | head -20

Read the mappings and confirm the OSDs land where the rule intends — distinct racks, distinct hosts, the right count. This catches both an unsatisfiable rule and one that is satisfiable and wrong.

Quiz

Production discipline

Let Ceph generate erasure-coded rules through the erasure-code-profile and pool creation commands rather than hand-writing them, because indep versus firstn is easy to get wrong and the consequence is corrupted chunk identity. Do the tolerance arithmetic explicitly for multi-level rules: chunks per domain against m decides what a domain failure actually costs. And validate with crushtool --test --show-mappings, which catches rules that are satisfiable and wrong as well as rules that cannot be satisfied.

Cross-course references

• Ceph: Part XXV (Erasure Coding Fundamentals) for k and m.
• Ceph: Part XV (CRUSH Maps and Rules) for the full rule syntax.
• Ceph: Part XXVI (Erasure Coding Trade-offs) for profile selection.