The erasure-coded rule — indep, chunk positions, and profiles

Why this matters in production

EC rules are generated from profiles, which is the right way to create them. The reason to understand the generated result is that inherited clusters have hand-written ones, and the failure mode is subtle.

The form

rule ec42_rule {
  id 2
  type erasure
  step set_chooseleaf_tries 5
  step set_choose_tries 100
  step take default
  step chooseleaf indep 0 type host
  step emit
}

Differences from a replicated rule:

• type erasure rather than type replicated.
• indep rather than firstn.
• Explicit retry settings, because EC rules need more selections and therefore fail more often on marginal topologies.

Why indep

An EC acting set is positional: entry i holds chunk i.

k=4 m=2  →  6 chunks  →  acting set [12, 47, 83, 91, 105, 118]
                          chunk:      0   1   2   3    4    5

If osd.83 fails, firstn would shift positions 3-5 down, so osd.91 would be read as chunk 2 when it holds chunk 3. indep keeps positions stable, leaving a gap:

after failure with indep:  [12, 47, NONE, 91, 105, 118]

The failure-domain requirement

An EC pool needs k + m distinct buckets at its failure-domain level:

k=4 m=2  →  6 chunks  →  6 hosts (or 6 racks for rack-level)
k=8 m=3  → 11 chunks  → 11 hosts
k=2 m=1  →  3 chunks  →  3 hosts

This is the constraint that surprises people: EC needs many more failure domains than replication. A size 3 replicated pool needs three hosts; a k=8 m=3 EC pool needs eleven.

Where domains are scarce, a multi-level rule spreads chunks:

step take default
step choose indep 3 type rack
step chooseleaf indep 2 type host
step emit

Six chunks, two per rack across three racks. Losing a rack loses two chunks, so m must be at least 2 to survive it — and at least 3 for any margin.

Auditing an inherited EC rule

PROFILE=profile
RULE=replicated_rule
ceph osd pool ls detail | grep erasure
ceph osd erasure-code-profile get ${PROFILE}
ceph osd crush rule dump ${RULE}

Check: indep not firstn; the failure domain matches the profile’s crush-failure-domain; enough buckets exist at that level; and m exceeds the chunks lost by a single domain failure.

Quiz

Production discipline

Create EC pools from erasure-code profiles and let Ceph generate the rule, because indep versus firstn is easy to get wrong by hand and the consequence — chunks misidentified during degraded operation — never appears as an error. Audit inherited EC rules for firstn specifically. Compute the domain requirement before creating the pool: k + m distinct buckets, or a multi-level rule with m exceeding what one domain failure costs — because the profile cannot be changed afterwards.

Cross-course references

• Ceph: Part XXV (Erasure Coding Fundamentals) for k and m.
• Ceph: Part XXVI (Erasure Coding Trade-offs) for choosing a profile.
• Ceph: Part XIII (CRUSH Fundamentals) for indep and firstn.