How EC chunks are placed

Why this matters in production

EC chunks are not interchangeable the way replicas are. Chunk 3 must be reconstructed as chunk 3, because the decode arithmetic depends on which position each chunk occupies. That single fact drives a different CRUSH mode, a different acting-set format, and a different failure behaviour — all of which look strange until you know why.

Positions matter

For a replicated PG, acting [12,47,83] means three OSDs hold identical copies. Order determines the primary and nothing else.

For an EC PG, acting [12,47,83,19,55,71] means:

position 0 → osd.12 → data chunk 0
position 1 → osd.47 → data chunk 1
position 2 → osd.83 → data chunk 2
position 3 → osd.19 → data chunk 3
position 4 → osd.55 → coding chunk 0
position 5 → osd.71 → coding chunk 1

The decode needs to know which chunks it has and their positions. Losing the mapping is the same as losing the chunks.

firstn versus indep

Replicated rules use firstn. When a selected OSD is unavailable, CRUSH takes the next candidate and the list shifts left:

healthy:      [12, 47, 83]
osd.47 down:  [12, 83]         ← positions collapse

Harmless for replicas — the surviving copies are still identical.

EC rules use indep. When a selected OSD is unavailable, CRUSH leaves a hole at that position:

healthy:      [12, 47, 83, 19, 55, 71]
osd.83 down:  [12, 47, NONE, 19, 55, 71]

Position 2 stays position 2. The replacement, when one is chosen, becomes chunk 2 and is reconstructed as chunk 2.

ceph osd crush rule dump ec42-rule | jq -r '.steps[]'
# {"op":"take","item":-1,"item_name":"default"}
# {"op":"chooseleaf_indep","num":0,"type":"host"}
# {"op":"emit"}

Reading a degraded EC acting set

ceph pg map 12.a4
# ... -> up [12,47,2147483647,19,55,71] acting [12,47,2147483647,19,55,71]

2147483647 is 0x7FFFFFFF — the sentinel for NONE. Four chunks plus two holes on a 4+2 pool means two chunks are missing: recoverable, but at min_size 5 the PG is not serving.

Failure domain

Set it in the profile, not on the rule, because the profile generates the rule:

ceph osd erasure-code-profile set ec63 \
    k=6 m=3 crush-failure-domain=host

You need k+m failure domains, with headroom above that so recovery has somewhere to place replacement chunks.

Quiz

Production discipline

Count failure domains against k+m before creating any EC pool, and record the headroom in the pool documentation — the shortfall does not produce an error at creation time, only a pool that never goes clean. When reading EC acting sets during an incident, count the NONE entries first; the number is the shortfall against size and tells you immediately how close to min_size you are.

Cross-course references

• Kubernetes: this is the same as a StatefulSet whose pods cannot all be scheduled under anti-affinity constraints
• Linux: RAID stripe position matters for the same reason parity reconstruction does