Erasure-coded pools in production

Why this matters in production

Erasure coding is how a cluster stores a petabyte of archive without buying three petabytes of disk. It is also the wrong choice for VM disks, and the reasons are specific.

The model

An object is split into k data chunks and m parity chunks. Any k of the k+m chunks reconstruct the object.

k=4 m=2   6 chunks, tolerates 2 losses, 67% efficiency
k=8 m=3  11 chunks, tolerates 3 losses, 73% efficiency
k=2 m=2   4 chunks, tolerates 2 losses, 50% efficiency
size 3   (replication) tolerates 2 losses, 33% efficiency

Compare k=4 m=2 against size 3: the same two-failure tolerance at double the usable capacity.

ceph osd erasure-code-profile set ec42 \
    k=4 m=2 crush-failure-domain=host
ceph osd pool create rgw-data erasure ec42
ceph osd pool set rgw-data min_size 5
ceph osd pool application enable rgw-data rgw

The costs

Failure domains. k+m distinct buckets. k=8 m=3 needs eleven hosts, against three for size 3.

Write amplification for small writes. A partial write to an object requires reading existing chunks, recomputing parity, and writing back — a read-modify-write across several OSDs. Replication just writes.

Recovery cost. Reconstructing a lost chunk requires reading k chunks from k different OSDs. Replication copies one chunk from one OSD.

CPU. Parity computation on every write and every reconstruction.

Latency. Every operation touches k or k+m OSDs rather than size, so the slowest-member effect is worse.

Where EC belongs

Good: RGW data pools, backup and archive pools, CephFS bulk data, anything with large objects written once and read whole.

Poor: RBD for VMs, databases, CephFS metadata, RGW bucket indexes, anything with small random writes.

RBD on EC is supported and it performs poorly for the small-write reason above. Where it is used, it is for cold images rather than active VMs.

Verifying

ceph osd pool ls detail | grep erasure
ceph osd erasure-code-profile get ec42
ceph df                       # compare STORED and USED
ceph pg ls-by-pool rgw-data | head -3

Quiz

Production discipline

Match erasure coding to large-object, write-once, read-whole workloads — RGW data, archives, CephFS bulk — and keep small random writes on replication. Compute the failure-domain requirement of k+m before choosing a profile, since it cannot be changed afterwards without a pool migration. Keep min_size at k+1 and treat lowering it to k as a time-boxed emergency decision. And choose m with recovery duration in mind, not only simultaneous-failure tolerance.

Cross-course references

• Ceph: Part XXV (Erasure Coding Fundamentals) for the mechanism.
• Ceph: Part XXVI (Erasure Coding Trade-offs) for profile selection.
• Ceph: Part XV (CRUSH Maps and Rules) for EC rules.