CPU cost of erasure coding

Why this matters in production

Replication costs no CPU worth mentioning — it is a memcpy and a network send. EC costs arithmetic on every write and on every degraded read, and on a cluster sized for replication that arithmetic can become the limit before the disks or the network do.

Where the work happens

Encode — on the primary OSD, on every write. The primary splits the object into k chunks and computes m coding chunks, then distributes all k+m. The full encode cost lands on one OSD process.

Decode — on the reading OSD, only when a chunk is missing. A healthy read concatenates the k data chunks with no computation. A degraded read gathers k available chunks and solves for the missing ones.

Recovery — on the reconstructing OSD, for every rebuilt chunk. This is the sustained case, and it runs for hours.

Rough magnitude

Modern CPUs with SIMD acceleration encode Reed-Solomon at gigabytes per second per core. For most clusters that is comfortably above what the disks can deliver, so EC encode is not the bottleneck under client load.

Where it bites:

• Recovery on wide profiles. Sustained decode across many PGs at once, concurrent with client traffic.
• All-flash clusters. When devices deliver millions of IOPS, the CPU budget per operation gets tight and encode is a real line item.
• Under-provisioned OSD hosts. A host running 12 OSDs on 8 cores has no headroom for sustained encode plus decode.

The plugins

ceph osd erasure-code-profile set ec42 k=4 m=2 plugin=jerasure technique=reed_sol_van
ceph osd erasure-code-profile set ec42i k=4 m=2 plugin=isa technique=reed_sol_van

jerasure is the right default. Change it only with a measurement showing the alternative helps on your hardware and workload.

Recognising CPU as the constraint

# on the OSD host during recovery
top -H -p $(pgrep -d, ceph-osd)
mpstat -P ALL 5

The signature is OSD threads pinned near 100% CPU while device utilisation and network throughput both sit well below capacity. On a replicated cluster that combination is rare; on EC during recovery it is the common failure mode.

Quiz

Production discipline

Include CPU headroom in the EC design document alongside capacity and network, and state it against the recovery case rather than steady state. Monitor OSD host CPU during the first real recovery on an EC pool — that measurement is the one that tells you whether the host sizing was right, and it is cheap to collect once and expensive to guess at.

Cross-course references

• Kubernetes: CPU limits that are fine at steady state and throttle badly during startup storms are the same shape
• Linux: software RAID 6 parity computation has the identical CPU profile