EC recovery and what it costs

Why this matters in production

Replicated recovery is a copy. EC recovery is a computation over a gathered set, and the difference is not incremental — it is a multiplication of network traffic, disk reads, and CPU that reshapes how you plan capacity and windows for EC pools.

Replicated recovery

Missing replica of a 4 MB object. Read 4 MB from one surviving OSD, write 4 MB to the destination. One source, one stream, no computation.

EC recovery

Missing chunk 2 of a 4+2 object. To rebuild that one 1 MB chunk:

graph LR
    A[osd.12 chunk0 1MB] --> R[reconstructing OSD]
    B[osd.47 chunk1 1MB] --> R
    C[osd.19 chunk3 1MB] --> R
    D[osd.55 code0 1MB] --> R
    R -->|decode| E[chunk2 1MB written]

Read k chunks — 4 MB total — from 4 different OSDs, run the decode, write 1 MB. To produce 1 MB of data the cluster moved 4 MB across the network and touched four devices.

The amplification, quantified

Wider profiles are more capacity-efficient and more expensive to recover. 10+4 saves more disk than 4+2 and costs 2.5× the recovery traffic per unit.

Degraded reads pay the same cost

This is the part that catches people. While a chunk is missing, every read of an affected object performs the same gather-and-decode:

• Healthy read: fetch k data chunks, concatenate, done
• Degraded read: fetch k available chunks including coding chunks, decode, then return

So a degraded EC pool is not just slower to recover — it is slower to read, for the whole duration, for every client. On a replicated pool a degraded read is served by the primary at normal speed.

Planning consequences

# EC recovery deserves more conservative throttles than replicated
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 2

# and CPU headroom matters
ceph osd df | head          # check per-OSD utilisation

For a given raw capacity, budget several times the recovery window you would allow for a replicated pool, and confirm the cluster network has the headroom — EC recovery is the workload most likely to saturate a cluster network that looked adequate under client load.

Quiz

Production discipline

Measure actual recovery throughput on an EC pool once, in a controlled test, and use that number rather than an estimate in your planning. Set EC recovery throttles separately from replicated ones where your deployment allows it, and communicate to workload owners that degraded-mode read performance is part of the EC trade — it is the aspect they will notice and the one least often mentioned in advance.

Cross-course references

• Kubernetes: rebuilding a distributed cache from peers has the same gather-then-compute shape
• Linux: RAID 6 rebuild reads every surviving member for the same reason