The recovery cost of erasure coding

Why this matters in production

Recovery is when a storage system is tested, and EC recovery is where otherwise-sound EC designs fail. The traffic multiplier is exactly k, it is easy to compute in advance, and skipping that computation is the most common EC design error.

The arithmetic

Rebuilding an OSD holding C bytes on a profile with parameter k moves approximately C × k bytes across the cluster network.

The bottom row is the one that ends design discussions. 340 TB on a 25 Gb link is over 30 hours at theoretical full utilisation, which you will not get while also serving clients.

Windows on real networks

Assume 50% of link capacity is available for recovery — optimistic on a busy cluster:

Compare that against the degraded window you are willing to accept. That comparison is the design constraint, and it usually decides both the profile and the network before capacity efficiency gets a vote.

What is actually the bottleneck

During EC recovery, in rough order of likelihood:

1. Cluster network — k× traffic saturates links sized for client load
2. CPU — sustained decode on wide profiles
3. Source device IOPS — k source reads per rebuilt chunk, scattered
4. Recovery throttles — the deliberate limit, often the right answer

# during recovery
CLUSTER_NIC=cluster_nic
ceph -s | grep recovery
iftop -i ${CLUSTER_NIC}
mpstat -P ALL 5
iostat -x 5

Identify which of the four is saturated before adjusting anything.

Design responses

• Smaller OSDs. Halving device size halves per-failure recovery volume. This is the most effective single lever.
• Narrower k. 8+3 moves 20% less than 10+4 per rebuilt byte and needs fewer domains.
• Faster cluster network. Directly buys window.
• A dedicated cluster network, so recovery does not contend with client traffic on the same links.
• The clay plugin, which reduces recovery traffic below k× at some CPU cost — worth evaluating on wide profiles.

Quiz

Production discipline

Put the computed recovery window in the design document next to the capacity figure and treat it as a requirement rather than an outcome. Re-derive it whenever device sizes change — a cluster refreshed onto larger drives can silently move outside its own recovery tolerance without any configuration change at all.

Cross-course references

• Kubernetes: rebuild time for a large stateful workload after node loss follows the same volume-over-bandwidth arithmetic
• Linux: RAID rebuild windows on very large drives are the classic instance of this constraint