Correlated failure — why independent probability estimates mislead

Why this matters in production

The usual durability argument goes: each drive has an annualised failure rate of 1%, three replicas must all fail within the recovery window, so the probability is vanishingly small. The arithmetic is correct and the conclusion is wrong, because it assumes the three failures are independent events.

In a real cluster they frequently are not.

Where the independence assumption breaks

Manufacturing batch. Drives bought together often come from one production run. A batch defect affects them all, and they have all been running the same length of time.

Firmware. A firmware bug with a time or write-count trigger fires across every drive with that firmware at roughly the same point. There are documented cases of drives failing at a specific power-on-hour count.

Wear. Flash endurance is consumed roughly evenly by CRUSH, so identical drives commissioned together reach their limits together.

Environment. A cooling failure raises temperature across a rack. Vibration affects a chassis. Both act on many drives at once.

Operators. The single largest correlated cause. A command run against the wrong host, a configuration pushed everywhere, an upgrade applied fleet-wide.

What this changes about design

The mitigations are all about breaking the shared cause:

• Mix vendors or batches across failure domains where procurement allows, so a batch defect cannot take a whole domain.
• Stagger firmware upgrades across failure domains rather than applying fleet-wide, and leave time between them.
• Stagger commissioning, so the fleet does not age as one cohort.
• Spread failure domains across environmental boundaries — different cooling zones where possible.
• Rehearse operator procedures and require them to name the target host explicitly, because the operator-error correlation is not fixed by hardware diversity.

Estimating the effect

There is no clean formula, because correlation coefficients are not knowable in advance. The useful discipline is qualitative:

1. List every property the devices in one failure domain share — model, batch, firmware, age, power, cooling, operator.
2. For each, ask whether a single fault in that property could affect enough devices to exceed the redundancy.
3. Where the answer is yes, either break the sharing or accept the risk explicitly and write it down.

That exercise is more valuable than any probability estimate, because it produces a list of actions rather than a number.

Quiz

Production discipline

Treat the independence assumption as a modelling convenience rather than a property of your cluster. Run the sharing audit at build time: for each failure domain, list what its devices have in common — model, batch, firmware, age, power, cooling, operator — and for each shared property ask whether one fault could exceed the redundancy. Break the sharing where it is cheap to break, and write down what you are accepting where it is not. Two failures close together from one batch is a fleet signal, and the correct first response is procurement, not tuning.

Cross-course references

• Ceph: Part XXVI (Erasure Coding Trade-offs) for stripe width and exposure.
• Ceph: Part III lesson on endurance for wear as a correlated cause.
• Ceph: Part CXIII (Multiple OSD Failure) for the incident response.