Cascading and correlated failure

Why this matters in production

Every durability calculation you will read assumes failures are independent. They are not. Disks arrive in batches, sit in the same rack, share a power feed, run the same firmware, and experience the same temperature. The gap between the assumption and reality is where clusters actually die.

Sources of correlation

The batch case is the one people miss. Twenty-four drives bought together, installed together, and written to at the same rate reach their wear-out window together. The first failure is a signal about the other twenty-three, not an isolated event.

The cascade shape

A true cascade is worse than simultaneous failure because each step increases the load on the survivors:

1. One OSD fails; recovery starts
2. Recovery drives heavy read load across the remaining OSDs
3. A marginal device, already near failure, fails under that load
4. Recovery load increases further, on fewer devices
5. Repeat

The pattern is recognisable: failures arriving in accelerating succession during a recovery, all in the same rack, host, or batch.

Containment with CRUSH

The failure domain is the containment boundary. With failure_domain=host on a size-3 pool, no PG has two replicas on the same host — so losing an entire host takes at most one replica per PG, and every PG stays above min_size.

ceph osd crush rule create-replicated by-host default host
ceph osd crush rule create-replicated by-rack default rack
ceph osd pool set rbd-vms crush_rule by-rack

The rule to internalise: the failure domain must be at least as large as the largest thing that fails at once. If a rack can lose power, the domain must be rack. Three racks minimum for a size-3 pool with failure_domain=rack.

ceph osd tree            # confirm the hierarchy reflects physical reality
ceph osd crush rule dump # confirm rules use the right domain type

Slowing a cascade in progress

If you are watching failures arrive in succession:

# stop the recovery load that is stressing marginal devices
ceph osd set nobackfill
ceph osd set norecover

# keep the cluster from marking more OSDs out
ceph osd set noout

This is a deliberate pause: it stops the load that is triggering further failures, at the cost of leaving the cluster degraded. Use the pause to assess how many devices are marginal, then resume recovery at a much lower rate.

Quiz

Production discipline

Audit ceph osd tree against the physical layout at least annually and after every hardware move; a CRUSH hierarchy that drifted out of sync with reality is a protection you believe you have and do not. Track drive purchase batches in your asset records so the first failure in a batch can trigger a proactive review rather than a surprise.

Cross-course references

• Kubernetes: pod anti-affinity across zones is the same containment mechanism expressed differently
• Linux: RAID arrays built entirely from one drive batch have the same well-documented correlated-failure history