Recovery restores redundancy; it does not change the boundary

Why this matters in production

A degraded cluster is serving traffic and looks fine, which makes it easy to treat recovery as background housekeeping. It is not: the entire period between the failure and active+clean is a window in which your failure budget is spent, and the length of that window is one of the few durability parameters you directly control.

Safe versus redundant

Safe — the acting set meets min_size. I/O works and acknowledged writes are durable across the copies that exist.

Redundant — the acting set has size current copies. You can absorb another failure without losing availability.

A degraded PG is safe but not redundant. It works perfectly and it has no margin.

active+clean                 → safe and redundant
active+undersized+degraded   → safe, not redundant
undersized+degraded+incomplete → neither

Recovery does not move the boundary

This is the point people get wrong. Recovery does not make a degraded PG safer while it runs. The PG has two copies at the start of recovery and two copies until the third one is complete. There is no partial redundancy.

What recovery does is end the window. Until it does, a second failure lands on a set with no margin.

The window is a durability parameter

Risk of loss is roughly:

P(another failure) during the recovery window × P(it overlaps on the same PG)

You control the first term directly by shortening the window:

# how much is left, and at what rate
ceph -s | grep -E 'degraded|misplaced'
ceph -s | grep recovery          # objects/s and bytes/s

# speed it up
ceph config set osd osd_max_backfills 4
ceph config set osd osd_recovery_max_active 8
ceph config set osd osd_recovery_sleep_hdd 0

And structurally, by cluster design: more and smaller OSDs recover far faster than fewer large ones, because reconstruction draws from more peers in parallel.

The trade-off, stated honestly

Faster recovery consumes I/O capacity that clients would otherwise use. The defaults favour client latency, which is right for a size-3 pool with one failure and a short window. It is wrong when:

• Recovery would otherwise take days (very large OSDs)
• The pool is already at min_size on some PGs
• You have reason to expect further failures (bad batch, hot rack)

In those cases raising the throttles is the correct call, and you should be able to say why.

Quiz

Production discipline

Instrument time-to-clean and alert when it exceeds the value your durability planning assumed. Include recovery-throttle adjustment in the incident runbook with the criteria for using it, so the on-call engineer does not have to reason it out at 03:00 — and include the instruction to restore the defaults, which is the step that gets forgotten.

Cross-course references

• Kubernetes: MTTR for a failed node is the same exposure-window reasoning
• Linux: RAID rebuild time is the classic instance — and the classic argument against very large single devices