Two OSDs fail on a size-3 pool

Why this matters in production

Two simultaneous failures on a size-3 pool is the scenario that generates the most confused incident calls, because the symptom — “some workloads are down, most are fine” — looks arbitrary until you understand acting-set overlap. It is not arbitrary at all, and the affected set is precisely computable.

Overlap decides everything

For a PG to drop below min_size 2, both failed OSDs must be in its acting set. Most PGs have one of them, or neither.

With 120 OSDs and 4096 PGs at size 3, the fraction of PGs containing both osd.47 and osd.83 is small — on the order of a fraction of a percent under an even distribution. Concretely:

So a two-OSD failure typically blocks a handful of PGs out of thousands. Every client whose objects live in those PGs hangs completely; every other client notices nothing.

Finding the blocked PGs

ceph health detail | grep -E 'incomplete|inactive|down'
ceph pg dump_stuck inactive

Confirm the overlap:

ceph pg dump_stuck inactive --format json | \
  jq -r '.[] | "\(.pgid) up=\(.up) acting=\(.acting)"'

Every blocked PG should show an acting set reduced to one member.

Priorities

1. Restore either failed OSD. One returning copy takes the PG back to two and unblocks it immediately. Even a degraded, read-only, or temporarily-mounted OSD counts.
2. Do not mark the failed OSDs out while you are working on them — that commits to rebuilding from the single survivor.
3. Protect the survivor. For each blocked PG, one OSD holds the only current copy. Nothing should be done that risks it: no restarts, no reweights, no maintenance on its host.
4. Let the merely-degraded PGs recover on their own. They are not the emergency.

What this is not

This is not data loss. All three copies existed; two are unreachable. As long as one survivor holds current data and the failed devices are recoverable — or the survivor stays healthy long enough to rebuild — no data is gone.

The distinction matters when communicating: “some workloads are blocked pending recovery” is a very different statement from “we have lost data,” and jumping to the second one has consequences that are hard to walk back.

Quiz

Production discipline

When communicating during this scenario, be precise about the difference between blocked and lost — the words travel further than you expect. Record which OSDs held the sole surviving copies, because those devices deserve scrutiny afterwards even though they did not fail: they carried the cluster through the incident and their health is now part of your risk picture.

Cross-course references

• Kubernetes: losing two of three etcd members blocks the API for exactly the same quorum reason
• Linux: a two-disk failure in RAID 6 is survivable; the same two failures in RAID 5 are not