Choosing size: what each value actually buys

Why this matters in production

size is the most expensive number in your cluster: it multiplies every byte you buy. Getting it right is worth real money, and getting it wrong in the cheap direction is worth real data.

The three candidates

The column that matters is the third: how many OSDs in a PG’s acting set can fail while the PG still serves I/O.

Why size 2 does not work

size 2, min_size 2 blocks I/O on any single failure. That is not a production posture; a routine disk failure becomes an outage.

size 2, min_size 1 keeps serving, and this is where it gets dangerous. With one copy left, every write goes to a single OSD. Lose it before recovery completes and those writes are gone. Worse, if the two OSDs diverge — each accepting writes during a partition — Ceph has no way to determine which history is authoritative, because there is no third copy to break the tie.

Use size 2 only for data you can regenerate: caches, scratch, intermediate build artefacts.

Why size 3 is the default

Three copies gives you a genuine failure budget:

• One failure: full I/O, reduced redundancy, recovery starts
• Two failures: I/O blocked (with min_size 2), data intact
• A majority always exists to arbitrate divergence

And the recovery window matters as much as the count. After one failure a size-3 pool has two copies while it rebuilds. The rebuild takes hours. A second failure during those hours is not far-fetched — disks from the same batch, in the same rack, at the same temperature, fail together far more often than independence would predict.

When size 4 is justified

Four copies at 4× capacity is expensive, and it buys tolerance of two concurrent failures with continued I/O. It is justified when:

• Recovery windows are long (very large OSDs, slow rebuild)
• Failure correlation is high (single rack, shared power, one disk batch)
• The data cannot be regenerated and the RPO is zero

For most clusters the money is better spent on more failure domains — a fourth rack, a second power feed — than on a fourth copy. Correlated failure is what actually kills clusters, and independence is cheaper to buy than redundancy.

Quiz

Production discipline

Record the size decision and its justification alongside the pool definition, including the correlated-failure assumptions it rests on — those assumptions change when the cluster moves racks or grows. Report capacity to stakeholders in usable terms with the fill headroom already subtracted, and re-derive the number whenever size or the EC profile changes.

Cross-course references

• Kubernetes: replicas across zones versus more replicas in one zone is the identical trade-off
• Linux: RAID 6 versus RAID 10 turns on the same rebuild-window reasoning