size and min_size: what each one controls

Why this matters in production

Almost every “Ceph lost my data” story reduces to a min_size that someone lowered during an incident and never raised again. These two numbers are the entire durability contract of a replicated pool, and they are two words apart in the command line.

The definitions

size — how many copies of each object the cluster maintains. The target.

min_size — how many copies must be available for the PG to accept I/O. The floor.

ceph osd pool get rbd-vms size min_size
# size: 3
# min_size: 2

size is what you want. min_size is what you insist on.

What happens at each boundary

Take size 3, min_size 2 and remove OSDs one at a time:

The critical row is the third. At one surviving copy Ceph stops serving the PG. Clients hang. This looks like a failure and is in fact the system doing exactly what you asked: refusing to accept writes it cannot make safely.

Why blocking is the correct behaviour

If Ceph accepted a write with one copy available, that write would exist in exactly one place. Lose that OSD before recovery completes and the write is gone — not degraded, gone, with no record that it ever existed. Worse, the acknowledged write means the client believes it is durable.

Blocking converts silent data loss into visible unavailability. That is always the right trade for a storage system, because unavailability is recoverable and data loss is not.

Setting them

ceph osd pool set rbd-vms size 3
ceph osd pool set rbd-vms min_size 2

Raising size triggers replication of every object in the pool to an additional OSD — significant data movement. Lowering it deletes copies, which is fast but irreversible in the sense that the redundancy is gone immediately.

ceph config set global osd_pool_default_size 3
ceph config set global osd_pool_default_min_size 2

Quiz

Production discipline

Treat min_size as a value that only ever goes up in normal operations. If an incident forces it down, set a timer and an owner for restoring it in the same breath, and monitor for pools whose min_size is below the standard so a temporary change cannot quietly become permanent.

Cross-course references

• Kubernetes: a PodDisruptionBudget encodes the same floor — how much you may lose and still be serving
• Linux: a RAID array going read-only when redundancy is exhausted is the identical trade