Why min_size 1 loses data

Why this matters in production

min_size 1 is the single most effective way to lose data in Ceph. It is also the most tempting command during an outage, because it makes hung clients start working immediately. Understanding exactly what it costs is what lets you refuse it at 03:00 when someone senior is asking for it.

Failure mode one: the last copy dies

Set min_size 1 on a degraded PG with one surviving copy. Writes resume. Every one of them exists on exactly one OSD.

That OSD is, by construction, the survivor of a failure event — same batch, same rack, same power, same age as the ones that already died. If it fails before recovery completes, those writes are gone. Not degraded: gone, with no record they ever existed, and the clients were told they were durable.

Failure mode two: divergence with no arbiter

This one is subtler and worse.

graph TB
    subgraph "t0: partition"
        A[osd.12 - holds PG copy] -.network split.-> B[osd.47 - holds PG copy]
    end
    subgraph "t1: both accept writes at min_size 1"
        C[osd.12: writes A, B, C]
        D[osd.47: writes X, Y, Z]
    end
    subgraph "t2: partition heals"
        E[Two divergent histories, no majority to arbitrate]
    end

With size 3, min_size 2, only the side holding two copies can serve — the other side blocks, and there is no divergence. With min_size 1, both sides can go active. When the partition heals, the cluster has two histories and no basis for choosing. One set of acknowledged writes is discarded.

The majority requirement is not bureaucratic caution; it is the mechanism that makes the choice determinate.

The narrow legitimate uses

Regenerable data. A pool holding a rebuildable cache. Losing it costs CPU, not data.

Deliberate last-resort recovery. The cluster is already below min_size, the data exists on one OSD, and the alternative is a restore from backup. Then:

ceph osd pool set broken-pool min_size 1
# ... let clients read what they need, or let recovery start ...
ceph osd pool set broken-pool min_size 2      # the moment a second copy exists

Treat this as a data-recovery operation with a stated end condition, not as a configuration change.

What to do instead

When someone is asking for min_size 1 because clients are hung:

Most requests for min_size 1 turn out to be recovery that was throttled, paused by a forgotten flag, or blocked by a full OSD. Check all three before touching the number.

Quiz

Production discipline

Monitor for pools with min_size below the standard and alert on them continuously, not just at change time — the dangerous state is the one that persists after the incident everyone has forgotten. Rehearse the alternatives so the on-call engineer has something concrete to offer when the pressure to lower it arrives, because “no” is much easier to say alongside a list of things to try instead.

Cross-course references

• Kubernetes: forcing a quorum-based operator to accept a single member has the identical split-brain outcome
• Linux: mounting a filesystem with -o skip-checks to get past errors trades correctness for availability the same way