Replicated pools in production

Foundation⏱ ~15 minceph

What you'll learn

Choose size and min_size for a stated requirement
Compute the capacity and bandwidth cost of a replication factor
Predict pool behaviour under failure
Change replication settings safely on a live pool

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Replicated pools serve most Ceph workloads, and their two numbers — size and min_size — decide durability and availability respectively. Both are frequently set once and never examined.

The settings

ceph osd pool set rbd-vms size 3
ceph osd pool set rbd-vms min_size 2
ceph osd pool ls detail | grep rbd-vms

`size`	Usable	Survives	Verdict
2	50%	1 failure, unsafely	not for production data
3	33%	2 failures	the standard
4	25%	3 failures	very large clusters, critical data

min_size 2 with size 3 is the correct default. It guarantees every acknowledged write exists on at least two devices, so it survives any single subsequent failure unconditionally.

The costs

usable capacity   = raw / size
device writes     = client writes x size
network writes    = client writes x (size - 1)
write latency     = max over the acting set

A cluster ingesting 1 GB/s into a size 3 pool writes 3 GB/s to devices and moves 2 GB/s across the cluster network. Sizing from capacity alone misses both.

Behaviour under failure

Failures	State	Serving
0	`active+clean`	yes
1	`active+undersized+degraded`	yes
2	`active+undersized+degraded` at `min_size`	yes, no margin
3	below `min_size`	no

The pool serves throughout until copies fall below min_size, which is why that setting is the availability dial.

Changing settings safely

# raising size: adds copies, causes backfill
ceph osd pool set rbd-vms size 4

# lowering size: removes copies, fast but reduces durability
ceph osd pool set rbd-vms size 2      # think carefully

# min_size is instant, no data movement
ceph osd pool set rbd-vms min_size 2

Raising size requires enough failure domains — a size 4 pool with a host rule needs four hosts — and moves data. Changing min_size is instantaneous and moves nothing, which is why it is the setting reached for during incidents and why that reach deserves discipline.

Why min_size 1 during an incident is a decision with a deadline

Setting min_size 1 restores write availability when a pool has dropped to one copy. It is occasionally the right call — an outage costing more than the residual risk.

What makes it dangerous is duration. The window in which acknowledged writes exist once is not a moment; it lasts until recovery restores a second copy, which on a large pool is hours. Throughout that window, the surviving OSD is under recovery load, and it is likely the same age and batch as the one that failed.

So the decision has two parts and people usually make only the first. Lowering min_size is the visible half. Committing to a specific condition for raising it back — “when recovery reports two copies for all PGs” — is the half that prevents a temporary measure becoming the pool’s configuration.

ceph osd pool set rbd-vms min_size 1     # with a stated deadline
# ... recovery ...
ceph osd pool set rbd-vms min_size 2     # the other half

Verifying

ceph osd pool ls detail
ceph df
ceph pg ls-by-pool rbd-vms | awk '{print $10}' | sort | uniq -c

Quiz

Knowledge check · 4 questions

Q1. What does min_size 2 on a size 3 pool guarantee about an acknowledged write?

That it reached all three OSDs of the acting set before the client was acknowledged That it exists on at least two devices, so it survives any single subsequent failure That two copies are maintained throughout recovery, so backfill never drops below two That reads are served from two replicas and compared before the data is returned
Q2. Setting min_size 1 during an incident is a decision that should include a stated condition for reverting it.

True False
Q3. A team proposes size 2 for a 400 TB backup pool to save capacity. Evaluate.

Backup pool holding 400 TB of nightly backups on 60 HDD OSDs. Current proposal is size 2 with min_size 2, saving 200 TB against size 3. The team argues backups are secondary copies so the risk is acceptable. Restore tests happen quarterly. The drives were purchased as a single batch three years ago.
Q4. State the three costs of replication and how each scales.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Keep size 3 as the production floor and min_size 2 as the default, and reach for erasure coding rather than size 2 when capacity efficiency genuinely matters — size 2 buys 17 percentage points and a data-loss mode. When lowering min_size during an incident, state the condition for raising it back in the same breath, because the risk window lasts until recovery completes rather than for a moment. And size clusters from write rate as well as capacity.

Cross-course references

Ceph: Part XXIII (Replication) for the write path in depth.
Ceph: Part XXVII (Replication vs Erasure Coding) for the choice.
Ceph: Part XIX (PG States) for reading degraded and undersized.