The cost of replication

Intermediate⏱ ~16 minceph

What you'll learn

Compute replication overhead precisely
Compare size=2 and size=3 on cost and risk
Explain why size=2 is discouraged
Choose size per pool deliberately

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

size=3 triples the storage cost of every byte, and the pressure to use size=2 is real and recurring. The argument against it is specific rather than a matter of taste.

The arithmetic

`size`	Raw per logical byte	Usable fraction	Tolerates
2	2×	50%	1 failure (at `min_size=1`)
3	3×	33%	1 failure with `min_size=2`
4	4×	25%	2 failures with `min_size=2`

ceph osd pool get rbd-vms size
ceph osd pool get rbd-vms min_size
ceph df detail

Why size=2 is discouraged

The problem is not the failure tolerance in isolation; it is what happens during a recovery.

size=2, min_size=2:
  one OSD down  → PG below min_size → I/O blocked
size=2, min_size=1:
  one OSD down  → PG serves from one copy
                → any error on that copy is unrecoverable data loss
                → and a scrub inconsistency has no majority to repair from

Both configurations have a failure mode that size=3 does not: with min_size=2 a single failure blocks I/O, and with min_size=1 a single failure leaves the data with no redundancy at all until recovery completes.

Recovery is not instantaneous. On a 16 TB HDD it is hours, and the whole window is spent at zero redundancy.

Where size=2 is defensible

Situation	Defensible?
All-flash with fast recovery and a strong backup	arguably
Data reconstructible from another system	yes
A scratch or cache pool	yes
Production data with no other copy	no

The consistent thread: size=2 is acceptable where the cluster is not the only copy.

Choosing per pool

ceph osd pool set scratch size 2
ceph osd pool set scratch min_size 1
ceph osd pool set rbd-vms size 3
ceph osd pool set rbd-vms min_size 2

Changing size upward triggers backfill of a full additional copy:

ceph osd pool set backups size 3
ceph -s | grep misplaced

Plan that as a data movement, not as a configuration change.

Quiz

Knowledge check · 4 questions

Q1. Beyond surviving a second failure, what does the third replica provide?

Faster reads, because a client reads from whichever of the three copies is least busy Lower write latency, because the primary acknowledges once any two of the three copies commit Reduced recovery traffic, because a lost copy is rebuilt from two survivors in parallel A majority, so a scrub disagreement identifies the correct copy unambiguously
Q2. With size=2 and min_size=1, a single OSD failure leaves the data with no redundancy until recovery completes.

True False
Q3. Respond to a proposal to move production pools to size=2.

A capacity shortfall has prompted a proposal to change the production RBD pool from size=3 to size=2, which would free roughly a third of the used capacity. The data has nightly backups.
Q4. Why should raising a pool's size be planned as a data movement?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Reserve size=2 for data the cluster is not the only copy of; for production data the multi-hour zero-redundancy window after every single failure is the cost, not the 17 points of capacity. Plan any size increase as a data movement equal to one full copy of the pool.

Cross-course references

Kubernetes: two replicas mean a single eviction leaves no redundancy during rescheduling
Linux: RAID1 with two members faces exactly this no-majority repair problem