Placement decisions — turning failure domains into pool configuration

Intermediate⏱ ~16 minceph

What you'll learn

Choose size and min_size for a stated fault-tolerance requirement
Explain the availability consequence of each min_size value
Match a CRUSH rule to the failure domain the design requires
Verify that a pool actually delivers its intended tolerance

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Everything in this part converges here. size, min_size, and the CRUSH rule are the three settings that translate a failure-domain design into cluster behaviour, and each one has a cost that is worth being explicit about.

The three settings

ceph osd pool set rbd-vms size 3
ceph osd pool set rbd-vms min_size 2
ceph osd pool set rbd-vms crush_rule rack_replicated

size — how many copies exist. Determines durability and cost.
min_size — how many copies must be available for the pool to accept writes. Determines availability and safety.
crush_rule — which failure domain the copies are separated across. Determines what kind of failure the copies survive.

Reading min_size correctly

This is the setting most often misunderstood.

size	min_size	Writes continue with	Risk
3	2	1 failure	sound default
3	1	2 failures	writes to a single copy; a further failure loses data
3	3	0 failures	any single failure blocks writes
2	1	1 failure	writes to a single copy routinely
2	2	0 failures	any failure blocks writes

min_size 2 with size 3 is the right default because it keeps writes flowing through one failure while guaranteeing every acknowledged write exists on at least two devices.

Matching the rule to the design

ceph osd crush rule create-replicated rack_replicated default rack
ceph osd pool set rbd-vms crush_rule rack_replicated

The requirement is that the number of buckets at the chosen level is at least size. Three racks for a size 3 rack rule; five for a size 5. Fewer and CRUSH cannot satisfy the rule, and PGs stay undersized indefinitely.

Verifying rather than assuming

The check that actually proves tolerance:

# Which OSDs hold a given PG
ceph pg map 3.1f

# Where those OSDs live
ceph osd find 12
ceph osd tree

# Whether any PG has replicas in one domain
ceph pg dump | awk '{print $1, $17}' | head -30

Pick several PGs at random, map their acting sets onto the tree, and confirm the replicas sit in distinct failure domains. If they do not, the rule is not doing what the design assumed.

Why min_size 2 with size 3 is not arbitrary

Consider what an acknowledged write means. With min_size 2, Ceph does not acknowledge until at least two OSDs in the acting set have committed. So every acknowledged write survives the loss of any single device, unconditionally.

With min_size 1 there is a window in which an acknowledged write exists on exactly one device. That window is short in normal operation, and during degraded operation — precisely when min_size 1 gets set — it is not short at all: the pool may run for hours with single copies while recovery proceeds.

The other direction is equally principled. min_size 3 with size 3 means every write requires all three, so the pool has zero fault tolerance for writes even though it has three copies. It converts a durability setting into an availability liability, which is almost never what the operator intended.

The decision, written out

For each pool, the design should state:

The event it must survive (device, host, rack, power domain).
size, and the raw capacity that implies.
min_size, and the availability that implies.
The CRUSH rule, and the count of buckets at that level.
The verification that the acting sets actually span those buckets.

Five lines per pool. They are the difference between a cluster that is designed to survive something and one that is assumed to.

Quiz

Knowledge check · 4 questions

Q1. A size 3 pool has min_size set to 3. What is the consequence?

Durability improves, because all three copies have to be written before acknowledgement Any single OSD failure blocks writes to affected PGs, giving zero write fault tolerance Recovery completes faster, because no PG is allowed to operate while it is degraded The pool behaves identically to min_size 2 except while recovery is running
Q2. Setting min_size 1 during an incident to restore write availability creates a window in which acknowledged writes exist on only one device.

True False
Q3. A pool configured with size 3 and a rack-level CRUSH rule shows many PGs stuck undersized+degraded on a healthy cluster. Diagnose.

Cluster has 20 hosts. CRUSH map defines two racks: rack-a with 12 hosts and rack-b with 8. Pool rbd-vms has size 3, min_size 2, crush_rule rack_replicated created against type rack. All OSDs are up and in. ceph health shows PG_DEGRADED with a large number of undersized PGs. No hardware has failed.
Q4. List the five things a pool design should state, and explain what verification actually proves tolerance.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Write five lines per pool: the event it must survive, size, min_size, the CRUSH rule with the bucket count at that level, and the verification result. Keep min_size at 2 for size 3 as the default, treat min_size 1 as a time-boxed emergency decision that must be reverted, and never set min_size equal to size. Then prove the tolerance by mapping real PGs onto the tree rather than by reading the rule — undersized PGs on a healthy cluster mean the rule asks for more failure domains than exist.

Cross-course references

Ceph: Part XVII (Pools) for the full set of pool settings.
Ceph: Part XIX (PG States) for reading undersized and degraded correctly.
Ceph: Part XXIII (Replication) for how min_size affects the write path.