Tolerance is a per-pool property

Advanced⏱ ~18 minceph

What you'll learn

Tabulate size, min_size, and domain for every pool
Compute erasure-coded tolerance correctly
Identify pools with a narrower device reach
State tolerance per pool rather than per cluster

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A cluster does not have a failure tolerance; each pool does, and the one that fails first is rarely the one anybody sized.

Tabulating every pool

ceph osd pool ls detail

ceph osd dump --format json | python3 -c '
import sys,json,subprocess
rules = {r["rule_id"]: r for r in json.loads(subprocess.check_output(
    ["ceph","osd","crush","rule","dump","--format","json"]))}
def dom(rid):
    for s in rules.get(rid, {}).get("steps", []):
        if s.get("type"):
            return s["type"]
    return "?"
for p in json.load(sys.stdin)["pools"]:
    kind = "erasure" if p["type"] == 3 else "replicated"
    print("%-26s %-11s size %2d  min_size %2d  domain %s" %
          (p["pool_name"], kind, p["size"], p["min_size"], dom(p["crush_rule"])))'

cephfs-metadata            replicated  size  3  min_size  2  domain host
rbd-primary                replicated  size  3  min_size  2  domain rack
cold-archive               erasure     size  6  min_size  5  domain host

Erasure coding stops before it breaks

ceph osd erasure-code-profile ls
ceph osd erasure-code-profile get ec42

Pool	Shards	Reconstructable after	I/O stops after
Replicated size 3, min_size 2	3	2 losses	2 losses
EC 4+2, min_size 5	6	2 losses	2 losses
EC 8+3, min_size 9	11	3 losses	3 losses
EC 4+2 with min_size lowered to 4	6	2 losses	3 losses

An EC pool is an availability-tolerates-(m-1), durability-tolerates-m
design. Capacity planning that reads m as the operational tolerance is
planning against the wrong number.

The pool with the narrower reach

ceph osd crush class ls
ceph osd crush class ls-osd ssd | wc -l

for id in $(ceph osd crush class ls-osd ssd); do
  ceph osd find "$id" | python3 -c '
import sys,json; print(json.load(sys.stdin)["crush_location"]["host"])'
done | sort -u

Pool	Typical rule	Typical reach
RBD or RGW data	replicated or EC over all hosts	the whole cluster
CephFS metadata	replicated over the SSD class	the few hosts with SSDs
RGW bucket index	replicated over the SSD class	the same few hosts
`.mgr`	replicated, default rule	the whole cluster

Stating it

Wrong: "the cluster survives one rack."
Right: "rbd-primary survives one rack. cephfs-metadata lives on four SSD
hosts in two racks and survives one host."

Why an erasure-coded pool stops serving before it stops being recoverable

An EC pool of k data and m coding shards can reconstruct an object from any k shards, so the mathematics tolerates m losses. Ceph nonetheless defaults min_size to k+1. The reason is the same one behind min_size 2 on replicated pools: writing while exactly k shards are available produces an acknowledged write with no redundancy at all, and any further loss before recovery makes it unrecoverable. Refusing I/O one failure short of the mathematical limit keeps every acknowledged write reconstructable. The consequence is that an EC 4+2 pool goes inactive at the same point a size 3 replicated pool does, despite storing 50% overhead instead of 200%, and any design that justified EC by its failure tolerance rather than by its capacity efficiency was justified against the wrong property.

Quiz

Knowledge check · 4 questions

Q1. An EC 4+2 pool loses two shards. What is its state at default settings?

Serving normally, since any four of the six shards reconstruct the object Serving reads but not writes, because writes need the full shard set Inactive, because min_size defaults to k+1 and only four shards remain Marked incomplete, and needing an operator to force-create the PG
Q2. An erasure-coded 4+2 pool stops serving I/O after two shard losses even though the data remains reconstructable.

True False
Q3. State the cluster tolerance accurately before a rack maintenance.

A 40-host cluster across four racks is scheduled to lose one rack for electrical work. The RBD pool is size 3 with a rack failure domain. CephFS is in use.
Q4. Why is a device-class rule a tolerance question and not only a performance one?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Tabulate size, min_size, and failure domain for every pool including the small ones — the pool that goes inactive first is usually a metadata pool on a device-class rule reaching a handful of hosts. Compute erasure-coded tolerance from min_size, never from m.

Cross-course references

Kubernetes: a control plane spread across three zones is limited by whichever component is not
Linux: the availability of a system is the availability of its least redundant dependency