How much more you can afford to lose

Advanced⏱ ~18 minceph

What you'll learn

Derive remaining tolerance from the pool and rule
Count PGs sitting at exactly min_size
Interpret ok-to-stop correctly
Communicate tolerance without overstating it

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Every action taken during a host outage — a reboot, a firmware update, a scrub — is a bet on how much redundancy is left, and the number is knowable rather than guessable.

Where tolerance comes from

ceph osd pool ls detail | grep -E 'replicated|erasure'
ceph osd crush rule dump replicated_hdd | python3 -c '
import sys,json
d = json.load(sys.stdin)
for s in d["steps"]:
    print(s)'

{"op": "take", "item": -1, "item_name": "default"}
{"op": "chooseleaf_firstn", "num": 0, "type": "host"}
{"op": "emit"}

Rule element	What it fixes
`type host` in `chooseleaf`	one replica per host
pool `size`	how many domains a PG occupies
pool `min_size`	how few it can occupy and still serve
`size - min_size`	guaranteed domain failures before I/O stops

Counting what is left

ceph pg dump pgs --format json 2>/dev/null | python3 -c '
import sys,json,collections
d = json.load(sys.stdin)
rows = d.get("pg_stats", d) if isinstance(d, dict) else d
c = collections.Counter(len(p.get("acting", [])) for p in rows)
for n in sorted(c):
    print("acting set of %d: %6d PGs" % (n, c[n]))'

acting set of 2:    376 PGs      <- one more loss takes these below min_size
acting set of 3:   3721 PGs

ceph pg ls undersized | head
ceph pg dump_stuck degraded | head

The exposed PGs are not spread evenly. Find which hosts hold their surviving replicas, because those are the machines that must not be touched until recovery completes.

ceph pg dump pgs --format json 2>/dev/null | python3 -c '
import sys,json,collections,subprocess
d = json.load(sys.stdin)
rows = d.get("pg_stats", d) if isinstance(d, dict) else d
tree = json.loads(subprocess.check_output(["ceph","osd","tree","--format","json"]))
host = {}
for n in tree["nodes"]:
    if n["type"] == "host":
        for c in n.get("children", []):
            host[c] = n["name"]
c = collections.Counter()
for pg in rows:
    a = pg.get("acting", [])
    if len(a) == 2:
        for o in a: c[host.get(o, "?")] += 1
for h, n in c.most_common(10):
    print("%-24s %5d exposed PGs" % (h, n))'

Asking before acting

ceph osd ok-to-stop 44
ceph osd ok-to-stop 44 45 46 --max 3
ceph orch host ok-to-stop ceph-osd-09

ceph osd safe-to-destroy 18 19 20

Command	Question it answers
`ceph osd ok-to-stop`	would stopping these make PGs unavailable
`ceph orch host ok-to-stop`	the same question for every OSD on a host
`ceph osd safe-to-destroy`	do these OSDs still hold copies nothing else has

Saying it accurately

Wrong: "we can lose two more hosts, we have 40."
Right: "376 PGs are at two copies; any host holding one of those is a
service outage for that PG. Everything else tolerates one more host."

Why adding hosts changes the blast radius rather than the tolerance

A rule that places one replica per host with size 3 guarantees service through exactly one host failure, whether the cluster has four hosts or forty. The guarantee comes from size - min_size, which no amount of hardware changes. What more hosts do change is the distribution of the second failure: with four hosts every PG uses three of them, so a second loss affects nearly everything, while with forty hosts only the PGs that happened to place replicas on both failed machines are affected — a small fraction. Large clusters therefore degrade partially where small ones degrade completely, which is a real and valuable property, but it is graceful degradation and not additional tolerance. Stating it as tolerance is how teams talk themselves into concurrent maintenance during an outage.

Quiz

Knowledge check · 4 questions

Q1. What does `ceph osd ok-to-stop` actually evaluate?

Whether the OSDs hold the only surviving copies of any object in the cluster Whether stopping them would take PGs below min_size and make them inactive Whether the survivors have the free capacity to absorb the data those OSDs hold Whether the cluster is currently HEALTH_OK and no recovery is already running
Q2. A forty-host cluster with size 3 and min_size 2 tolerates more host failures than a four-host cluster with the same settings.

True False
Q3. Assess whether a second maintenance action is safe.

One host is down and recovery is in progress. A firmware update is scheduled for another host tonight. 376 PGs currently show an acting set of two.
Q4. Where does guaranteed failure domain tolerance come from?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Express remaining tolerance as a count of PGs at their minimum acting set, not as a count of spare hosts — the two diverge badly in large clusters. Confirm min_size on the affected pools before trusting ok-to-stop, since it is an availability check and not a durability one.

Cross-course references

Kubernetes: PodDisruptionBudget answers availability and says nothing about data redundancy
Linux: quorum arithmetic does not improve because you added more machines