Whether the rebuild can finish at all

Why this matters in production

After a permanent node loss the cluster either rebuilds to full redundancy or it does not, and the answer is arithmetic you can do in two minutes. Waiting to find out costs days of exposure.

The placement question comes first

ceph osd pool ls detail | grep -E 'replicated|erasure'
ceph osd crush rule dump | python3 -c '
import sys,json
for r in json.load(sys.stdin):
    types = [s.get("type") for s in r["steps"] if s.get("op","").startswith("chooseleaf")]
    print("%-20s failure domain: %s" % (r["rule_name"], types or "?"))'

# how many hosts still carry up OSDs
ceph osd tree --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
byid = {n["id"]: n for n in d["nodes"]}
live = {n["name"] for n in d["nodes"] if n.get("type") == "host"
        and any(byid.get(c, {}).get("status") == "up" for c in n.get("children", []))}
print(len(live), "hosts with up OSDs:", " ".join(sorted(live)))'

Reading the answer out of the cluster

ceph pg ls undersized | head -5
ceph health detail | grep -A3 PG_DEGRADED

active+undersized+degraded  is serving reads and writes with fewer copies
active+undersized+degraded+remapped+backfilling  is repairing
active+undersized+degraded  with no backfill and no progress  is stuck

# stuck means the rule cannot place the missing copy
ceph pg ls undersized --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
pgs = d.get("pg_stats", d) if isinstance(d, dict) else d
print(len([p for p in pgs if "backfilling" not in p.get("state","")]), "not backfilling")'

What a second failure would do now

ceph osd pool get rbd-vms min_size
ceph pg ls degraded | wc -l

min_size=1 is not a safety setting. It converts an availability outage
into a silent single-copy window, and the next failure is data loss.

The exposure window is a decision input

ceph osd ok-to-stop osd.22
ceph -s | grep -E 'degraded|misplaced'

Quiz

Production discipline

Count surviving failure domains against every pool’s rule before tuning anything — a rebuild that cannot place its last copy will not respond to recovery settings, and the two states look identical in ceph -s. Treat the exposure window as a freeze on unrelated maintenance, and reach for min_size=1 only as a deliberate, time-boxed availability trade.

Cross-course references

• Kubernetes: a PodDisruptionBudget that cannot be satisfied blocks rather than degrading quietly
• Linux: an array below its minimum member count refuses to assemble rather than guessing