Reasoning about data safety from the cluster map

Why this matters in production

“Is this cluster safe?” deserves a specific answer, not a reassuring one. The answer is derivable from three facts you can read in under a minute, and being able to state it precisely is what makes a capacity or layout review meaningful.

The three facts

ceph osd pool get rbd-vms size min_size crush_rule
ceph osd crush rule dump by-host | jq -r '.steps[]'
ceph osd tree

1. size and min_size — how many copies, and how many are required
2. The rule’s failure domain — what CRUSH treats as a unit
3. How many of those units exist, and how evenly OSDs are spread across them

The derivation

With size 3, min_size 2, failure_domain=host, and 9 hosts:

• Each PG has 3 replicas on 3 distinct hosts
• Losing 1 host: every PG keeps at least 2 → full I/O
• Losing 2 hosts: PGs with replicas on both drop to 1 → those PGs block, the rest serve
• Losing 3 hosts: PGs with all three replicas there → inactive, data intact but unreachable

So the honest statement is: this pool survives the loss of any single host with no client impact, and the loss of any two hosts with partial unavailability confined to the PGs those hosts shared.

Counting failure domains

The rule is simple: you need at least size failure domains, and comfortably more than size for recovery to have somewhere to go.

With exactly three hosts and size 3, losing one host means PGs stay degraded until the host returns; CRUSH has no fourth host to place the third replica on. The cluster is safe but cannot self-heal, which is a materially weaker position than it appears.

Verifying rather than assuming

# does the rule actually use the domain you think?
ceph osd crush rule dump by-rack | jq -r '.steps[] | select(.op|test("choose"))'

# are the OSDs actually spread across the domains?
ceph osd tree | grep -E 'rack|host'

# where does a specific PG actually live?
ceph pg map 7.3d
ceph osd find 47

The last pair is the definitive check: take a PG, look up the OSDs in its acting set, and confirm they are in different domains. If they are not, the rule and the hierarchy disagree with your intent.

Quiz

Production discipline

Write the derived safety statement into the pool’s documentation in plain terms — “survives any one host, partial unavailability on any two” — and re-derive it whenever the rule, size, or hierarchy changes. Include the domain weight balance in the statement, because it is the part that drifts silently as hardware is added and removed.

Cross-course references

• Kubernetes: topologySpreadConstraints with skew limits address exactly the uneven-domain problem
• Linux: verifying a RAID layout against physical enclosure slots is the same assumption-checking exercise