Pre-check: failure domain placement

Why this matters in production

The node’s CRUSH placement determines what failures it shares with other nodes, and correcting it after the OSDs hold data costs a full rebalance.

Establishing the physical position

Questions to answer before adding:
  which rack is it in?
  which power feed?
  which top-of-rack switch?
  which room or row, if the hierarchy has those levels?

The answers must come from the physical installation, not from an assumption about where it was supposed to go.

# what the hierarchy currently expresses
ceph osd tree
ceph osd crush tree

Verifying the hierarchy can express it

ceph osd crush tree --format json | python3 -c '
import sys, json
def walk(n, d=0):
    print("  "*d + n["type"] + " " + n["name"])
    for c in n.get("children", []):
        if isinstance(c, dict): walk(c, d+1)
for r in json.load(sys.stdin)["nodes"][:1]: walk(r)'

# creating a bucket that does not exist
ceph osd crush add-bucket rack4 rack
ceph osd crush move rack4 root=default

Planning the placement

# the intended position, verified before the host exists
ceph osd crush tree | grep -A5 rack4

The plan states:
  hostname → bucket → parent bucket → root
  ceph-07 → rack4 → default

Writing it down before adding means the move is a single known command rather than a decision made after the host appears in the wrong place.

When the topology needs extending

Adding the fourth rack to a three-rack cluster:
  the failure domain may now support a wider rule
  size=3 across 4 racks tolerates a rack failure with a spare
  or size=4 becomes possible

# Substitute your own value before running:
POOL=rbd-vms

ceph osd tree | grep -c 'rack '
ceph osd pool get "$POOL" size
ceph osd pool get "$POOL" crush_rule

Extending the topology is an opportunity to reconsider the rules, and doing so before the new capacity holds data avoids moving it twice.

Quiz

Production discipline

Confirm the node’s rack physically rather than from the plan — a hierarchy describing a layout that does not exist passes every check and fails only when that rack loses power. Extend the hierarchy and reconsider the rules while the new capacity is empty; both changes then share one rebalance.

Cross-course references

• Kubernetes: zone labels assumed rather than verified produce the same silent misplacement
• Linux: topology descriptions must be confirmed against the physical installation