CRUSH map hygiene after removals

Why this matters in production

A CRUSH map accumulates empty buckets, stale rules, and orphaned entries. None breaks anything and together they make the map an unreliable description of the cluster.

The residue

ceph osd crush tree
ceph osd crush rule ls
ceph osd tree | grep -E 'destroyed|DNE'
ceph osd crush class ls

Finding it

# empty buckets
ceph osd crush tree --format json | python3 -c '
import sys, json
def walk(n, path=""):
    p = path + "/" + n["name"]
    kids = n.get("children", [])
    if n["type"] not in ("osd",) and not kids:
        print("empty:", n["type"], n["name"])
    for c in kids:
        if isinstance(c, dict): walk(c, p)
for r in json.load(sys.stdin)["nodes"]:
    if r.get("type") == "root": walk(r)'

# rules not used by any pool
used=$(ceph osd pool ls detail | grep -oE 'crush_rule [0-9]+' | awk '{print $2}' | sort -u)
ceph osd crush rule ls | while read r; do
  id=$(ceph osd crush rule dump "$r" | python3 -c 'import sys,json;print(json.load(sys.stdin)["rule_id"])')
  echo "$used" | grep -qx "$id" || echo "unused rule: $r (id $id)"
done

# destroyed OSDs with no replacement
ceph osd tree | grep destroyed

What to clean and what to keep

# the destroyed OSD id from the `ceph osd tree | grep destroyed` output above:
OSD_ID=13

ceph osd purge "$OSD_ID" --yes-i-really-mean-it

Cleaning safely

# a bucket must be empty before removal
RULENAME=rulename
ceph osd crush remove ceph-07

# a rule must be unused
ceph osd crush rule rm ${RULENAME}

Both commands refuse if the object is in use, which makes the cleanup safe: an attempt that would break something fails rather than succeeding.

# verify the tree afterwards
ceph osd crush tree
ceph -s

Preventing accumulation

Make cleanup part of the removal procedure rather than a separate task:
  remove OSDs
  remove the host
  remove the host bucket
  remove the rack bucket if now empty
  purge any destroyed IDs not awaiting a replacement

# a periodic audit
ceph osd crush tree | grep -B1 -A1 'rack\|host' | head -40
ceph osd tree | grep -cE 'destroyed|DNE'

Quiz

Production discipline

Add CRUSH cleanup to the removal procedure rather than treating it as a separate task — empty buckets and destroyed entries accumulate and make failure domain calculations wrong. The cleanup commands refuse when the object is in use, so the audit is low-risk.

Cross-course references

• Kubernetes: stale resource definitions accumulate the same way and mislead the same way
• Linux: configuration describing hardware that no longer exists misleads every reader