Scenario modelling: what survives each failure

Why this matters in production

Every design decision assumes some failure will not happen. Writing those assumptions down is what makes them reviewable.

Building the model

# what the cluster's structure is
ceph osd tree
ceph osd crush rule ls
ceph osd crush rule dump | python3 -c '
import sys,json
for r in json.load(sys.stdin):
    dom = [s.get("type") for s in r.get("steps",[]) if s.get("type")]
    print("%-20s failure domain: %s" % (r["rule_name"], dom))'

ceph osd pool ls detail | grep -E '^pool' | head

For each failure scenario, the question is: does a complete copy of
each pool's data survive?

The scenarios

# does the CRUSH topology actually reflect the physical layout?
ceph osd tree | head -30

A cluster whose CRUSH map has every host under one rack bucket cannot
survive a rack failure regardless of what the racks physically are.

# simulate: which PGs would be affected by losing a bucket
ceph osd crush tree --format json | python3 -c '
import sys,json
def walk(n, depth=0):
    print("  "*depth + "%s (%s)" % (n.get("name"), n.get("type")))
    for c in n.get("children", []): pass
for n in json.load(sys.stdin).get("nodes", [])[:15]:
    print("%-6s %-16s %s" % (n.get("type"), n.get("name"), n.get("id")))'

Finding the uncovered scenarios

For each scenario, record:
  what survives
  where it is
  how it would be used
  how long that would take

Two uncovered scenarios are usually acceptable and should be written
down as accepted risk. Two uncovered scenarios nobody has noticed are
not the same thing.

Using the model

The model turns "should we invest in X" into "X covers scenario N,
which is currently uncovered and has impact Y".

Quiz

Production discipline

Verify the CRUSH tree against actual physical placement before trusting a scenario model — pool settings say the cluster survives rack loss even when every host sits under one rack bucket. Write down the uncovered scenarios as accepted risk; that is what makes them decisions.

Cross-course references

• Kubernetes: topology spread constraints only work if node labels reflect reality
• Linux: an assumption written down is reviewable; the same assumption implicit is not