Reading the shape of a correlated failure

Why this matters in production

Twenty OSDs down in one rack and twenty scattered across the cluster are opposite problems, and the health output presents them identically.

Grouping by position

ceph osd tree down

ceph osd tree --format json | python3 -c '
import sys,json,collections
t = json.load(sys.stdin)
node = {n["id"]: n for n in t["nodes"]}
parent = {}
for n in t["nodes"]:
    for c in n.get("children", []):
        parent[c] = n["id"]
def up(i, want):
    while i in parent:
        i = parent[i]
        if node[i]["type"] == want:
            return node[i]["name"]
    return "-"
c = collections.Counter()
for n in t["nodes"]:
    if n["type"] == "osd" and n.get("status") == "down":
        c[(up(n["id"], "rack"), up(n["id"], "host"))] += 1
for (r, h), k in sorted(c.items()):
    print("%-12s %-22s %3d down" % (r, h, k))'

Dating it from the map

ceph osd dump | grep -E '^osd[.][0-9]+ ' | head -5

osd.23 down out weight 0 up_from 4412 up_thru 4478 down_at 4501 last_clean_interval [4380,4500)

ceph osd dump --format json | python3 -c '
import sys,json,collections
c = collections.Counter(o["down_at"] for o in json.load(sys.stdin)["osds"]
                        if o["up"] == 0)
for e, n in sorted(c.items()):
    print("osdmap epoch %-8d %3d OSDs down" % (e, n))'

The map is a declaration, not a survey

ceph osd crush tree
ceph osd crush tree --show-shadow | head -20

Stabilising first

ceph osd set nodown
ceph log last 200 info cluster | grep -i 'osd[.]'
ceph osd unset nodown

`nodown` stops the monitors acting on failure reports while a fabric is
being repaired. It does not fix anything; it stops the map churning while
you work.

Quiz

Production discipline

Group down OSDs by CRUSH position and by down_at epoch before touching anything — the shape of the failure decides the response. Expect no automatic recovery when a whole rack is down; the subtree limit suppresses it until an operator marks the OSDs out.

Cross-course references

• Kubernetes: zone labels describe intent, and the scheduler cannot verify the physical topology
• Linux: two power supplies on one circuit are one power supply