The OSD down alert and its response

Why this matters in production

One OSD down is routine and the cluster handles it. Twelve OSDs down is a host failure. Forty is something else. The response scales with the count and the first step is establishing it.

Establishing the scope

ceph -s
ceph osd tree | grep -E 'down|host'
ceph health detail | grep -o 'osd\.[0-9]*' | sort -u | wc -l

# grouped by host — the question that matters
ceph osd tree --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
nodes = {n["id"]: n for n in d["nodes"]}
hosts = {}
for n in d["nodes"]:
    if n["type"] == "host":
        down = [nodes[c]["name"] for c in n.get("children",[])
                if nodes.get(c,{}).get("status") == "down"]
        if down: hosts[n["name"]] = down
for h, o in hosts.items(): print(h, len(o), o[:6])'

Scaling the response

The last row is important: OSDs marked down across many hosts is more often a network or monitor issue than mass device failure.

ceph osd tree | awk '/down/ {print}' | wc -l
ceph -s | grep -E 'mon|quorum'

Waiting versus intervening

ceph config get osd mon_osd_down_out_interval

The default is 600 seconds. Within that window the cluster is waiting to see whether the OSD returns; recovery has not started. Intervening means choosing between:

# expect it back soon — prevent the rebalance
ceph osd set noout

# it is not coming back — start recovery now
ceph osd out 44

The decision rests on whether the OSD will return within a time comparable to the recovery duration. A host rebooting returns in minutes and noout is right; a failed device does not return and waiting only extends the degraded window.

Actions that make it worse

Quiz

Production discipline

Group down OSDs by host before doing anything else — the count alone does not distinguish a device failure from a host failure from a fabric event. Check the network and monitor state first when OSDs are down across many hosts; simultaneous mass device failure is rare.

Cross-course references

• Kubernetes: NotReady nodes across racks point at the network, not the nodes
• Linux: correlated failures across independent hardware implicate the shared layer