The dependency order of recovery

Why this matters in production

Under pressure people work on the most visible symptom, and in a multi-failure incident that is usually not the thing blocking everything else.

The order

1. Network — daemons must reach each other
2. Time    — monitors will not form quorum with excessive skew
3. Monitor quorum — nothing else can commit map changes without it
4. Manager — orchestration and metrics; not required for data
5. OSDs    — data availability
6. MDS / RGW — service layers, which depend on OSDs
7. Clients — repoint, remount, restart

# 1. network
PEER_MON=mon-01
ping -c2 ${PEER_MON}
ss -ltn | grep -E ':3300|:6789'

# 2. time
chronyc tracking 2>/dev/null || timedatectl status
ceph health detail | grep -i clock

# 3. quorum
ceph quorum_status --format json 2>/dev/null | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("quorum:", d["quorum_names"], "of",
      [m["name"] for m in d["monmap"]["mons"]])' || echo "NO QUORUM"

# 4-6
ceph -s
ceph osd tree | grep -c up
ceph fs status 2>/dev/null

Recovering out of order

The generic symptom of out-of-order work is effort producing no
observable change. That is the signal to step back and check the layer
below.

# the check that tells you where you actually are
ceph quorum_status >/dev/null 2>&1 && echo "quorum OK" || echo "STOP: fix quorum first"

Establishing position

#!/bin/bash
# where-am-i.sh
echo -n "network to peers: "
ceph quorum_status >/dev/null 2>&1 && echo "reaching monitors" || echo "cannot reach monitors"

echo -n "time skew:        "
ceph health detail 2>/dev/null | grep -qi clock && echo "SKEW PRESENT" || echo "ok or unknown"

echo -n "quorum:           "
ceph quorum_status --format json 2>/dev/null | python3 -c '
import sys,json; d=json.load(sys.stdin); print(len(d["quorum_names"]),"/",len(d["monmap"]["mons"]))'   2>/dev/null || echo "NONE"

echo -n "osds up:          "
ceph osd stat --format json 2>/dev/null | python3 -c '
import sys,json; d=json.load(sys.stdin); print(d["num_up_osds"],"/",d["num_osds"])'   2>/dev/null || echo "unknown"

echo -n "pgs active:       "
ceph pg stat 2>/dev/null | head -1 || echo "unknown"

Sequencing a multi-failure recovery

Work strictly downward. At each layer:
  confirm it is healthy before moving up
  if it is not, that layer is the current work
  resist requests to fix a higher layer first

Quiz

Production discipline

Work strictly downward — network, time, quorum, OSDs, services, clients — and confirm each layer before moving up. Repeated action with unchanged state is the reliable signal that the layer beneath is the actual work.

Cross-course references

• Kubernetes: etcd quorum gates the control plane the same way
• Linux: recovery order follows dependency order, not symptom visibility