Two nodes gone at once

Why this matters in production

A single node loss is a rebuild. Two at once may be a rebuild, an outage, or a restore, and the three are told apart by PG state rather than by how alarming the health output looks.

Which PGs are inactive

ceph -s
ceph health detail | grep -E 'PG_AVAILABILITY|PG_DEGRADED|OBJECT_UNFOUND'

ceph pg ls incomplete
ceph pg ls stale
ceph pg ls down

Degraded is a schedule problem. Peered, incomplete and down are outages,
and they are outages only for the objects in those PGs.

Reading a PG that will not peer

ceph pg 4.1f3 query | python3 -c '
import sys,json
d = json.load(sys.stdin)
rs = d.get("recovery_state", [])
print("state:", d.get("state"))
for s in rs[:2]:
    print("-", s.get("name"))
    for b in s.get("blocked_by", []):
        print("  blocked_by osd:", b)
    for p in s.get("peer_info", [])[:3]:
        print("  peer:", p.get("peer"))'

ceph osd blocked-by
ceph pg 4.1f3 query | grep -A5 might_have_unfound

might_have_unfound names the OSDs that could still hold the objects peering is missing. If they are the two dead nodes, the objects exist only there and nowhere else.

The capacity arithmetic

ceph df
ceph osd df | tail -3

Losing 2 of 8 hosts leaves 6 to hold what 8 held. A cluster at 65%
becomes 65 x 8/6 = 87% once re-replication completes — past nearfull,
with backfill needing headroom it no longer has.

ceph osd df --format json | python3 -c '
import sys,json
n = json.load(sys.stdin)["nodes"]
up = [o for o in n if o.get("status") == "up" and o.get("kb")]
used = sum(o["kb_used"] for o in up); size = sum(o["kb"] for o in up)
print("surviving OSDs: %d  used %.1f%% of surviving capacity"
      % (len(up), 100.0*used/size))'

Wait, force, or restore

# only after the OSD is confirmed unrecoverable and its objects accepted lost
ceph osd lost 31 --yes-i-really-mean-it
ceph pg 4.1f3 mark_unfound_lost delete
ceph osd force-create-pg 4.1f3 --yes-i-really-mean-it

Quiz

Production discipline

Split the PG list into degraded and inactive before doing anything else; they have different causes, different client effects, and only one of them is urgent. Compute post-recovery utilisation before letting re-replication run — losing a quarter of the hosts raises utilisation by a third, and a recovery that hits nearfull stops halfway with the cluster still exposed.

Cross-course references

• Kubernetes: a quorum-based controller refuses to act rather than act on a stale view
• Linux: a journal that cannot prove it is current is replayed or refused, never assumed