Classifying a failure by what it costs

Advanced⏱ ~18 minceph

What you'll learn

Distinguish degraded from inactive from lost
Determine which state a cluster is in
Match the response to the state
Communicate the state accurately

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Treating an inactive cluster as a degraded one wastes the window in which data is still recoverable, and treating degraded as lost causes unnecessary restores.

The three states

State	Meaning	Data
Degraded	fewer copies than desired, still serving	intact
Inactive	some PGs cannot serve I/O	intact but unavailable
Lost	some PGs have no surviving copy	gone unless restored

ceph -s
ceph pg stat
ceph health detail

  data:
    pgs:     3812 active+clean
             184  active+undersized+degraded      ← degraded
             12   down                            ← inactive
             2    incomplete                      ← possibly lost

Determining the state

ceph pg dump_stuck inactive
ceph pg dump_stuck stale
ceph pg dump_stuck undersized

# which PGs cannot serve, and why
ceph health detail | grep -E 'PG_AVAILABILITY|PG_DEGRADED|OSD_DOWN'

# for a specific PG, what it needs
# PG id from the dump_stuck output above:
PGID=3.1f

ceph pg "$PGID" query | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("state:", d.get("state"))
rp = d.get("recovery_state", [])
for s in rp[:2]:
    print("-", s.get("name"))
    b = s.get("blocked", "")
    if b: print("  blocked:", b[:200])'

Query output	Means
`peering_blocked_by` naming down OSDs	those OSDs must return
`might_have_unfound`	data may exist on an OSD that is down
`unfound` objects listed	copies are genuinely missing
`incomplete`	insufficient history to peer

# PG id from the dump_stuck output above:
PGID=3.1f

ceph pg "$PGID" list_unfound | head -20

Matching the response

State	Response
Degraded	let recovery run; do not intervene beyond capacity checks
Inactive, OSDs recoverable	bring those OSDs back; peering resumes
Inactive, OSDs unrecoverable	assess `might_have_unfound` before any destructive step
Unfound objects	attempt OSD recovery first; `mark_unfound_lost` is last
Lost	restore from backup

# never the first action
# ceph pg <pgid> mark_unfound_lost revert|delete

`mark_unfound_lost` tells the cluster to stop waiting for data that may
still exist on a down OSD. Running it while any OSD that might hold a
copy is recoverable discards data that was not lost.

# which down OSDs might hold the missing copies
# PG id from the dump_stuck output above:
PGID=3.1f

ceph pg "$PGID" query | python3 -c '
import sys,json
d = json.load(sys.stdin)
for s in d.get("recovery_state", []):
    m = s.get("might_have_unfound")
    if m:
        for e in m: print("osd:", e.get("osd"), "status:", e.get("status"))'

Communicating accurately

Say	Not
“Reduced redundancy; data intact and serving”	“we lost disks”
“Some volumes unavailable; data intact, recovering access”	“data loss”
“Assessing whether N objects are recoverable”	“we may have lost data” — until assessed
“N objects confirmed unrecoverable; restoring from backup”	vague reassurance

The distinction between unavailable and lost is the one stakeholders
most need and least often receive.

Quiz

Knowledge check · 4 questions

Q1. What distinguishes an inactive PG from one with lost data?

Inactive means the surviving OSDs cannot establish who holds the authoritative data — uncertainty; unfound means their logs prove writes were acknowledged that none of them holds — evidence The count of OSDs currently down: below the pool's min_size the PG is inactive, and at zero surviving replicas its objects are unfound The pool type: a replicated pool goes inactive once it drops below min_size, while an erasure-coded pool short of its required k shards reports its objects as unfound instead of blocking on them The health check name: PG_AVAILABILITY is raised for an inactive PG and OBJECT_UNFOUND only once the objects have been confirmed gone
Q2. Running `mark_unfound_lost` while an OSD listed in `might_have_unfound` is still recoverable destroys data that was never lost.

True False
Q3. Classify a cluster failure.

A rack has lost power. `ceph -s` shows 200 PGs degraded, 15 down, and 3 with unfound objects. The rack's hosts may be recoverable.
Q4. What should stakeholders be told during a degraded-but-serving incident?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read ceph pg query and its might_have_unfound list before any destructive step — mark_unfound_lost discards objects that a recoverable OSD may still hold. Communicate “unavailable” and “lost” as the different things they are.

Cross-course references

Kubernetes: a pod that is Pending is not a pod that failed
Linux: uncertainty and evidence require different responses