Recovery under capacity constraint

Why this matters in production

Recovery is the mechanism that restores redundancy, and it consumes space to do so. A cluster without that space does not recover slowly — it does not recover at all.

What recovery needs

When an OSD is lost, the PGs it held are remapped and their data written to the new locations. The requirement is concrete:

space needed ≈ data on the lost OSD
distributed across the remaining OSDs in the affected PGs' CRUSH placement
without any of them passing backfillfull

# how much the lost OSD held
ceph osd df | grep '^ *44 '

# where it will go
ceph pg ls-by-osd 44 | wc -l
ceph osd df | sort -k17 -rn | head -5

Behaviour under constraint

ceph -s | grep -E 'degraded|misplaced'
ceph pg dump pgs | grep -c toofull

Partial recovery is the common and confusing case: the cluster makes progress, the degraded count falls, and then it stops with a residual count that does not move. That plateau is the signature.

watch -n 60 'ceph -s | grep degraded'
# if the number is static for 20 minutes, check for toofull

Estimating the requirement

# for a single OSD failure
ceph osd df | awk '$1 ~ /^[0-9]+$/ {print $1, $5}' | sort -k2 -rn | head -1

# for a host failure
ceph osd df tree | awk '/host/ {h=$NF} /osd\./ {s[h]+=$5} END {for (k in s) print k, s[k]}'

Compare that against the aggregate space available below backfillfull:

ceph osd df | awk 'NR>1 && $17 < 90 {avail += $4 * (0.90 - $17/100)} END {print avail " usable for recovery"}'

Creating the space

The remedies from the capacity lessons apply, with one addition specific to a constrained recovery:

# reduce the requirement itself
ceph osd pool set backups size 2      # temporarily, on a pool that tolerates it

Lowering a non-critical pool’s size releases a full copy of it immediately, which is often the fastest large reclamation available — and it is reversible once the emergency passes, at the cost of backfilling the copy back.

# after the recovery
ceph osd pool set backups size 3

Quiz

Production discipline

Watch the degraded count over twenty minutes before concluding a recovery is merely slow — a static count means it has stopped, and grep toofull confirms why. Keep a temporary size reduction on a non-critical pool in mind as the fastest large reclamation available, and plan the backfill needed to restore it.

Cross-course references

• Kubernetes: pods stuck Pending rather than slowly starting is the same distinction
• Linux: a RAID rebuild that halts rather than crawls needs a different diagnosis