PGs stuck in recovery

Why this matters in production

Every hour a PG stays degraded is an hour of reduced redundancy. Recovery that is merely slow is a tuning conversation; recovery that has stopped is an availability risk with a deadline attached, because the next failure lands on a set that has not healed from the last one.

Slow versus stalled

ceph -s | grep -E 'objects (degraded|misplaced)'
sleep 300
ceph -s | grep -E 'objects (degraded|misplaced)'

A falling number is slow. An identical number is stalled. Make the measurement before forming an opinion — human perception of “not moving” over thirty seconds is unreliable.

The common causes of a real stall

A full or nearly-full OSD. The most frequent cause by a wide margin.

7.3d  active+undersized+degraded+remapped+backfill_toofull

Backfill will not push data onto an OSD above osd_backfill_full_ratio (0.85 by default). If every candidate destination is above it, the PG waits indefinitely.

ceph osd df | sort -k17 -rn | head
ceph health detail | grep -i full

A down or unresponsive OSD in the acting set. Recovery needs a source for the missing objects. If the only OSD holding them is down, the PG sits at recovery_unfound or simply waits.

ceph pg 7.3d query | jq -r '.recovery_state[] | select(.blocked_by) | .blocked_by'

Unfound objects. The cluster knows objects should exist and cannot find a copy.

ceph pg 7.3d list_unfound

This is the serious one: it means the OSDs that held the only current copies are gone. The resolutions are to bring those OSDs back, or to accept the loss.

Throttles set to zero. Somebody set osd_max_backfills 0 during a previous incident and never reverted it.

ceph config get osd osd_max_backfills
ceph config dump | grep -E 'recovery|backfill'

Cluster flags. norecover, nobackfill, or norebalance left set.

ceph osd dump | grep flags

Working backfill_toofull

The fix is capacity, but there are graduated steps:

# 1. see how bad it is
ceph osd df

# 2. rebalance away from the fullest OSDs
ceph osd reweight-by-utilization 110

# 3. temporary headroom — buys hours, not a solution
ceph osd set-backfillfull-ratio 0.88

# 4. delete reclaimable data, or add OSDs

Raising the ratio is a stopgap that trades safety margin for progress. Doing it repeatedly without adding capacity walks the cluster toward osd_full_ratio, at which point writes stop cluster-wide.

Quiz

Production discipline

Alert on degraded object counts that have not fallen over a defined interval, not just on their absolute value — a stall is a different page from a large recovery. Keep a documented capacity floor and treat crossing it as an incident, because backfill_toofull is the state where a capacity-planning failure becomes a durability failure.

Cross-course references

• Kubernetes: Pods Pending for lack of schedulable capacity is the same class of stall
• Linux: a filesystem at 100% halts writers the same way a full OSD halts backfill