The backfillfull threshold

Why this matters in production

backfillfull is the threshold that turns a capacity problem into a durability problem, because it stops recovery. Recognising it quickly is what determines how long the cluster stays degraded.

What it blocks

ceph osd dump | grep backfillfull_ratio
ceph osd set-backfillfull-ratio 0.90    # the default

At the ratio, an OSD stops accepting backfill — data being moved onto it. It continues to accept client writes, serve reads, and source backfill to other OSDs.

Recognising a stall

ceph health detail | grep -A5 BACKFILL_FULL
ceph pg dump pgs | grep -c toofull
ceph -s

[WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself)
    pg 3.a2 is active+undersized+degraded+remapped+backfill_toofull

The PG state carries backfill_toofull and the cluster stays degraded indefinitely, because the mechanism that would restore redundancy has nowhere to write.

# how many PGs are affected and where they wanted to go
ceph pg dump pgs | awk '/toofull/ {print $1, $15, $17}'

Why the block is protective

Allowing backfill to continue past 0.90 would fill the OSD toward 0.95, at which point client writes stop. The cluster would trade a degraded state — data intact, redundancy reduced — for an unavailable one.

without the block:  degraded → recovery fills OSD → full → writes blocked
with the block:     degraded → recovery stops → writes continue → operator acts

The block preserves availability at the cost of prolonging degradation, which is the correct trade when an operator can intervene.

Restoring progress

# 1. balance — moves data off the constrained OSDs
ceph balancer status && ceph balancer on
ceph balancer optimize plan && ceph balancer eval plan

# 2. delete
rbd snap ls --all --pool rbd-vms
ceph osd pool ls detail | grep -i quota

# 3. raise the ratio, temporarily and deliberately
ceph osd set-backfillfull-ratio 0.92
watch -n 30 'ceph osd df | sort -k17 -rn | head -3'
# restore when recovery completes
ceph osd set-backfillfull-ratio 0.90

# 4. add capacity
ceph orch apply osd --all-available-devices

Raising the ratio while watching the fullest OSD is the safe form: the margin to full is what is being consumed, so it must be watched rather than assumed.

Quiz

Production discipline

Treat PG_BACKFILL_FULL as immediate: the blocked work is the recovery that would restore redundancy, so the cluster stays degraded until someone acts. Watch the fullest OSD continuously whenever the backfillfull ratio is raised — the margin being consumed is the one protecting client writes.

Cross-course references

• Kubernetes: unschedulable pods due to node pressure stall the same self-healing
• Linux: an array that cannot rebuild for lack of a spare stays degraded indefinitely