Why a full cluster cannot heal itself

Why this matters in production

Ceph’s central operational property is that it repairs itself. A full cluster does not, and understanding exactly why prevents waiting for a recovery that will never start.

The dependency each remedy has

flowchart TD
  A[Cluster full] --> B[Recovery needs a destination with space]
  A --> C[Rebalance needs a destination with space]
  A --> D[Expansion needs backfill, which needs space at the source's peers]
  A --> E[Deleting frees space directly]
  B --> F[blocked]
  C --> F
  D --> G[slow — depends on backfill]
  E --> H[works]

Adding OSDs is the case worth being precise about: the new OSDs are empty, so backfill onto them is permitted. What is constrained is that the existing OSDs must source that backfill while also being at their limit, and any PG whose other destinations are full stays stuck.

The compounding risk

cluster full
  → recovery blocked
  → a failure during this period cannot be repaired
  → the cluster is degraded and unable to heal
  → a second failure has a worse outcome than usual

ceph -s | grep -E 'degraded|undersized'
ceph health detail | grep -E 'PG_DEGRADED|OSD_FULL'

A full cluster that is also degraded is the most dangerous ordinary state Ceph reaches: it is not losing data, and it has no mechanism to stop losing redundancy either.

Prioritising under the constraint

1. stop the bleeding — halt whatever is still writing
2. delete — the only unconstrained remedy
3. restore writes — get below the full ratio
4. restore recovery — get below backfillfull
5. rebalance — recover capacity from imbalance
6. expand — the durable fix

# 1
# quiesce the heaviest writers at the application layer

# 2
rbd snap ls --all --pool rbd-vms
radosgw-admin bucket list | head
ceph df detail

# 3–4
ceph osd df | sort -k17 -rn | head -3

The ordering is not arbitrary: each step restores a capability the next one needs.

Quiz

Production discipline

Recognise that a full cluster has lost its self-healing property entirely — waiting for recovery to start is waiting for something that cannot happen. Work the sequence in order: stop writes, delete, restore writes, restore recovery, rebalance, expand; each step restores a capability the next one needs.

Cross-course references

• Kubernetes: a cluster with no schedulable capacity cannot reschedule failed pods either
• Linux: a full filesystem that cannot write its own journal presents the same deadlock