The full-cluster trap and how it forms

Why this matters in production

The trap is a real circular dependency, not a metaphor. Seeing exactly where the circle is breakable is what makes the escape systematic rather than improvised.

The circle

flowchart LR
  A[OSD at full ratio] --> B[Client writes blocked]
  A --> C[Backfill onto it blocked]
  C --> D[Cannot rebalance away from it]
  C --> E[Cannot drain it]
  D --> A
  E --> A
  F[Add OSDs] --> G[Backfill needed]
  G --> C

Stated plainly:

the OSD is full
  → data cannot be moved onto other OSDs that are also constrained
  → the fullest OSD cannot be relieved by rebalancing
  → it cannot be drained, because draining is backfill
  → adding hardware requires backfill, which is constrained
  → so the cluster stays full

Where the circle breaks

There are exactly three places:

The first two share the essential property: they reduce consumption without requiring a destination. That is what makes them the only true escapes.

# neither of these needs anywhere to put anything
rbd snap rm rbd-vms/vm-disk-1@old
ceph osd pool set backups size 2

Raising the ratios is different — it does not reduce consumption, it permits the constrained movement to proceed by moving the line. It works, and it spends the margin.

The sequence that produces the trap

1. capacity warnings acknowledged without action
2. growth continues
3. an OSD reaches backfillfull; rebalancing stops working
4. imbalance worsens because the balancer is now blocked
5. the fullest OSD reaches full; writes block
6. hardware is ordered; lead time is weeks

Step 4 is the accelerant: once rebalancing stops, the imbalance that caused the first OSD to fill grows unchecked, so subsequent OSDs reach the thresholds faster than the growth rate alone would predict.

ceph balancer status
ceph osd df | awk 'NR>1 {print $17}' | sort -n | sed -n '1p;$p'

Escaping

# 1. reduce consumption — the only unconditional escape
ceph osd pool set scratch size 2
rbd snap ls --all --pool rbd-vms

# 2. once below full, restore writes and confirm
ceph osd df | sort -k17 -rn | head -3

# 3. once below backfillfull, rebalancing works again
ceph balancer on

# 4. now expansion is effective
ceph orch apply osd --all-available-devices

The ordering is forced by the circle: nothing that moves data works until consumption has been reduced by something that does not.

Quiz

Production discipline

Recognise that every remedy which moves data is blocked in a trapped cluster — attempting them wastes the time that matters. Break the circle by reducing consumption in place: delete, or lower a non-critical pool’s size, then restore the movement-based remedies in threshold order.

Cross-course references

• Kubernetes: a cluster too full to evict anything faces the identical circular block
• Linux: needing free space to free space is the classic full-filesystem deadlock