CephLXV · Why Full Clusters Are DangerousWhy Full Clusters Are Dangerous
How a full cluster narrows your options
What you'll learn
- List the operations lost at each threshold
- Plan while the options still exist
- Recognise the point of no easy return
- Preserve optionality deliberately
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
Capacity problems are not a gradient of severity — they are a sequence of capabilities being removed. Knowing which goes when is what makes acting early obviously worthwhile.
What is lost at each stage
| Stage | Still available | Lost |
|---|---|---|
| Below 0.80 | everything | — |
| Nearfull (0.85) | everything | comfort |
| Backfillfull (0.90) | client I/O, deletes, reads | recovery, rebalancing, draining OSDs |
| Full (0.95) | reads, most deletes | client writes |
| Beyond | reads | some deletes |
The step from nearfull to backfillfull is the significant one: it removes the cluster’s ability to heal and to move data, which are the two things every remedy depends on.
Operations lost at backfillfull
# these all stop working usefully
ceph osd out 12 # drain: nowhere for the data to go
ceph balancer on # rebalance: no destination has room
ceph orch osd rm 12 # removal: requires draining
ceph osd crush reweight ... # any placement change: same problemceph health detail | grep -i backfill
ceph pg dump pgs | grep -c toofullA cluster past backfillfull cannot be reconfigured. Any change that
moves data is queued and stalls, which means even a planned maintenance
involving an OSD drain becomes impossible.
Recognising the point of no easy return
easy: below 0.85 — every remedy works, on any timeline
harder: 0.85–0.90 — remedies work, timeline shortens
hard: 0.90–0.95 — only deletion and ratio changes work
emergency: above 0.95 — writes blocked; deletion may need a ratio bumpceph osd df | sort -k17 -rn | head -1The single number that places a cluster on this scale is the fullest OSD’s utilisation, not the average.
Preserving optionality
| Practice | Preserves |
|---|---|
| Projection alerting with a long horizon | procurement as an option |
| Balancer enabled continuously | the imbalance remedy staying small |
| Lifecycle rules on RGW buckets | deletion happening automatically |
| A documented list of deletable data | fast reclamation under pressure |
| Host failure absorption checked monthly | recovery remaining possible |
Each of these costs almost nothing and each keeps one remedy available at the moment it is needed.
Quiz
Knowledge check · 4 questions
Q1. Why does draining an OSD require free space even though it ends with more space available?
Q2. A cluster past the backfillfull ratio can still have OSDs drained for maintenance.
Q3. Plan maintenance on a cluster near backfillfull.
A planned maintenance requires draining two OSDs to replace their host's HBA. The fullest OSD is at 89% and the cluster average is 84%.
Q4. Which single number places a cluster on the capacity-severity scale?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Treat 0.90 on the fullest OSD as an operational ceiling rather than a
warning; past it the cluster cannot be drained, rebalanced, or
reconfigured. For a hardware swap that does not need evacuation, noout
with the OSDs stopped avoids the data movement a drain would require.
Cross-course references
- Kubernetes: a cluster with no schedulable capacity cannot drain a node either
- Linux: needing free space to shrink a filesystem is the same paradox