Recovering a cluster after a full event

Why this matters in production

Getting writes flowing again is the first half. The second half — restoring the margins, clearing the degradation, and fixing what allowed it — is what determines whether it happens again next quarter.

The sequence

1. reduce consumption below the full ratio    → writes resume
2. reduce below backfillfull                  → recovery resumes
3. let recovery complete                      → degradation clears
4. rebalance                                  → capacity recovered from imbalance
5. restore any raised ratios                  → margins restored
6. restore any lowered pool sizes             → durability restored
7. expand                                     → durable headroom
8. fix the detection gap                      → prevention

Verifying each stage

# 1 — writes resumed
ceph health detail | grep -E 'OSD_FULL|POOL_FULL' || echo 'clear'
ceph osd df | sort -k17 -rn | head -3

# 2 — recovery resumed
ceph -s | grep -E 'degraded|recovery'
ceph pg dump pgs | grep -c toofull

# 3 — degradation cleared
ceph -s | grep 'active+clean'
ceph health detail | grep PG_DEGRADED || echo 'clear'

# 4 — rebalance
ceph balancer status
ceph osd df | awk 'NR>1 {print $17}' | sort -n | sed -n '1p;$p'

Do not proceed past step 3 while any PG remains degraded; the cluster’s redundancy matters more than its balance.

Restoring temporary changes

# ratios
ceph osd set-full-ratio 0.95
ceph osd set-backfillfull-ratio 0.90
ceph osd set-nearfull-ratio 0.85
ceph osd dump | grep ratio

# pool sizes — this is a backfill, so plan it
ceph osd pool set backups size 3
watch -n 30 'ceph -s | grep misplaced'

Restoring a pool’s size moves a full copy of it and should be scheduled rather than run immediately after an incident, when the cluster has other work.

# a checklist of everything changed during the incident
ceph config dump | grep -E 'full_ratio|backfill'
ceph osd pool ls detail | grep -E 'size|min_size'

Addressing the causes

ceph balancer on
ceph balancer mode upmap

The post-incident record

Full event, prod-ceph-01, 2026-08-18
Trigger:   growth outpaced the 85% nearfull alert; procurement lead 6 weeks
Detection: application ENOSPC reports, 40 minutes before the nearfull page
Duration:  writes blocked 2h14m; degraded a further 9h
Actions:   lowered backups size to 2; removed 340 RBD snapshots; raised
           backfillfull to 0.92 temporarily
Restored:  ratios 2026-08-18; backups size restored 2026-08-21
Fixes:     projection alert at 60 days; balancer enabled; snapshot retention
           automation; reclamation list written

Quiz

Production discipline

Let recovery reach active+clean before rebalancing — durability first, efficiency second. Track every temporary change made during the incident with an owner and a restoration date, and schedule the pool size restoration for a quiet window since it is a pool-sized backfill that consumes the space just recovered.

Cross-course references

• Kubernetes: restoring resource limits relaxed during an incident needs the same tracking
• Linux: temporary tuning applied under pressure becomes permanent without a restoration list