Purpose
A Ceph cluster does not degrade gracefully as it fills. It crosses
nearfull_ratio, warns, keeps taking writes, and then crosses full_ratio
and stops taking them altogether - and the free space you need to re-replicate
a dead host is the same free space the tenants just consumed. This quarterly
review exists so that the decision to buy disks is made a quarter early rather
than during an outage.
Order of work
The list runs in four passes. The measurement pass is df and osd-df:
take the pool-level and the per-OSD numbers together, because a cluster that
looks half empty in aggregate can still have one OSD at eighty percent, and it
is the fullest OSD that sets the ceiling. The trend pass is growth-rate and
forecast, which turn this quarter’s snapshot into a date. The headroom pass
is headroom-recovery and headroom-maintenance, the two reservations that
must survive the forecast. The standing control is alert, which is what
covers the eleven weeks between one review and the next.
Before you start
Bring the previous review’s numbers, the current nearfull_ratio and
full_ratio settings, the size of the largest failure domain, and the
procurement lead time for new hardware. The forecast is only actionable if it
is compared against that lead time.
Verification
The review is complete when every item is checked and the forecast carries a date and a signature. Mark an item N/A rather than leaving it unchecked when it does not apply to this cluster.
Escalation
If the forecast puts the cluster past nearfull inside the procurement lead time, raise it with capacity planning immediately rather than at the next review.