The nearfull alert and the capacity conversation

Why this matters in production

Nearfull is the alert most likely to be acknowledged repeatedly without action, because nothing breaks when it fires. That is precisely why it needs a defined response.

The first check

ceph osd df | sort -k17 -rn | head -3
ceph osd df | awk 'NR>1 {n++; s+=$17} END {printf "avg %.1f\n", s/n}'

fullest: 86.2
average: 63.4
spread:  22.8   → imbalance

fullest: 86.2
average: 82.7
spread:   3.5   → genuine shortage

One command distinguishes the two situations, which have completely different responses.

The imbalance path

ceph balancer status
ceph balancer mode upmap
ceph balancer optimize plan
ceph balancer eval plan
ceph balancer eval          # current score, for comparison
ceph balancer execute plan

# verify the spread narrows
watch -n 60 'ceph osd df | sort -k17 -rn | head -1'

If the balancer is already enabled and the spread is still wide, investigate why — often require-min-compat-client blocks upmap, or a CRUSH rule prevents movement.

The shortage path

# what can be reclaimed?
rbd snap ls --all --pool rbd-vms | wc -l
ceph df detail

# what is the growth rate?
# from Prometheus, or from recorded ceph df history

# how long remains?
ceph df | grep MAX

The output of this path is a date and a decision, not an acknowledgement:

Fullest OSD 86%, average 83% — genuine shortage
Growth 1.4 TB/day, MAX AVAIL 61 TB → 43 days to exhaustion
Procurement lead time 8 weeks → order must be placed this week
Decision required: approve expansion, or reduce retention

Avoiding routine acknowledgement

- alert: CephNearfullUnresolved
  expr: ceph_health_detail{name="OSD_NEARFULL"} == 1
  for: 14d
  labels: { severity: page }
  annotations:
    summary: "Nearfull unresolved for 14 days — capacity decision overdue"

An escalating duplicate of the same alert is what prevents it being acknowledged indefinitely.

Quiz

Production discipline

Compute the fullest-versus-average spread in the alert expression and include it in the annotation — it performs the imbalance-versus-shortage triage before anyone reads the alert. Add an escalating duplicate that pages if nearfull persists, since nothing observable changes while it is ignored.

Cross-course references

• Kubernetes: quota-approaching alerts have the same indefinite-acknowledgement problem
• Linux: disk-space warnings are the archetypal ignorable alert