Monitoring capacity

Why this matters in production

Capacity is the one problem with a reliable warning period, and the warning is only useful if the alert fires while the remedy — usually procurement — still fits in the time remaining.

The commands

ceph df                    # pool-level, with MAX AVAIL
ceph df detail             # adds compression and quota columns
ceph osd df                # per-OSD, with the utilisation column
ceph osd df tree           # per-OSD grouped by host
ceph osd pool autoscale-status

# the three numbers that matter
ceph df | awk '/POOLS/,0' | awk 'NR>2 {print $1, $(NF)}'   # MAX AVAIL per pool
ceph osd df | sort -k17 -rn | head -1                       # fullest OSD
ceph osd df | awk 'NR>1 {n++; s+=$17} END {print s/n}'      # average

The metrics

ceph mgr module enable prometheus
curl -s localhost:9283/metrics | grep -E '^ceph_(cluster|pool|osd).*bytes'

The per-OSD pair is what gives the fullest-OSD figure:

max(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes)

Alerts with useful lead time

An alert at 85% on a cluster growing 1% per week gives ten weeks. The same alert on a cluster growing 5% per week gives two — less than most procurement cycles.

The alert that scales is a projection:

predict_linear(ceph_pool_max_avail[14d], 86400 * 60) < 0

“This pool will be full within 60 days at the current rate.” That fires with the same lead time regardless of growth rate.

Alerts that fire too late

Quiz

Production discipline

Alert on predict_linear over ceph_pool_max_avail with a horizon longer than the procurement lead time; a fixed threshold gives less warning precisely when the cluster is filling fastest. Track the fullest OSD and the utilisation spread alongside it — the spread is capacity a rebalance can recover without hardware.

Cross-course references

• Kubernetes: PVC capacity projection alerts follow the same pattern
• Linux: disk-fill prediction beats fixed percentage thresholds for the same reason