Alerting on capacity thresholds

Intermediate⏱ ~17 mincephprometheus

What you'll learn

Define alerts for each capacity threshold
Assign severities that match the response
Write alerts that carry their own remedy
Avoid alerts that arrive too late to act on

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A capacity alert is only useful if it arrives while the remedy it implies still fits in the time available. Threshold and severity both follow from that.

The alerts

Condition	Severity	Response
Pool projected full within 60 days	ticket	start procurement
OSD utilisation spread > 20 points	ticket	run the balancer
Fullest OSD > 0.80	ticket	investigate; balance or plan
`OSD_NEARFULL`	page during hours	balance, delete, or expand
`PG_BACKFILL_FULL`	page immediately	cluster degraded and stuck
`OSD_FULL`	page immediately	writes blocked

The severities are not proportional to the numbers — they are proportional to how much time the responder has.

The expressions

- alert: CephPoolProjectedFull
  expr: predict_linear(ceph_pool_max_avail[14d], 86400 * 60) < 0
  for: 6h
  labels: { severity: ticket }
  annotations:
    summary: "Pool {{ $labels.name }} projected full within 60 days"
    runbook: "Rebalance, review deletable data, or start procurement"

- alert: CephOSDImbalance
  expr: |
    (max(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes)
     - avg(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes)) > 0.20
  for: 24h
  labels: { severity: ticket }
  annotations:
    summary: "OSD utilisation spread exceeds 20 points"
    runbook: "ceph balancer status; balancer optimize/eval/execute"

- alert: CephBackfillFull
  expr: ceph_health_detail{name="PG_BACKFILL_FULL"} == 1
  for: 5m
  labels: { severity: critical }
  annotations:
    summary: "Recovery is stalled for lack of space; cluster is degraded"
    runbook: "Balance, delete, raise backfillfull ratio temporarily, add capacity"

Alerts that carry their own remedy

The difference between a useful alert and a noisy one is what the responder does next:

Poor	Better
“Ceph capacity high”	“Pool rbd-vms projected full in 47 days; procurement lead time is 6 weeks”
“OSD 44 nearfull”	“osd.44 at 86%, cluster average 63%; spread suggests balancer will resolve”
“HEALTH_WARN”	the specific check, with the command that diagnoses it

Including the average alongside the outlier is what lets the responder distinguish imbalance from a genuine shortage without any investigation.

Verifying the alerting works

# does the metric exist?
curl -s localhost:9283/metrics | grep ceph_pool_max_avail

# does the projection produce a sensible number?
# in Prometheus:
#   predict_linear(ceph_pool_max_avail[14d], 86400 * 60)

An alert on a metric that is not being scraped is worse than no alert, because it creates the impression of coverage.

Quiz

Knowledge check · 4 questions

Q1. Why include the cluster average in a per-OSD capacity alert?

Alertmanager requires a cluster-scope label on every per-OSD alert before it will group them It lets the responder distinguish imbalance from a genuine capacity shortage immediately It suppresses duplicates, since only the OSD furthest from the average is allowed to fire The average is the figure MAX AVAIL is computed from, so it is what the responder must act on
Q2. `PG_BACKFILL_FULL` should be a ticket rather than a page.

True False
Q3. Review a capacity alerting setup.

A cluster alerts only on HEALTH_WARN and HEALTH_ERR. Both fire regularly for unrelated reasons and are routinely acknowledged without investigation.
Q4. Why use a 14-day window for predict_linear rather than 24 hours?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Alert on specific capacity conditions rather than on generic HEALTH_WARN; a check that fires for many reasons trains the team to acknowledge without reading. Put the diagnostic command and the cluster average in every annotation so the responder can act rather than investigate.

Cross-course references

Kubernetes: alerts carrying their runbook link are the same practice
Linux: monitoring that distinguishes cause from symptom saves the same investigation time