Keeping a forecast accurate

Intermediate⏱ ~17 mincephprometheus

What you'll learn

Track forecast error over time
Recalibrate on a schedule
Recognise when a forecast has broken
Communicate uncertainty honestly

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A forecast made in January and quoted in June is an artefact, not a prediction. The discipline that keeps it useful is recalibration plus a record of how wrong the previous ones were.

Tracking forecast error

date       forecast made   predicted exhaustion   actual/current   error
2026-01-15 90d out         2026-04-15             reached 2026-05-02   +17d
2026-04-01 120d out        2026-08-01             reached 2026-07-18   −14d
2026-07-01 100d out        2026-10-09             pending              —

Two forecasts with errors of opposite sign and similar magnitude suggest noise rather than bias. Consistent error in one direction suggests the model is missing something systematic — usually deletion behaviour or a seasonal pattern.

# record the inputs alongside the forecast
date +%F,$(ceph df --format json | python3 -c '
import sys,json; d=json.load(sys.stdin)
print(max(p["stats"]["max_avail"] for p in d["pools"]))') >> forecasts.csv

Recalibrating

Quarterly is a reasonable default, with triggers that force an earlier review:

Trigger	Why
A new workload onboarded	step change not in the trend
A retention policy change	the steady state moved
Hardware added	the denominator changed
Growth rate changed by more than 30%	the model no longer fits
The previous forecast was wrong by more than 20%	the model needs revisiting

# current rate against the rate the forecast assumed
# in Prometheus:
#   rate(ceph_cluster_total_used_raw_bytes[14d]) * 86400

When a forecast has broken

symptoms:
  the projected date moves by weeks between reviews
  the growth rate has changed step-wise rather than gradually
  a single event dominates the regression window
  the forecast disagrees with what workload owners say is coming

The last is the most useful check and the least often performed. If a trend says twelve months and a team says they are migrating 300 TB in October, the team is right and the trend is measuring the past.

Communicating uncertainty

Poor	Better
“We’ll be full in 117 days”	“Between 95 and 140 days; the range comes from growth varying between 1.0 and 1.4 TB/day”
“The forecast says December”	“December on the current trend, which excludes the tenant onboarding in October”
“It’s fine for now”	“Next review 1 October; nothing before then requires action”

Stating what the forecast excludes is what stops it being quoted as though it included everything.

Quiz

Knowledge check · 4 questions

Q1. A projected exhaustion date swings by several weeks between reviews. What does this most likely indicate?

Growth really is accelerating and decelerating on a weekly cycle The metric is scraped from more than one mgr, so the series contains gaps A rebalance is in progress, and MAX AVAIL moves while PGs are in flight The regression window is too short and edge events dominate the slope
Q2. Consistent forecast error in one direction indicates random noise.

True False
Q3. Recalibrate a drifting forecast.

A quarterly forecast has been wrong by more than 20% twice, both times predicting exhaustion later than it occurred. Growth appears steady in the graphs.
Q4. What is the cheapest and most often skipped forecast calibration?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Record each forecast with its inputs and score it against what actually happened; consistent error in one direction points at a missing term, most often balance drift or deletion behaviour. Ask workload owners what is planned before publishing a forecast — the trend cannot see decisions that have not yet reached the cluster.

Cross-course references

Kubernetes: capacity forecasts miss planned workload migrations for the same reason
Linux: any growth projection needs the same recalibration and error tracking