Retention and churn

Why this matters in production

A cluster ingesting 5 TB/day and deleting 4 TB/day grows at 1 TB/day. A forecast built on the ingest rate is wrong by a factor of five, and one built on net growth breaks the moment retention changes.

Gross and net

net growth = ingest rate − deletion rate

# net, from the cluster
rate(ceph_cluster_total_used_raw_bytes[14d]) * 86400

# gross ingest, from the application side
# RGW request logs, application metrics, or:
rate(ceph_rgw_put[14d])

Only the net figure predicts capacity. The gross figure predicts throughput requirements, which is a different planning question.

Modelling retention

A cluster with a defined retention period reaches a steady state:

steady state size = ingest rate × retention period

5 TB/day × 90 days = 450 TB

Once at steady state the cluster stops growing, which is the property that makes retention-governed workloads far easier to size. The date it reaches steady state matters as much as the size:

days to steady state = retention period (from the start of ingest)

Where retention is absent

# is any lifecycle configured?
for b in $(radosgw-admin bucket list --format json | python3 -c 'import sys,json;[print(x) for x in json.load(sys.stdin)]'); do
  echo -n "$b: "
  radosgw-admin lc get --bucket="$b" 2>/dev/null | head -1 || echo 'none'
done

# snapshots by age
for i in $(rbd ls rbd-vms); do rbd snap ls rbd-vms/$i; done | wc -l

Data without retention grows without bound, so any forecast for it is a forecast of when the cluster fills rather than of a steady state.

Forecasting with retention defined

pool: rgw-buckets
  ingest:     5 TB/day
  retention:  90 days (lifecycle rule, verified active)
  steady:     450 TB logical, ×1.5 EC = 675 TB raw
  reached:    90 days after the rule took effect

pool: rbd-vms
  ingest:     0.8 TB/day net
  retention:  none — VM disks persist
  steady:     none; grows indefinitely
  forecast:   linear, exhaustion at current MAX AVAIL / 0.8 TB

Stating “retention: none” explicitly is what turns an unbounded pool from an oversight into a decision.

Quiz

Production discipline

Record every pool’s retention policy explicitly, including “none, by design” — the unbounded pools are what determine the expansion date and they are usually unbounded by omission. Forecast on net growth over a fourteen-day window, since asynchronous reclamation distorts shorter samples.

Cross-course references

• Kubernetes: log and metric retention drives the same steady-state calculation
• Linux: logrotate policies are the difference between bounded and unbounded growth