Capacity planning: forecasting growth and avoiding surprises

Capacity surprises are the worst kind of outage. They don’t announce themselves — they happen when someone tries to create a VM and the pool is full, or runs a backup and the PBS datastore is full, or attempts to live-migrate and the target node has no memory.

This lesson covers the discipline of capacity planning: how to forecast, what to measure, and how to turn forecasts into hardware orders with enough lead time.

The four resources

A PVE cluster has four resource constraints to track:

Each constraint grows at a different rate and has a different lead time for procurement. Storage is usually the cheapest to add (and cheapest to run out of). CPU is usually the most expensive to add.

Measuring growth

A capacity forecast starts with measurement. The cheapest source of data is your existing Prometheus:

# Cluster CPU utilisation (avg over last 30 days)
avg(avg_over_time(100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)[30d:1h]))

# Cluster memory utilisation
avg(avg_over_time((1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100)[30d:1h])

# Cluster storage used
sum(pve_disk_usage_bytes) / sum(pve_disk_total_bytes)

# Cluster network utilisation (avg)
avg(rate(node_network_receive_bytes_total{device!~"lo|veth.*"}[5m]) * 8 +
    rate(node_network_transmit_bytes_total{device!~"lo|veth.*"}[5m]) * 8)

For the forecast, you need:

• Current value: live utilisation (today)
• Trend: slope over the last 90 days
• Headroom: how much buffer is between current and total capacity
• Lead time: how long it takes to procure more

The forecasting formula

For each resource:

months_to_exhaustion = (capacity - current_usage) / monthly_growth
days_to_procure = lead_time_in_days
months_to_order = days_to_procure / 30
warning_threshold = months_to_order + 1   # Order one month before you run out

Example for storage:

capacity = 100 TB
current_usage = 65 TB
monthly_growth = 4 TB/month
months_to_exhaustion = (100 - 65) / 4 = 8.75 months
lead_time_for_disks = 30 days
months_to_order = 1.0
warning_threshold = 2.0 months
# Current months_to_exhaustion = 8.75 > warning_threshold 2.0: NO action needed
# When it drops below 2.0, order more storage

A simple spreadsheet or Prometheus recording rule can compute this for each resource and alert when any resource is below its warning threshold.

The growth rate problem

Most capacity forecasts are wrong because they assume linear growth. Real growth is:

• Linear: organic growth as the team adds more VMs
• Step: a new application launches; +10 TB overnight
• Seasonal: end-of-quarter batch jobs; backup size spikes
• Death: someone deletes an old VM; capacity released

To handle all of these, track growth with multiple metrics:

# 7-day moving average (smooths daily noise)
avg_over_time((pve_disk_usage_bytes / pve_disk_total_bytes)[7d])

# 30-day moving average (catches seasonal)
avg_over_time((pve_disk_usage_bytes / pve_disk_total_bytes)[30d])

# 7-day rate of change
deriv((pve_disk_usage_bytes / pve_disk_total_bytes)[7d:1h])

The 7-day average is for capacity planning. The 30-day average is for long-term trend. The derivative catches sudden spikes.

Identifying the binding constraint

For each resource, compute:

• Current utilisation %
• Months to exhaustion at current growth rate
• Lead time to add capacity
• Cost to add capacity (CPU/RAM cheapest, storage next, network most expensive per unit)

The binding constraint is the resource that runs out first after adding capacity. Most clusters hit storage first because storage is the easiest to add but easiest to fill.

Once you’ve identified the binding constraint, address it first:

• CPU: add nodes, upgrade CPUs, consolidate workloads onto fewer larger nodes
• Memory: add nodes, increase RAM per node, right-size VMs
• Storage: add disks to existing pools, expand to new pools, tier storage (hot SSD + cold HDD)
• Network: upgrade NICs and switches, add bandwidth

Capacity planning template

For each cluster, maintain a spreadsheet:

Cluster: production-cluster
Last updated: 2024-01-15

Resource | Capacity | Used | Free | % | Monthly growth | Months to exhaust | Lead time | Order by
---------|----------|------|------|---|----------------|-------------------|------------|----------
CPU      | 96 cores | 65   | 31   | 68% | 4 cores         | 7.75              | 14 days    | 2024-08-15
Memory   | 768 GB   | 580  | 188  | 76% | 30 GB           | 6.3               | 14 days    | 2024-06-15
Storage  | 200 TB   | 145  | 55   | 73% | 8 TB            | 6.9               | 30 days    | 2024-06-30
Network  | 40 Gbps  | 18   | 22   | 45% | 1 Gbps          | 22                | 30 days    | 2025-10-15

The “Order by” column is the trigger for procurement. Set alerts in Prometheus to fire when this date approaches.

PBS capacity

PBS is often the first resource to run out because backup retention accumulates over time. Two patterns:

• Aggressive prune: keep backups for 7 days, 4 weeks, 12 months. Total storage roughly equal to 2x weekly backup size.
• Tiered storage: PBS writes to fast SSD; nightly sync to cold HDD or S3 with Object Lock. Cold tier is the long-term retention.

For either pattern, monitor PBS datastore free space as a critical alert. A full PBS rejects new backups, which means the next disaster loses data.

Storage tiering for cost

For most clusters, not all data is equal:

• VM disks: hot, fast SSD, latency-critical
• ISO templates and snippets: warm, fast SSD
• Backups (recent): warm, HDD, throughput not latency
• Backups (archive): cold, slow HDD or S3, throughput matters

A tiered design:

Tier 1: NVMe SSD pool (mirrored)
  - VM operating system disks
  - Latency-critical workloads (databases)

Tier 2: SATA SSD pool (mirrored)
  - VM data disks
  - Bulk storage

Tier 3: HDD pool (RAIDZ2)
  - ISO templates
  - Cold backups

Tier 4: S3 with Object Lock (off-site)
  - Long-term archive
  - Disaster recovery

PVE supports this via separate ZFS pools with different storage classes.

Production considerations

• Lead time is the constraint. 14 days for SSDs, 30 days for HDDs, 60–90 days for new servers. Plan backwards from the lead time.
• Vendor consolidation. Use one vendor for SSDs, one for HDDs, one for servers. Smaller vendor list = better pricing, faster shipping, fewer driver issues.
• Spare parts. Keep at least one spare SSD and one spare HDD per cluster, on-site. A failed disk that takes 5 days to replace is acceptable for cold storage, unacceptable for production.
• Right-sizing vs headroom. A 70% utilised cluster has 30% headroom for growth spikes. Below 50% means over-provisioning. Above 85% means under-provisioning.

Common mistakes

• Linear forecasts in a step-growth world. A 2 TB/month trend breaks when the team deploys a new analytics platform.
• Ignoring PBS. Backup capacity runs out faster than VM capacity because of retention policies.
• Lead time underestimation. “We can order SSDs and have them tomorrow” — until you can’t, because the vendor is back-ordered.
• No formal trigger. Without an alert on the forecast, capacity planning is a quarterly manual exercise that gets skipped.

Key takeaways

• Forecast: capacity, usage, growth, lead time, order trigger.
• The binding constraint dictates the next purchase.
• PBS capacity runs out first — monitor it as a critical alert.
• Tier storage by access pattern for cost efficiency.

Knowledge check