Growth, headroom, and the quarterly capacity review

The four preceding lessons produced numbers for a single host at a single moment. This one turns those numbers into two things a manager can act on: a date by which the current capacity is exhausted, and a reserve that keeps the fleet up when a host dies.

“We are at 60% and growing” is not a capacity plan. “We cross the 80% line on 14 March, and the procurement lead time is nine weeks, so we order in the second week of January” is.

From a slope to a date

The projection is one division:

days_to_threshold = (capacity x threshold - current) / daily_growth

Apply it to a disk carrying 57 GB on a 126 GB filesystem, growing at 420 MB a day, with the alert at 80%:

capacity x threshold = 126 x 0.80        = 100.8 GB
headroom remaining   = 100.8 - 57        =  43.8 GB
daily growth                             =   0.42 GB/day
days_to_threshold    = 43.8 / 0.42       = 104 days

104 days is mid-November. If the procurement or migration lead time is six weeks, the decision date is late September, and that date belongs in the calendar rather than in someone’s memory.

STAMP=$(date +%F)
USED=$(df --output=used -B1 /var/lib/docker | tail -1)
IMG=$(docker system df --format '{{.Size}}' 2>/dev/null | head -1)
printf '%s\t%s\t%s\n' "$STAMP" "$USED" "$IMG" >> /var/log/docker-capacity.tsv
tail -3 /var/log/docker-capacity.tsv

2026-08-09	60129542144	17.98GB
2026-08-10	60578234368	18.21GB
2026-08-11	61013622784	18.44GB

CAP=$(df --output=size -B1 /var/lib/docker | tail -1)
awk -v cap="$CAP" -F'\t' '
  { n++; x[n]=n; y[n]=$2; sx+=n; sy+=$2; sxy+=n*$2; sxx+=n*n }
  END {
    if (n < 7) { print "need at least 7 samples"; exit 1 }
    slope = (n*sxy - sx*sy) / (n*sxx - sx*sx)
    remain = cap*0.80 - y[n]
    printf "growth:  %.2f GB/day\n", slope/1073741824
    printf "to 80%%:  %.0f days\n", remain/slope
  }' /var/log/docker-capacity.tsv

growth:  0.42 GB/day
to 80%:  104 days

Operational headroom: the arithmetic of N+1

Per-host utilisation targets from the CPU lesson assume the host stays up. Fleet capacity has to assume one does not.

If N hosts serve the load and you must survive F of them failing, the survivors absorb everything:

utilisation_after_failure = U x N / (N - F)

Which inverts to the steady-state target you may actually run at:

U_steady  <=  U_max x (N - F) / N

Read the first row again, because it is the one that surprises people: a two-host pair that must survive one failure can only be run at 40%. Half of every host is reserved for the other host’s death. That is not waste; it is the price of the redundancy that was promised. The cheapest way to buy it back is more, smaller hosts — the ten-host fleet runs at 72% for the same guarantee.

FLEET=5
TOLERATE=1
SUM=$(docker ps -q | xargs -r docker inspect --format '{{.HostConfig.NanoCpus}}' | awk '{s+=$1} END {print s+0}')
CORES=$(nproc)
awk -v s="$SUM" -v c="$CORES" -v n="$FLEET" -v f="$TOLERATE" 'BEGIN {
  u = (s/1e9)/c
  printf "steady utilisation:   %.2f\n", u
  printf "after %d host loss:    %.2f\n", f, u*n/(n-f)
  printf "verdict: %s\n", (u*n/(n-f) > 0.80 ? "OVER BUDGET" : "within budget")
}'

steady utilisation:   0.67
after 1 host loss:    0.84
verdict: OVER BUDGET

The largest-container constraint

The fleet-average calculation hides a second constraint. A container that needs 24 GB of memory can only be rescheduled onto a host with 24 GB free. Average headroom of 30% spread across five hosts as 6% each does not run that container anywhere.

largest_unit  <=  free capacity on at least one surviving host

Check it explicitly. It is the constraint that turns a routine host reboot into an incident, and no average will tell you about it.

The quarterly capacity review

The review is a meeting with a fixed input and a required output. Its purpose is a decision, not a status update.

1. Inputs, gathered before the meeting. Peak (not mean) CPU, memory working set, disk used and inodes, conntrack peak, and the current growth slope for each — one table, one row per host or host class.
2. Confirm the sizing hypotheses. For every container: nr_throttled / nr_periods, memory.events oom_kill, and memory.peak against memory.max. Each is a pass or fail, not a discussion.
3. Reclaim first. Limits well above measured peaks are free capacity. Right-sizing usually recovers more than the next purchase would add, and it is available this week.
4. Project each resource to its threshold date using the slope, and note which is nearest. That resource is the one the plan is about; the others are noise this quarter.
5. Apply the N+1 constraint to the projected numbers, not the current ones. A fleet that is inside budget today and outside it at the projected date is already over budget.
6. Check the largest-unit constraint against the projected free space per host.
7. Produce a decision with a date. Order hardware, resize limits, shard the workload, or explicitly accept the risk with a review date. "Keep monitoring" is not one of the four.

Sanity check