The capacity plan — the document that turns numbers into a budget

The capacity plan is the artefact that closes the loop between the workload model and the bill. Without it, the pool size is whatever someone configured last quarter; the SLO is whatever someone promised in a meeting; the alert threshold is whatever someone guessed. With it, every capacity decision has a written rationale, a cost, and a review date.

What the plan contains

A capacity plan has six sections, in this order:

flowchart TB
  P["Capacity plan"] --> W["Workload model\narrivals, duration, retries"]
  P --> S["SLOs\nwait time, queue depth,\navailability"]
  P --> H["Headroom rules\npeak factor, retry factor,\nburst budget"]
  P --> A["Alerting\nthresholds, runbooks,\nescalation"]
  P --> C["Cost\ncompute, operations,\nrisk premium"]
  P --> R["Review cadence\nowner, frequency,\ntriggers"]

1. Workload model. Jobs/day, peak-hour factor, average and p95 duration, retry factor. Numbers with sources and measurement windows.
2. SLOs. Quantified targets: “95% of jobs start within 60 seconds of being queued”, “99% of jobs complete successfully”, “queue depth never exceeds 10 for more than 5 minutes”. SLOs without numbers are wishes; SLOs with numbers are commitments.
3. Headroom rules. The multiplicative buffer between measured load and pool capacity. A common rule: pool sized for 2x the steady-state concurrency limit, with explicit rules for burst and retry storms.
4. Alerting. Thresholds derived from SLOs. An SLO of “queue depth never exceeds 10” alerts at 8, not at 10 - the alert must fire before the SLO is breached, not when.
5. Cost. Compute, operations, and risk premium broken out per month. The hosted-vs-self decision, with the break-even calculation.
6. Review cadence. Who owns the plan, how often it is re-derived, and what triggers an out-of-cycle review (a new repo joins, a dependency doubles build time).

Translating the workload into SLOs

The SLOs are not invented; they are derived from the workload model. A team whose peak-hour arrival rate is 3 jobs/minute with average duration 8 minutes has a steady concurrency of 24 jobs. The SLO is built from there:

ARRIVAL_PER_MIN=3
DURATION_MIN=8
CONCURRENCY=$(awk -v a="$ARRIVAL_PER_MIN" -v d="$DURATION_MIN" \
  'BEGIN { printf "%.0f", a*d }')
HEADROOM=2
POOL_SIZE=$((CONCURRENCY * HEADROOM))
echo "Concurrency limit: $CONCURRENCY; pool with 2x headroom: $POOL_SIZE"

The pool size is 48 runners. The SLO is built against that:

• Wait-time SLO. 95% of jobs start within 60 seconds. This requires the pool to operate below 80% utilisation, which the sizing delivers at the modelled arrival rate.
• Queue-depth SLO. Queue depth never exceeds 10. The pool can absorb a 20% burst over the modelled rate without breaching.
• Availability SLO. 99% of jobs complete successfully. This is a workload SLO; runner availability is part of the calculation, not all of it.

The cost section

The cost section makes the financial commitment explicit. For a hosted pool sized at 48 medium runners with 8-minute average jobs:

• Compute. 48 runners × 8 hours × 60 min × $0.008 = $184/day at full saturation; the real number depends on actual utilisation, which the workload model gives.
• Operations. Engineering time for patching, image rebuilds, registration, secrets rotation. Documented in hours per month and converted to a dollar figure.
• Risk premium. The expected cost of an incident response, an audit, or a compliance gap. A simple model uses the team’s last three incidents and their response cost; a more sophisticated model uses insurance-style actuarial tables.

The total monthly cost is the line item that goes into the budget request. Without it, the budget conversation happens without the data.

The review cadence

A plan that is not reviewed is a plan that has rotted. The cadence has three components:

• Quarterly review. Re-derive the workload model from the last quarter’s metrics. Update arrivals, duration, retry factor, headroom rules.
• Out-of-cycle triggers. A new repository joins the org; a major dependency upgrade changes build time; a security incident reveals a runner-config gap; a cloud price change shifts the hosted-vs-self break-even.
• Owner. A named engineer or team. The plan is a document, but ownership is what keeps it current. Plans without owners go stale within two quarters.

A capacity plan in one page

A reasonable capacity plan fits on a single page. A template:

# Runner capacity plan - $TEAM - $QUARTER

## Workload
- Jobs/day: 1,800 (peak hour factor 0.12)
- Average duration: 8 min; p95: 18 min
- Retry factor: 1.2

## SLOs
- 95% of jobs start within 60 s of being queued
- Queue depth less than 10 for more than 99% of the time
- 99% of jobs complete successfully

## Sizing
- Concurrency limit: 24
- Pool with 2x headroom: 48 runners
- minRunners: 1; maxRunners: 60

## Cost
- Compute (hosted): $COMPUTE_USD/month
- Operations: $OPS_HOURS hours/month ($OPS_USD)
- Risk premium: $RISK_USD/month
- Total: $TOTAL_USD

## Alerting
- Queue depth greater than 8 for 5 min: page on-call
- Listener disconnected: page on-call
- Utilisation greater than 80% for 15 min: warn

## Review
- Owner: platform team
- Cadence: quarterly; triggers listed above

The format matters less than the discipline of writing it. A team that writes the plan, even roughly, will revise it when reality diverges. A team that does not will discover the divergence in production.

Production discipline

1. Write the plan before the next capacity decision. A plan written after the decision is a post-mortem, not a plan.
2. Quantify every SLO. “Jobs should be fast” is not an SLO.
3. Alert at 80% of the SLO threshold. The SLO breach is the incident, not the warning.
4. Name an owner. Documents without owners go stale.
5. Review quarterly or on trigger. Whichever comes first.

Cross-course references

• Git, CI/CD & GitOps - Part LXIII (Observability) covers the alerting patterns referenced in this lesson.
• FinOps for Production Sysadmins - Part XII (CapacityPlans) covers the same artefact for general workloads.
• SRE for Production Sysadmins - Part IV (SLODocs) covers the SLO format in depth.

Quiz