Lab: Cluster capacity model - design and forecast

This lab builds a capacity model for a sample cluster workload and documents the plan. The arithmetic is easy; the discipline is keeping one definition of capacity and testing every forecast year against a node failure.

Tasks

Task 1: Define the workload

Sample workload:

• 3-node cluster.
• Web application: nginx.
• Steady-state load: 1000 RPS, 200 GB memory total.
• Growth: 20% per year.
• Target availability: 99.95%.

Task 2: Fix the terms before you do any arithmetic

Most broken capacity plans are broken here, not in the maths. Write the terms down once and never mix them:

L0  Total steady-state load       = 1000 RPS
N   Nodes                         = 3
C   Per-node capacity (MEASURED)  = 667 RPS
NC  Total capacity  = N x C       = 2000 RPS

Per-node steady-state load = L0 / N = 333 RPS
Per-node utilisation       = 333 / 667 = 50%
Memory: 200 GB total = 67 GB per node

Note where C comes from. It is measured by load-testing a single node to the point where latency leaves its objective - it is not “the load a node happens to carry today”. If each node carries 333 RPS at 50% utilisation, then C = 333 / 0.5 = 667 RPS, and the cluster total is 3 x 667 = 2000 RPS.

Task 3: Forecast growth, then apply the N-1 test

Year 0: 1000 RPS, 200 GB
Year 1: 1200 RPS, 240 GB
Year 2: 1440 RPS, 288 GB
Year 3: 1728 RPS, 346 GB

Against total capacity NC = 2000 RPS, that is 50%, 60%, 72% and 86% utilisation. But those are the all-nodes-up numbers, and no node failure has been modelled yet. The whole point of an N+1 design is that the cluster survives losing a node, so the load must be checked against the capacity that remains:

Surviving capacity after 1 node loss = (N-1) x C = 1333 RPS

N+1 is satisfied in year Y only if  L(Y) <= (N-1) x C

Year 0: 1000 <= 1333  OK        (75% of surviving capacity)
Year 1: 1200 <= 1333  MARGINAL  (90% - past the 85% emergency band)
Year 2: 1440 >  1333  FAILS     (108% - one node loss is an outage)
Year 3: 1728 >  1333  FAILS     (130% - collapse)

The conclusion is the opposite of the comfortable reading. This cluster stops being genuinely N+1 during year 1, not year 3. Scale to 4 nodes before year 1 ends: (4-1) x 667 = 2001 RPS of surviving capacity, which covers year 3’s 1728 RPS with the same one-node-down guarantee.

Task 4: Define alert thresholds

Warning (60%): investigate growth
Critical (75%): plan scaling
Emergency (85%): scale immediately

Read those against surviving capacity, not total capacity, or the alert fires after the cluster has already stopped being resilient:

Thresholds expressed as load, against (N-1) x C = 1333 RPS
  60% warning   ->  800 RPS   crossed before Year 0
  75% critical  -> 1000 RPS   crossed at Year 0
  85% emergency -> 1133 RPS   crossed during Year 1

The cluster is already at the critical band on day one. That is the honest reading of a 3-node cluster carrying 1000 RPS with 667 RPS nodes, and it is the number that justifies the fourth node in the budget conversation.

Task 5: Maintenance headroom is additive, not shared

Failure headroom      : survive 1 node lost      -> L <= (N-1) x C
Maintenance headroom  : survive 1 node drained   -> L <= (N-1) x C
Both at once (N+2)    : a node fails DURING the
                        maintenance window       -> L <= (N-2) x C

Task 6: Document the plan

CAPACITY PLAN
=============
Workload: 3-node cluster, web application
Steady-state: L0 = 1000 RPS, 200 GB
Growth: 20% per year

Measured capacity:
- Per-node capacity C  = 667 RPS (load test, latency SLO)
- Total capacity  NC   = 2000 RPS
- Surviving capacity   = (N-1) x C = 1333 RPS

Per-node target:
- 333 RPS steady-state
- 67 GB memory steady-state
- 50% util on each node, all nodes up

Failure capacity:
- Survives 1 of 3 nodes lost while L <= 1333 RPS
- N+1 holds today at 75% of surviving capacity

Maintenance capacity:
- ADDITIVE to failure capacity, not shared
- Full cover during maintenance needs N+2: L <= 667 RPS
- Not met. Accepted risk: maintenance windows run without
  failure cover, capped at 60 min, off-peak, one node at a
  time, signed off by <service owner>

Alert thresholds (measured against surviving capacity):
- 60% warning   =  800 RPS
- 75% critical  = 1000 RPS
- 85% emergency = 1133 RPS

Forecast and N-1 verdict:
- Year 0: 1000 RPS -  75% of surviving capacity - PASS
- Year 1: 1200 RPS -  90% of surviving capacity - MARGINAL
- Year 2: 1440 RPS - 108% of surviving capacity - FAILS N+1
- Year 3: 1728 RPS - 130% of surviving capacity - FAILS N+1

Scaling plan:
- Add node 4 BEFORE the end of Year 1
  -> surviving capacity (4-1) x 667 = 2001 RPS, covers Year 3
- Re-measure C after any change to node spec or application
- Re-forecast every 6 months