Capacity planning and N+1 design

Why this matters in production

Capacity exhaustion is one of the most preventable outage causes. A cluster that runs at 95 % utilisation has no headroom for HA failover, no room for growth, and no slack for maintenance windows. The cost of buying more capacity now is always less than the cost of an outage later.

Mental model

Three numbers drive every sizing decision:

1. Sum of workload requirements — total vCPU, RAM, IOPS, storage the business needs.
2. Headroom for growth — the multiplier on top of (1) for the planning horizon.
3. N+1 (or N+2) for HA — capacity to run the full workload minus one node (or two).

flowchart LR
  A[Workload inventory] --> SUM[Sum of requirements]
  G[Growth target] --> SUM
  SUM --> PER[Per-node requirement]
  HA[HA headroom: N+1 or N+2] --> PER
  PER --> NODES[Node count]

Worked example

Workload inventory (simplified):

Target: 40 % growth over three years. N+1 (cluster of 3 nodes, can lose 1).

After growth multiplier (×1.4):

• Total vCPU: 26 × 1.4 ≈ 37
• Total RAM: 84 × 1.4 ≈ 118 GB
• Total IOPS: 12 200 × 1.4 ≈ 17 100
• Total storage: 820 × 1.4 ≈ 1 150 GB

The cluster has 3 nodes and must keep running with 2. So each node must be sized to carry its share when there are only 2 nodes left — divide by N − 1, which is 2, not by 3 and not by the survivable-ceiling percentage:

• Per-node vCPU: 37 / 2 ≈ 19
• Per-node RAM: 118 / 2 = 59 GB
• Per-node IOPS: 17 100 / 2 ≈ 8 600
• Per-node storage: 1 150 / 2 ≈ 575 GB

This is the per-node requirement. Each node must be sized to handle its share during a node failure, not during normal operation.

The survivable ceiling

The N+1 requirement above has a direct expression as a utilisation limit. With N nodes each at utilisation u, one failure spreads N × u over N − 1 survivors:

per-node load after one failure = N x u / (N - 1)

Set that to 100 % and solve for u:

The ceiling rises with node count, because each survivor absorbs a smaller share. That is the capacity argument for larger clusters, independent of the availability one.

Per-node overhead does not fail over

The arithmetic above treats the whole workload as mobile. Part of a hyper-converged node’s load is not.

When a node dies its guests move to the survivors. Its Ceph OSDs, its pvedaemon, its ZFS ARC and its kernel are simply gone — and every surviving node is still running its own. So the per-node requirement is:

capacity per node  >=  guest demand / (N - 1)  +  per-node overhead

For a hyper-converged node, budget roughly 1 core and the osd_memory_target (4 GiB by default) per Ceph OSD, plus a couple of cores and 8 GiB for PVE and the kernel. Recovery after a failure makes that overhead heavier at exactly the moment guest demand rises, which is one reason to design to 90 % after a failure rather than 100 %.

N+1 vs N+2

Storage capacity calculation

For Ceph, raw capacity is not the same as usable capacity. With size=3 replication:

usable = (raw capacity) / replication factor

A 3-node Ceph cluster with 4 × 1 TB SSDs per node has 12 TB raw, and every object is stored three times, so roughly 4 TB is usable. Two further reductions apply before you can allocate any of it:

• The near-full ratio. Ceph’s default nearfull_ratio is 0.85 and full_ratio is 0.95; crossing full stops writes cluster-wide. Divide by 0.85 to get the raw capacity you must actually own.
• Re-replication after a failure. When a node is lost, Ceph restores the third copy onto the survivors. So the raw requirement has to be met by N − 1 nodes, not by all N. A cluster sized across all three has room for the data and no room to make the copy again, and stays degraded until someone buys disks.

raw required = usable x size / nearfull_ratio,  present on (N - 1) nodes

For ZFS local storage:

usable = (raw capacity) - (parity overhead) - (slop for snapshots, copies)

A 4-disk RAIDZ2 has 50 % usable capacity; reserve 20 % for snapshots, copy-on-write fragmentation, and copy-on-write overhead.

The output of a capacity plan is a date

A single utilisation figure is not a finding. 60 % is healthy, and 60 % is also a countdown; only the growth rate distinguishes them.

Take the 3-node cluster above with a 66.7 % ceiling and 20 % annual growth from 50 % per node:

• Year 0: 50 % — comfortable.
• Year 1: 60 % — still under the ceiling.
• Year 2: 72 % — past it. A single failure is now an outage.

So the useful output is “a fourth node must be in service before year 2”. That is something a budget cycle can act on, by someone who does not operate the cluster. A utilisation percentage is not.

Run the same arithmetic for every dimension — cores, memory, storage, IOPS — and the plan expires on the earliest of them. It is frequently not the one being watched.

Common mistakes

• Sizing for normal operation, not failover. A cluster that fits all VMs on 3 nodes is at 100 % utilisation — it cannot lose a node.
• Dividing by N × ceiling instead of by N − 1. The ceiling checks an existing design; it is not a sizing divisor.
• Quoting the two-node 50 % rule at a larger cluster. The ceiling is (N−1)/N.
• Treating per-node overhead as mobile. Ceph OSDs and host daemons do not fail over, and recovery makes them heavier during the failure.
• Forgetting that ZFS snapshots and Ceph replication consume capacity that is not “data” but is still on the disks.
• Sizing Ceph raw capacity across all N nodes rather than the survivors.
• Assuming growth is linear. Three years at 15 % is 52 %, not 45 % — and cloud-native workloads often grow in steps rather than smoothly (a new application launches; all of a sudden the cluster is full).
• Reporting a utilisation figure instead of a date.

Key takeaways

• Size per node for the workload spread over N − 1 nodes, plus the per-node overhead that does not move — that is HA headroom.
• The survivable ceiling is (N−1)/N: 50 % at two nodes, 66.7 % at three, 75 % at four, 80 % at five. It rises with node count.
• The ceiling is where a failure becomes an outage, not where to operate. Set the target below it.
• Include a growth multiplier explicitly, and compound it.
• For Ceph, raw = usable × size ÷ nearfull_ratio, and it must fit on the survivors.
• Alert on after-failure utilisation, not current utilisation.
• Deliver a date, not a percentage.

Production considerations

Knowledge check