Sizing a cluster: a worked example with real numbers

The previous lesson gave the concepts. This one does the arithmetic, once, completely, on one inventory, and arrives at a bill of materials and a date. Every number below follows from the one above it, and the working is shown so you can substitute your own inventory and get your own answer.

The single most important structural idea, and the one that most sizing spreadsheets get wrong:

Guest demand fails over. Per-node overhead does not.

When a node dies, its guests move to the survivors. Its Ceph OSDs, its pvedaemon, its ARC, its kernel — those are simply gone, and each surviving node is still carrying its own. Any formula that divides total demand by node count without separating these two will undersize the cluster.

Step 0: the inventory

Totals across 27 guests:

6,040 GiB is 5.90 TiB of provisioned guest storage.

pvesh get /cluster/resources --type vm --output-format json | jq -r '
.[] | select(.status=="running")
| [.vmid, .name, .maxcpu, (.maxmem/1073741824|floor), (.mem/1073741824|floor), (.maxdisk/1073741824|floor)]
| @tsv' | sort -n

pvesh get /cluster/resources --type vm --output-format json | jq '
[ .[] | select(.status=="running") ]
| { guests: length,
    vcpu: ( map(.maxcpu) | add ),
    ram_gib: ( map(.maxmem) | add / 1073741824 | floor ),
    disk_gib: ( map(.maxdisk) | add / 1073741824 | floor ) }'

Step 1: growth

Planning horizon three years, growth 15% per year, compounding.

multiplier = 1.15 ^ 3 = 1.52

Compounding matters. Three years at 15% is 52% more, not 45% — and at 20% it is 73%, not 60%. Linear growth assumptions are the commonest arithmetic error in capacity plans and they always err in the same direction.

Step 2: vCPU to physical cores

vCPU is not a unit of capacity. The conversion is an overcommit ratio, and it is a per-workload decision rather than a cluster-wide constant.

Two decisions embedded there, both worth stating explicitly in the plan.

Physical cores, not SMT threads. A 32-core CPU presents 64 threads. Counting them as 64 units of capacity assumes an SMT sibling delivers as much as a physical core, which it does not — the gain is workload dependent and commonly well under half. Plan against physical cores and treat SMT as the headroom that absorbs your estimation error.

1:1 for latency-sensitive guests. Overcommitting CPU does not make a guest slower on average; it makes it slower sometimes, which is exactly what a tail-latency SLO measures. The databases and the broker get real cores.

At year 3: 119 × 1.52 = 181 cores of guest demand.

Step 3: the survivable ceiling, and node count

Here is the formula the rest of the lesson hangs on. With N nodes each at utilisation u, losing one spreads N × u over N − 1 survivors:

per-node load after one failure = N x u / (N - 1)

Set that to 100% and solve for the maximum steady-state utilisation:

We will design to 90% after a single failure, not 100%, which leaves a tenth of each survivor for uneven redistribution and for the incident itself.

Now separate the two kinds of demand.

• Guest demand G — moves on failure. 181 cores at year 3.
• Per-node overhead H — stays put. For a hyper-converged node: roughly 1 core per Ceph OSD plus 2 cores for PVE daemons and the kernel. With 5 OSDs per node that is 7 cores.

The requirement, per node:

G / (N - 1) + H  <=  0.90 x C

Choose N = 5, 2 × 32 physical cores per node. Four nodes needs a much larger CPU SKU for no availability gain; six nodes buys a further node’s worth of resilience at the cost of a sixth chassis, a sixth licence and a sixth thing to patch.

Check the design point: at year 3 a surviving node carries 181/4 + 7 = 52.3 of 64 cores, which is 81.7%. Under 90%. Steady state with all five up is 181/5 + 7 = 43.2 of 64, or 67.5% — below the 80% five-node ceiling with room to spare.

Step 4: memory

Memory does not overcommit the way CPU does, and the mechanisms that make it appear to — ballooning, KSM — all stop working during exactly the event you are sizing for. Size against configured maximum memory.

• Guest demand G: 1,350 GiB at year 3.
• Per-node overhead H: Ceph OSD memory at the 4 GiB default target × 5 OSDs = 20 GiB, plus 8 GiB for PVE and the kernel = 28 GiB.

Memory is quantised by DIMM configuration, so it jumps. Two defensible answers: five nodes at 512 GiB, or six nodes at 384 GiB. Both work; the first is fewer things to operate, the second gives more failure resilience and more Ceph OSDs.

We take five nodes at 512 GiB. Note that this decision was driven by memory, not by CPU — which is typical, and is why sizing every dimension matters rather than sizing the one you happen to think about.

Step 5: IOPS, and what replication does to them

Two conversions, both routinely skipped.

Peaks do not coincide. Summing 27 guests’ individual peaks assumes every one of them peaks simultaneously. Apply a coincidence factor — 70% is a defensible starting point, and the honest version is to measure the cluster’s actual aggregate peak rather than to sum.

year 3 client IOPS = 66,900 x 0.70 = 46,800

Replication multiplies writes. On a size=3 Ceph pool, one client write becomes three device writes; reads are served from the primary and are 1:1. At a 70/30 read/write mix:

Now spread that over the OSDs that survive a node failure — five nodes × 5 OSDs = 25 OSDs, of which 20 remain:

74,880 / 20 = 3,744 device IOPS per OSD

Step 6: Ceph raw to usable

This runs backwards from what you need to what you must buy, and there are three multipliers, applied in order.

Starting point: 8.97 TiB of provisioned guest disk at year 3. Round to 9.0 TiB usable.

1. Replication. size=3 means three copies:

9.0 TiB usable x 3 = 27.0 TiB of data

2. The near-full ratio. Ceph’s default nearfull_ratio is 0.85 and full_ratio is 0.95; crossing full stops writes. Plan to stay under near-full:

27.0 / 0.85 = 31.8 TiB of raw capacity

3. The failure case. After losing a node, Ceph re-replicates to restore three copies — onto the survivors. So that 31.8 TiB must be present on N − 1 = 4 nodes, not on five:

31.8 / 4 = 7.95 TiB raw per node

Round up to 8 TiB raw per node. With 1.92 TB NVMe drives (1.75 TiB each):

5 x 1.92 TB = 8.73 TiB per node

Five OSDs per node, 25 OSDs, 43.7 TiB raw across the cluster.

Sanity check both states:

Under 85% in both. The design survives a node loss and the re-replication that follows it, which is the part people forget: a cluster can have room for the data and no room to make the third copy again.

Step 7: network, sized by the recovery window

The instinct is to size the cluster network for peak client throughput. That is the smaller of the two numbers.

Client replication traffic. 14,040 client write IOPS at a 16 KiB average is 225 MiB/s. Each write is sent to two replicas, so the cluster network carries twice that:

225 MiB/s x 2 = 450 MiB/s = 3.8 Gbit/s

Recovery traffic. Losing a node means regenerating its share of the data — 27.0 / 5 = 5.4 TiB — and distributing it across the survivors. That number, divided by the recovery window you are willing to accept, is the requirement:

The recovery window is a durability decision, not a performance one. Until re-replication completes, some placement groups have two copies instead of three, and a second failure in that window can lose data. A cluster that takes eight hours to recover has an eight-hour period of elevated risk after every disk or node failure.

Combining a two-hour target with client traffic gives roughly 3.8 + 6.6 = 10.4 Gbit/s on the cluster network at year 3.

Note that 10 GbE would technically carry the traffic. It is rejected on the recovery argument: at 10 Gbit/s shared with client I/O, the two-hour window becomes most of a day, and LACP hashing means a single recovery flow cannot exceed one link’s 10 Gbit/s regardless of the bond.

Step 8: the answer, and its expiry date

Five nodes, each:

Now the number that makes this a plan rather than a purchase order.

A capacity plan’s output is a date, because a single utilisation figure is simultaneously healthy and a countdown, and only the growth rate distinguishes them. For each dimension, find the guest demand at which the after-failure load reaches 90%, then solve for when growth gets there.

CPU. Ceiling per surviving node 0.90 × 64 = 57.6 cores, minus 7 overhead, is 50.6 guest cores per node × 4 survivors = 202 cores.

119 x 1.15^n = 202   ->   n = 3.8 years

Memory. 0.90 × 512 = 460.8 GiB, minus 28 overhead, is 432.8 per survivor × 4 = 1,731 GiB.

888 x 1.15^n = 1731   ->   n = 4.8 years

Storage. Raw on 4 survivors is 34.9 TiB; at 85% that is 29.7 TiB of data; at size=3 that is 9.9 TiB usable.

5.90 x 1.15^n = 9.9   ->   n = 3.7 years

The plan expires at 3.7 years, on storage. That is the deliverable: a sixth node, or additional OSDs, must be in service before then — and storage has the longest procurement lead time of the three, so it is the one that needs the earliest decision.

pvesh get /cluster/resources --output-format json | jq '
( [ .[] | select(.type=="node") ] ) as $n
| ( [ .[] | select(.type=="vm" and .status=="running") ] ) as $v
| { nodes: ($n|length),
    node_cores_total: ( $n | map(.maxcpu) | add ),
    node_mem_gib_total: ( $n | map(.maxmem) | add / 1073741824 | floor ),
    guest_vcpu: ( $v | map(.maxcpu) | add ),
    guest_mem_gib: ( $v | map(.maxmem) | add / 1073741824 | floor ) }'

ceph df
ceph osd df tree | head -20

Common mistakes

• Dividing total demand by N × ceiling. The requirement is that the work fits on N − 1 nodes. G/(N-1) + H is the formula.
• Applying the two-node 50% rule to larger clusters. The ceiling is (N-1)/N: 66.7% at three, 75% at four, 80% at five.
• Not separating guest demand from per-node overhead. OSDs and host daemons do not fail over, and recovery makes them heavier at the worst moment.
• Counting SMT threads as cores. Plan against physical cores; treat SMT as the margin that absorbs estimation error.
• Entering KSM or ballooning savings as capacity. Both fail during the failover you are sizing for.
• Summing per-guest peak IOPS. Peaks do not coincide; apply a coincidence factor or measure the aggregate.
• Forgetting the write amplification of size=3. One client write is three device writes.
• Sizing Ceph raw capacity across all N nodes. Re-replication after a failure happens onto the survivors, so the requirement is on N − 1.
• Sizing the cluster network from client throughput. The recovery window is the larger number and it is a durability decision.
• Assuming linear growth. Three years at 15% is 52%, not 45%.
• Producing a utilisation figure instead of a date.

Key takeaways

• Guest demand fails over; per-node overhead does not. Size with G/(N-1) + H ≤ 0.90 × C.
• The survivable ceiling is (N-1)/N — 50% at two nodes, 66.7% at three, 75% at four, 80% at five. It is a check on a design, not a divisor for sizing.
• Convert vCPU to physical cores with a per-workload overcommit ratio, 1:1 for anything with a tail-latency SLO.
• Size memory against configured maximums. KSM, ballooning and thin provisioning all stop helping during a failover.
• Apply a coincidence factor to summed IOPS, then multiply writes by the replication factor to get device IOPS.
• Ceph raw = usable × size ÷ nearfull_ratio ÷ (N-1). The re-replication must fit on the survivors.
• Prefer more, smaller OSDs: recovery is more parallel and each failure is a smaller share.
• Size the cluster network from the recovery window, because the window is a period of reduced redundancy.
• The deliverable is a date, and it is the minimum across every dimension. Here: storage, at 3.7 years.
• Alert on after-failure utilisation, not current utilisation.

Knowledge check