Capacity utilisation bands - the 90%-on-3-nodes trap · Linux

Capacity utilisation bands are the discipline that keeps services running. The 90%-on-3-nodes trap is the classic mistake. This lesson covers the design.

The 90%-on-3-nodes trap

A 3-node cluster with each node at 90% utilisation looks fine on the dashboard. It is not. Work out the total load first, then divide it by the survivors:

Total load    = nodes x per-node utilisation
              = 3 x 0.90
              = 2.70 node-units

Survivors after one node fails = 2

Per-survivor  = total load / survivors
              = 2.70 / 2
              = 1.35 = 135% per surviving node

The survivors cannot serve 135%. They saturate, run queues build, Corosync tokens are missed, the peers declare each other dead and fence. One node failure becomes a full cluster outage.

The same arithmetic applies to a planned drain. Taking one node out for patching is a node loss the cluster cannot tell apart from a crash.

The right design

For an N-node cluster that must survive k simultaneous node losses, the maximum safe steady-state per-node utilisation is:

U_max = (N - k) / N

3 nodes, tolerate 1 failure (N+1): U_max = 2/3 = 66.7%
3 nodes, tolerate 1 failure and 1 node in maintenance (N+2): U_max = 1/3 = 33.3%
2 nodes, tolerate 1 failure: U_max = 1/2 = 50%
5 nodes, tolerate 1 failure: U_max = 4/5 = 80%

U_max is a ceiling, not a target. Run below it so that growth does not silently breach N+1. A 3-node N+1 cluster run at 50% lands the survivors at 3 x 0.50 / 2 = 75% after a failure - uncomfortable but serviceable, with room left for the traffic spike that a failover usually causes.

Larger clusters are cheaper per unit of resilience. The failed node’s share is split more ways, so U_max rises with N for the same k.

Utilisation bands

Bands are fractions of U_max, not fixed percentages. The same absolute number is safe on a 5-node cluster and fatal on a 3-node one.

< 0.75 x U_max: healthy. Headroom for failure and growth.
0.75-0.90 x U_max: caution. Watch the growth rate.
0.90-1.00 x U_max: warning. Plan scaling now.
> U_max: critical. N+1 is already breached; a single node loss takes the service down.

For a 3-node N+1 cluster, U_max is 66.7%, so the bands land at:

healthy below 50%
caution 50-60%
warning 60-66.7%
critical above 66.7%

Adjust the target within the healthy band for the workload:

Stable, predictable load: sit near the top of healthy.
Variable or bursty load: sit lower, so a burst does not cross into warning.
Critical services: sit lower still, and size for N+2.

Alert thresholds

Alert on the band boundaries, so the alerts and the bands cannot drift apart:

0.75 x U_max (warning): investigate the growth rate.
0.90 x U_max (critical): start the scaling change.
U_max (emergency): N+1 is breached, scale immediately.

For the 3-node N+1 example those are 50%, 60% and 66.7%.

Scale before the emergency. The lead time for new capacity is days to weeks.