Maintenance capacity - what a change window costs the cluster

The cluster does not know why a node is gone. A drained node being patched and a node whose power supply failed present the same arithmetic to the survivors: the work has to go somewhere, and there is one less place for it to go.

Teams routinely size for failure and then schedule maintenance as if it were free. It is not. It is a failure you chose the timing of, and choosing the timing is the only advantage it gives you.

The general ceiling

The N+1 lesson gives the single-loss form: N x u / (N - 1) on each survivor. Generalise it to k nodes gone, from any cause:

load per survivor = N x u / (N - k)

Set that to 1.0 and solve for the highest steady-state utilisation that survives k losses:

u_max = (N - k) / N

Tabulated, with the survivor load shown at the ceiling:

Two things fall straight out of the table.

Larger clusters are cheaper per unit of resilience. Going from three nodes to six raises the k=1 ceiling from 66.7% to 83.3%, because the failed node’s share is split five ways instead of two.

The second loss costs far more than the first on a small cluster. On three nodes, tolerating a second loss halves the ceiling from 66.7% to 33.3% - you are paying for two thirds of the cluster to be idle. On eight nodes the same step costs 12.5 points. This is why “just add N+2” is sound advice on a large fleet and an expensive one on a three-node cluster.

Maintenance is k=1, for the length of the window

During a patching window with one node drained, the cluster is running at N - 1. Its remaining tolerance for a real failure is k = 0. Any fault during the window is an outage.

This is not a theoretical exposure. Maintenance is exactly when faults are most likely: you are rebooting hosts, restarting services, changing kernels, and touching storage. The window raises the failure rate at the same moment it removes the protection against failure.

Sizing the exposure

You have three ways to deal with a maintenance window that removes your redundancy, and they are genuine alternatives rather than a ranked list.

Shorten the window. Exposure is duration times failure rate. Halving the drain time halves the risk, costs nothing but preparation, and is usually the largest available win: staged artefacts, pre-downloaded packages, a rehearsed procedure and a kexec or fast-reboot path all attack the same term.

Accept the exposure, explicitly. For many services, two hours a month at N+0 with a known and communicated risk is a perfectly rational trade against the cost of another node. The word that matters is “explicitly”: the exposure appears in the change record, the on-call knows, and the severity of a fault during the window is agreed in advance.

Buy the headroom. Size for k = 2 so the maintenance window still leaves N+1. On a small cluster this is expensive, and the arithmetic is worth doing before proposing it.

$ awk 'BEGIN { for (n=3; n<=8; n++) printf "N=%d  k=1 ceiling %.1f%%  k=2 ceiling %.1f%%  extra nodes for same load at k=2: %.2f\n", n, 100*(n-1)/n, 100*(n-2)/n, n*(n-1)/(n-2) - n }'

N=3  k=1 ceiling 66.7%  k=2 ceiling 33.3%  extra nodes for same load at k=2: 3.00
N=4  k=1 ceiling 75.0%  k=2 ceiling 50.0%  extra nodes for same load at k=2: 2.00
N=5  k=1 ceiling 80.0%  k=2 ceiling 60.0%  extra nodes for same load at k=2: 1.67
N=6  k=1 ceiling 83.3%  k=2 ceiling 66.7%  extra nodes for same load at k=2: 1.50
N=7  k=1 ceiling 85.7%  k=2 ceiling 71.4%  extra nodes for same load at k=2: 1.40
N=8  k=1 ceiling 87.5%  k=2 ceiling 75.0%  extra nodes for same load at k=2: 1.33

Read that last column carefully before proposing N+2 on a three-node cluster. Carrying the same load with two spares instead of one means six nodes rather than three. That is a budget conversation, not a configuration change, and it is better had with the number in hand.

Bounding the batch

The rolling maintenance lessons bound batch size by quorum: B <= N - (floor(N/2) + 1). Capacity is the second bound, and on a large cluster it usually binds first.

capacity bound:  B <= N x (1 - u)      (survivors stay at or below 100%)
quorum bound:    B <= N - (floor(N/2) + 1)

Take the smaller. Then apply risk appetite on top.

An eight-node cluster at 70% utilisation:

capacity: B <= 8 x (1 - 0.70) = 2.4  ->  B = 2
quorum:   B <= 8 - (4 + 1)    = 3

Batch size = min(2, 3) = 2

Take three and the five survivors carry 8 x 0.70 / 5 = 112%. Quorum holds perfectly - the cluster stays up and reports healthy - while the service is saturated. That is the failure mode this bound exists to prevent, and it is nastier than losing quorum because the cluster tooling shows nothing wrong.

Put maintenance in the plan

Maintenance capacity is a line item in the capacity plan, not an afterthought. For each cluster, record:

• Steady-state p95 utilisation per node, and the k=1 ceiling for the node count
• Batch size, as the smaller of the quorum and capacity bounds
• Drain-to-return duration per node, measured on the last window rather than estimated
• Total exposure per maintenance cycle: batches x duration, and whether the cluster is at N+0 during it
• The decision taken about that exposure: shorten, accept, or buy - with a name against it

Those five lines convert an argument about whether the cluster is “big enough” into a statement about what it can do and what it costs to keep it patched.

Knowledge check