Cluster capacity during a kernel campaign - the reboot budget · Linux

linux-maintenance-capacity establishes the general rule: a maintenance window consumes the same headroom as a node failure, so a cluster doing planned work is a cluster running with its redundancy already spent.

Kernel upgrades are the case where that rule bites hardest, and for a reason that has nothing to do with the kernel. It is the only change in this course that combines a long per-node outage with a long mandatory hold — and multiplying those by the node count turns “apply a security patch” into a schedule measured in weeks, during every hour of which the cluster is one node short.

Nobody measures the right number

Ask how long a reboot takes and you will be told sixty seconds, because that is what a virtual machine does. The number that matters is not the reboot. It is node-out time: from the moment the node stops carrying work to the moment it is carrying its full share again.

node-out time =
    drain                       seconds to minutes
  + shutdown                    stopping services, unmounting, flushing
  + firmware POST               the one nobody counts
  + bootloader + kernel + initrd
  + userspace start             systemd units to their targets
  + application readiness       pool warm-up, cache fill, JIT
  + return + verification
  + observation hold            the big one

Firmware POST is the term that surprises people who have only operated virtual machines. A dense server with a large memory population, several HBAs and a RAID controller that inventories its drives on every boot can spend five to ten minutes before the bootloader is even reached. Memory training after a DIMM change can add several more. None of it is visible to any tool running on the host, because the host is not running.

Read-only / Safesystemd-analyze— On a UEFI host systemd-analyze reports a firmware figure taken from the firmware performance table, which is the only easy way to see POST time from inside the OS. On a BIOS host, or a VM, the firmware and loader lines are simply absent - which is exactly why operators who work mostly with VMs have no intuition for how long POST takes on physical hardware. Note that this measures boot, not node-out time: it starts after the shutdown and ends before the application is ready.

$ systemd-analyze

Startup finished in 8.412s (firmware) + 2.104s (loader) + 891ms (kernel) + 1.464s (initrd) + 24.367s (userspace) = 37.238s
graphical.target reached after 24.366s in userspace.

Illustrative output

The honest way to get the number is to measure it on the canary node and write it in the plan:

# On the control node, immediately before the reboot
date -u +%s > /tmp/reboot-start

# ... reboot the node, wait for it to return to full service ...

# When the node is genuinely back
echo "node-out seconds: $(( $(date -u +%s) - $(cat /tmp/reboot-start) ))"

Then check where the time went:

systemd-analyze                  # firmware, loader, kernel, initrd, userspace
systemd-analyze blame | head     # which units dominated userspace
systemd-analyze critical-chain   # the dependency path that set the total

A node that passes its health check is not yet carrying its share

The reboot finishes, systemctl is-system-running says running, /healthz returns 200, the load balancer puts the node back — and the node then serves at a fraction of its normal capacity for several minutes.

The reasons are all mundane and all real: the page cache is empty so every read hits disk, connection pools are empty so every request pays a handshake, a JIT-compiled runtime is interpreting until it warms up, and any local cache the application keeps starts cold.

Two consequences.

First, the surviving nodes are carrying more than the N-1 capacity model predicts for a while after the node returns, because the returned node is contributing a fraction of a node. On a tier that is already tight, the moment of highest risk in the whole maintenance is shortly after a node comes back, not while it is out.

Second, this is why the observation period is defined by node behaviour and not by a stopwatch. A node whose error rate and latency have converged with its peers is warm. A node that has been up for ten minutes may not be.

Where the application supports it, ramp the traffic instead of switching it on: HAProxy’s slowstart on a server line raises its weight gradually after it comes back, which is exactly this problem solved at the load balancer.

Campaign arithmetic

Now multiply. For N nodes, batch size B, node-out time T and hold H between batches:

campaign duration  ~=  ceil(N / B) x (T + H)
exposure           =   B nodes down, continuously, for that whole duration

Put the recommended kernel numbers in:

N = 40, B = 1, T = 20 min, H = 24 h
  ceil(40/1) x (0.33 + 24) h  =  40 x 24.33 h  =  973 h  =  ~41 days

Forty days to apply a kernel patch, with the tier running at N-1 for the entire period. That is not a criticism of the 24-hour hold — the hold exists because kernel regressions are frequently load-dependent and take a full daily cycle to appear. It is an argument that the schedule has to be a deliberate decision rather than an emergent property of a policy nobody did the arithmetic on.

The 24-hour hold applies to the canary, not to every node

The hold is a statement about the change, not about each individual host. Its purpose is to discover whether this kernel has a regression on this hardware and this workload. Once the canary has been through a full daily cycle — including the nightly batch, the backup window and the morning peak — that question has been answered.

The standard shape is therefore:

Batch 1   1 node,  hold 24 h        does this kernel regress?
Batch 2   1 node,  hold 1 h         did anything change on real hardware variation?
Batch 3+  B nodes, health gate      the ceilings from linux-batch-sizing-and-rollback

N = 40, canary 24 h, then B = 2 with a 1 h gate, T = 20 min
  24 h + ceil(39/2) x (0.33 + 1) h  =  24 + 20 x 1.33  =  ~51 h  =  ~2 days

Two days rather than forty-one, with the deep validation still happening once, on the node where it is useful. Holding every node for 24 hours does not buy twenty times the confidence; it buys the same confidence spread over a period long enough that the next CVE lands mid-campaign.

The three levers against a deadline

A kernel CVE with a remediation deadline turns the campaign length into a hard constraint. There are exactly three things you can change, and each has a cost.

Shorten the hold. Cheap, and defensible after the canary has proven the kernel. The cost is that a load-dependent regression appearing on node 12 now has eleven other nodes already carrying it. Bound this by keeping the canary hold intact and only shortening the subsequent gates.

Increase the batch. Bounded, hard, by the two ceilings from linux-batch-sizing-and-rollback: B <= N - quorum and the capacity test at peak load. A deadline does not move either ceiling. Exceeding the quorum ceiling converts the campaign into a guaranteed outage, which is a worse outcome than missing the deadline.

Decouple the reboot from the fix. Live patching applies the security fix now and moves the reboot to the normal maintenance schedule. linux-live-kernel-patching-concepts covers what it does and does not cover — it handles a subset of kernel fixes and no driver changes — but where it applies, it is the only lever that satisfies the deadline without spending capacity.

The lever that does not exist is “do it faster with the same safety”. The observation period is where the safety lives.

What the fleet looks like mid-campaign

For most of the campaign the fleet is running two kernel versions. That is normal, expected, and supported: the kernel maintains a stable userspace ABI, so a mixed-kernel fleet behind a load balancer is unremarkable and a mixed-kernel Corosync or Pacemaker cluster is explicitly supported during upgrades.

Two things are worth checking rather than assuming.

Out-of-tree modules must exist for both kernels. ZFS, DRBD, vendor HBA and NIC drivers are built per kernel version by DKMS. The pre-reboot gate in linux-kernel-package-install catches a build failure on the node being patched; what it does not catch is a module version that behaves differently across the two kernels in the fleet — relevant for anything that talks between nodes, DRBD in particular.

The campaign has to be tracked. A campaign that runs for days across shift changes needs a record of which hosts are done, which is the subject of the next lesson.

Nobody measures the right number

Campaign arithmetic

The three levers against a deadline

What the fleet looks like mid-campaign

Knowledge check