Batch sizing and rollback - the change control discipline · Linux

Batch sizing and rollback are the change control discipline for HA clusters. Wrong batch size or no rollback turns a change into an outage.

Batch size has two hard ceilings

Batch size is how many nodes change at once. It is not a preference, and it is not a speed dial. Two ceilings bound it before risk appetite is even considered.

1. Quorum. An N-node Corosync cluster is quorate while at least floor(N/2) + 1 members are present. Every node in the batch is a node that may restart, reboot, or lose membership, so the batch size B is bounded by:

B  <=  N - (floor(N/2) + 1)

N = 3  ->  quorum 2  ->  B = 1
N = 4  ->  quorum 3  ->  B = 1
N = 5  ->  quorum 3  ->  B = 2
N = 7  ->  quorum 4  ->  B = 3

Exceed that and the survivors are inquorate. With the default no-quorum-policy=stop, an inquorate partition stops every resource it is running. The “faster” batch becomes a guaranteed full outage, caused by the maintenance rather than by any fault.

2. Capacity. The N - B surviving nodes must carry 100% of the load for the whole batch window. If steady-state per-node utilisation is above (N - B) / N, the survivors saturate and you get a latency incident instead of a clean rolling change. Reduce B until the arithmetic works.

Take the smaller of the two ceilings. Then apply risk on top: for high-risk changes - kernel, storage stack, cluster stack, anything that reboots - use B = 1 with a canary hold regardless of what the ceilings would allow.

When to use each

Change	Batch size
Kernel update	1 node, hold 24h
Security patch	1 node canary
Application update	`min(quorum ceiling, capacity ceiling)`
Configuration change	`min(quorum ceiling, capacity ceiling)`
Routine patch	1 node at a time, health gate between batches

The risk of the change chooses where you sit below the ceilings. It never lets you exceed them.

Between batches, gate on health rather than on a timer: quorum restored, all resources started, no failed actions, and the service’s own smoke test green. If the gate fails, the change stops there - a half-applied fleet is recoverable, a fully-applied broken fleet is not.

sudo pcs status --full          # quorate? any Failed Resource Actions?
sudo corosync-quorumtool -s     # expected votes vs total votes

Rollback

For every change, have a tested rollback:

Packages: capture the old .deb before the change, then reinstall it and hold. apt-get install --allow-downgrades PKG=VERSION only works while that version is still fetchable, and after a security update it usually is not - see below. On RHEL, sudo dnf history undo <id> works while the old package remains in the cache or the repository.
Configuration: revert from git.
Container: redeploy the previous image.
Service: restart with the previous config.

Test the rollback in staging. The first time should not be a real production change.

The version you want to roll back to is usually already gone

Debian and Ubuntu pockets carry one version per pocket. When a security update lands, the version it replaced is removed from -security and -updates, leaving only the current version and the original release version. The local archive cache does not save you either: the default APT::Periodic::AutocleanInterval empties /var/cache/apt/archives.

Check any host you maintain:

apt-cache madison nginx
ls /var/cache/apt/archives/*.deb | wc -l

Two distinct versions and an empty cache is the normal answer. The version your fleet was running last Tuesday is not obtainable, so apt-get install --allow-downgrades nginx=1.27.1-1 fails with Version '1.27.1-1' for 'nginx' was not found - during the incident, with the change window running.

Make the rollback an artefact you own before you start:

# PRE-CHANGE, on the first host of the first batch
OLD=$(dpkg-query -W -f='${Version}' nginx)
sudo install -d -m 0755 /var/backups/rollback
sudo apt-get install --download-only --reinstall -y "nginx=$OLD"
sudo cp /var/cache/apt/archives/nginx_*.deb /var/backups/rollback/

# ROLLBACK
sudo dpkg -i /var/backups/rollback/nginx_*.deb
sudo apt-mark hold nginx     # or unattended-upgrades re-applies it tonight

This is also why batch one exists. The first batch is where you find out whether the rollback artefact was captured and whether restoring it works - on a small enough blast radius that the answer is affordable.

linux-package-cache-and-cleanup states the constraint plainly: apt has no built-in rollback, so use a captured artefact or a snapshot.

Time the change

Time the change and the rollback:

START=$(date +%s)

# Apply change
...

END=$(date +%s)
echo "Change took $((END-START)) seconds"

If the change takes longer than expected, abort. If the rollback is faster, that’s the target for the next change.

Change record

Every change produces a record:

CHANGE: 2026-08-09 14:00 - nginx 1.27.2
DESCRIPTION: Security update for CVE-2024-XXXXX
NODES: 1 canary, then 1/3 cluster
CHANGE DURATION: 5 minutes per node
ROLLBACK DURATION: 3 minutes per node
VALIDATION: smoke test, latency, log scan
RESULT: success
NOTES: no issues observed

The change record is the audit trail. Compliance and post-incident review depend on it.