Sizing the patch run before you write it · Ansible

Patching is the highest-stakes ordinary thing automation does.

It is ordinary because it happens every month, on every host, and mostly without incident. It is high-stakes because it is the one routine change that can replace the kernel, restart every service, and take the machine down and back up again — and because the operator who wrote the play is usually asleep when it runs.

Everything Part XXXI taught about serial and failure tolerance applies here, and it applies harder. A rolling application deploy that goes wrong leaves you with a running host serving old code. A patch run that goes wrong can leave you with a host that does not boot.

So the first decision is not which module to call. It is how many hosts may be in flight at once, and where the run is permitted to stop. Write that down before you write a task.

Three numbers, decided before the playbook exists

Every patch run is described by three numbers, and none of them come from Ansible.

How many hosts can the service lose simultaneously? This is a capacity question. If a six-node service needs four nodes to carry peak load, the answer is two, and no serial value above 2 is defensible regardless of how long the run takes.

How long does one host take? Package download, install, reboot, reconnect, validate. On a real fleet this is rarely under four minutes and often ten. Multiply by the number of batches and compare against the change window you actually have.

How many failures make this a failed run rather than a run with failures? A fleet of 300 will always have a host with a full /boot, a host with a broken repository, a host that is already down. Deciding that in advance is the difference between a judgement and a panic.

The canary is a host, not a percentage

serial: [1, 5, 25%] reads as a ramp. For a patch run, the leading 1 is doing something specific and different from the rest of the list: it is the only batch whose purpose is to discover that the change is fundamentally wrong.

Service impact possiblethe shape of a fleet patch play— Patches and potentially reboots hosts. serial makes the damage proportional; max_fail_percentage decides when the run stops. Neither makes the patch safe.

- name: Monthly security patching
hosts: webservers
become: true
serial:
  - 1
  - 5
  - 25%
max_fail_percentage: 0
any_errors_fatal: false
tasks:
  - name: Apply security updates
    ansible.builtin.include_role:
      name: patching

  - name: Prove the host is serving before the next batch starts
    ansible.builtin.include_tasks: validate-host.yml

max_fail_percentage: 0 is the right default for a patch run and it is worth being explicit about why. Zero means “any failure in a batch stops the play”, because a patch failure is almost never a property of one host — it is a bad package, a repository that is serving a broken update, or a service that does not survive the new version. Those generalise. The second host will fail the same way.

The canary must be the same host every time

Batching slices the play host list positionally, in inventory order. That is what makes serial: [1, ...] deterministic: the canary is the first host of the resolved play host list, and it is the same host on every run until inventory changes.

Read-only / Safefinding out which host is the canary before you run— Read-only. Prints the effective host list in the order the play will slice it. The first line is the host that carries the risk first.

ansible-inventory -i inventories/production --list --yaml --limit webservers
ansible-playbook -i inventories/production patch.yml --limit webservers --list-hosts

Two things destroy this property, and both are worth checking for in review:

order: shuffle on the play randomises the host list before slicing. The canary becomes a lottery. There are reasons to shuffle a convergence run; there is no reason to shuffle a patch run.
A dynamic inventory whose order is not stable — a cloud plugin returning hosts in API order can silently change which host is first between runs.

A canary you cannot name is not a canary. It is just the host that happened to be unlucky.

`max_fail_percentage` is per batch, and must be exceeded

This is the single most misread keyword in a patch play, and it is worth executing rather than trusting.

Read-only / Safethe boundary, executed on ansible-core 2.21.3— Ten local-connection hosts, serial: 4, one host failing an assert. Only the assert module runs; nothing on the host is touched. The only difference between the two runs is the max_fail_percentage value.

$ ansible-playbook -i inv10.ini mfp.yml -e mfp=25   # then -e mfp=24

max_fail_percentage: 25   (1 failure in a batch of 4 = 25%)
PLAY banners: 3
REACHED SECOND TASK: h01,h03,h04
REACHED SECOND TASK: h05,h06,h07,h08
REACHED SECOND TASK: h09,h10
exit=2

max_fail_percentage: 24   (1 failure in a batch of 4 = 25%)
PLAY banners: 1
NO MORE HOSTS LEFT
NO MORE HOSTS LEFT
exit=2

At exactly 25 the play ran all three batches. At 24 it stopped after the first. Upstream states the rule plainly: the percentage set “must be exceeded, not equaled”.

Three consequences that matter for a patch run:

The denominator is the batch, not the fleet. max_fail_percentage: 10 over a 300-host fleet sounds like “stop after thirty failures”. With serial: 10, one failure in a batch is 10%, which does not exceed 10, so the run continues; two failures is 20%, which does, so it stops. The fleet-wide number never enters the calculation.

Small batches make the percentage coarse. In a batch of one, the only achievable failure rates are 0% and 100%. Any max_fail_percentage below 100 stops the play on a canary failure, which is what you want, and it means the exact value is irrelevant for that batch.

A surviving run still exits non-zero. Both runs above exited 2. A play that tolerated its failures and completed every batch is still a failed run as far as your scheduler is concerned. If you gate on exit code alone you cannot distinguish “stopped at the canary” from “finished with two casualties”, and those want very different responses.

`forks` is throughput. It is never safety.

forks controls how many hosts the controller talks to in parallel. It is a controller resource setting: more forks means more SSH connections, more Python processes, more memory on the controller.

It does not restrict which hosts are changed, or when. With no serial, a play against 300 hosts and forks: 5 still changes all 300 hosts; it merely takes longer to get through them, in waves of five, with no gap in which anything is evaluated.

There is one honest interaction: forks caps parallelism inside a batch. With serial: 25 and forks: 5, the batch of 25 is worked through five at a time, so the batch takes five waves. If your batch is larger than your forks, the batch duration is set by forks, and that matters for window arithmetic. It still does not change what is patched.

Read-only / Safewhat the controller is configured to do— Read-only. Reports the effective forks value and its source, so you know whether it comes from ansible.cfg, an environment variable or the built-in default.

$ ansible-config dump --only-changed | grep -i forks

DEFAULT_FORKS(/srv/ansible/ansible.cfg) = 25

If that command prints nothing, forks is at the built-in default of 5, and a 300-host patch run is going to take considerably longer than whoever scheduled it expects.

Putting the three numbers into a shape

Worked example. A 300-host web tier, capacity requires 80% of instances at peak, one host takes seven minutes end to end including reboot, and the window is six hours.

Capacity ceiling: 20% of 300 is 60 hosts. That is the absolute maximum batch, and it assumes every host in the batch is down for the whole seven minutes, which is the honest assumption for a reboot.
A batch of 60 gives five batches after the canary phases, roughly 35 minutes of hosts-down time plus soak. Comfortably inside six hours.
But 60 is the ceiling, not the plan. A ceiling batch leaves no headroom for a host that fails to come back. Halving it to 30 costs another 35 minutes and keeps 10% of the fleet as slack.

That gives serial: [1, 5, 30] and max_fail_percentage: 0. Seventeen batches, about two hours of moving parts, and a run that stops on the first host that does not come back.

Write the shape into the inventory, not the playbook

A single patch playbook that serves several service tiers cannot carry one serial value, because the tiers have different capacity arithmetic.

# group_vars/webservers.yml
patch_serial: [1, 5, 30]
patch_max_fail: 0

# group_vars/database_replicas.yml
patch_serial: 1
patch_max_fail: 0

# patch.yml
- name: Patch a service tier
  hosts: '{{ patch_target }}'
  serial: '{{ patch_serial }}'
  max_fail_percentage: '{{ patch_max_fail }}'

The batch shape is then a property of the service, reviewed with the service, and visible to anyone reading the inventory rather than buried in a playbook nobody opens.

Under the hood

The failure-policy evaluation happens in the linear strategy, at the end of each task, and this is why the two keywords behave differently from how their names read.

After every task, the strategy compares the number of hosts that have failed or become unreachable in the current batch against the batch size. If max_fail_percentage is set and the failure share exceeds it, the play is marked to stop; the strategy prints NO MORE HOSTS LEFT and returns to the iterator, which has no further batches to hand out.

Two details fall out of that implementation.

The check is after the task, not during it. All hosts in the batch run the task. That is why a serial: 5 batch with one bad package leaves five hosts having attempted the change, not one.

Unreachable counts. A host that drops off the network mid-patch — exactly what a reboot looks like if the reboot happens outside the reboot module — is counted alongside outright task failures for the purposes of the percentage. This is usually what you want during patching and it surprises people who expected only failed to count.

The NO MORE HOSTS LEFT banner printed twice in the executed run above. That is not a duplicate: the strategy emits it once for the batch it abandoned and once as the iterator finds no remaining work.

What happens if the canary host is already broken before the run starts?

The play stops on the first batch and the fleet goes unpatched — which looks identical to the play stopping because the patch is bad.

This is the commonest false alarm in canary-first patching, and it costs an hour of an engineer’s night every time. The host with a full /boot partition, the host whose repository mirror is unreachable, the host that was decommissioned three weeks ago and never removed from inventory: all of them fail the canary batch and all of them mean nothing about the patch.

The defence is a separate, unbatched pre-flight play that runs against the whole fleet before the patch play and reports every host that would fail for reasons unrelated to the change. Free disk on /boot and /var, repository reachability, no held or broken packages, the host actually responding. It is read-only, it takes a minute across a fleet, and it converts “the canary failed” from a mystery into a list you already have.

The rule that follows: the canary’s job is to tell you something about the patch. Anything the canary can tell you about the host should have been asked earlier, of every host at once.

Knowledge check

Why this matters

The named incident for this part is forty hosts rebooting together at 02:00 because a patch play had no serial and an unconditional reboot at the end. Every module call in that play was correct. Every task reported success. The recap was green.

What was missing was a decision — one that takes ten minutes, happens before any YAML is written, and produces three numbers: how many hosts the service can lose, how long one host takes, and how many failures mean stop.

Those numbers are not Ansible concepts. They come from the service, the capacity model and the change window, and Ansible has no way to infer any of them. What it offers is serial and max_fail_percentage to encode them, and a default of “all hosts, no tolerance policy” for everyone who does not.

The rest of this part is about the modules — apt, dnf, reboot, the validation that proves a host came back. None of it protects a fleet that was patched all at once, because by the time the modules run, the blast radius has already been decided.