Stopping a bad change at host twelve · Ansible

Everything in this part so far has been about a failure that already happened. This lesson is about the four keywords that decide how big it is allowed to get.

The 300/30/50 scenario is a failure of scope control before it is anything else. A play with no serial, no failure threshold and no abort criteria hits all 300 hosts before anyone can react, because Ansible works through the fleet at forks hosts at a time and does not pause to ask. The same play with a canary batch fails at host 12, someone reads the error, and the other 288 never run.

The four controls, and the question each answers

Keyword	Question it answers	Scope of its effect
`serial`	How many hosts change at once?	Defines the batch. Everything else is relative to it
`max_fail_percentage`	How much failure within a batch is tolerable?	Aborts the play after the current batch
`any_errors_fatal`	Is one failure enough to stop everything?	Aborts the play and the playbook
`force_handlers`	Do notified handlers still run when a play aborts?	Changes what state the aborted hosts are left in

They compose, and the composition is the point. serial alone limits concurrency but not total reach — without a threshold, a play with serial: 20 and a failure in every batch still visits all 300 hosts, twenty at a time, failing all the way.

`serial` is the unit of damage

serial decides the largest number of hosts that can be in a broken state before anything can stop the run, because thresholds are evaluated between batches.

That gives a rule of thumb worth stating directly: your first batch is your maximum acceptable loss. If losing twelve hosts simultaneously is survivable and losing thirty is not, the first batch is twelve, whatever the rest of the ramp looks like.

Service impact possiblea canary ramp with a bounded first failure— serial accepts a list: each entry is one batch size, and the last entry repeats for the remainder. One host, then eleven, then percentages. A real change play against production hosts — at least SERVICE-IMPACT.

- name: Application rollout
hosts: appservers
become: true
serial:
  - 1        # one host. If this fails, the damage is one host
  - 11       # a dozen total. Enough to see a pattern
  - '10%'    # then ramp
  - '25%'
max_fail_percentage: 0
tasks:
  - name: Deploy the release
    ansible.builtin.include_role:
      name: app_release

The first two entries are the whole safety mechanism. Everything after them is throughput.

`max_fail_percentage` must be exceeded

This is the semantic that catches people, and it is worth stating with its arithmetic.

The play aborts when the percentage of failed hosts in the batch is greater than the configured value. Equal is not greater.

Batch size	`max_fail_percentage`	Failures	Percentage	Aborts?
20	`10`	2	10%	No — equal, not exceeded
20	`10`	3	15%	Yes
20	`0`	1	5%	Yes
3	`30`	1	33.3%	Yes
4	`30`	1	25%	No

Two consequences follow.

max_fail_percentage: 0 means “any failure stops the play”, because any non-zero percentage exceeds zero. It does not mean “tolerate no failures” in some special way; it is the ordinary rule applied to zero. This is the most useful setting for a change where consistency matters, and it is more precise than any_errors_fatal because it stops the play without stopping the whole playbook.

Small batches make the percentage coarse. With serial: 3, the available percentages are 0, 33, 67 and 100. A threshold of 30 and a threshold of 10 behave identically. During the canary phase of a ramp, you are effectively choosing between “stop on any failure” and “do not stop”, so choose deliberately rather than tuning a number that cannot express what you meant.

`any_errors_fatal` stops the playbook, not just the play

Covered in full in the failure-tolerance part; recalled here because its reach is the part that produces incidents.

Verified on 2.21.3: with any_errors_fatal: true, a failure in batch two of five means batches three, four and five never run and produce no recap lines at all, and a later play in the same playbook does not run either.

Read-only / Safeany_errors_fatal with serial 2 over six hosts, h03 fails — executed on ansible-core 2.21.3— Local connections and assert tasks, so nothing is modified. h04 completes the failing task alongside h03, then stops. h05 and h06 are absent from the recap entirely, because they never became a batch.

$ ansible-playbook -i inv6.ini aef.yml

TASK [Health check] ************************************************************
fatal: [h03]: FAILED! => {"msg": "health check failed"}
ok: [h04] => { "msg": "HEALTHY h04" }

PLAY RECAP *********************************************************************
h01   : ok=3  changed=0  unreachable=0  failed=0
h02   : ok=3  changed=0  unreachable=0  failed=0
h03   : ok=1  changed=0  unreachable=0  failed=1
h04   : ok=2  changed=0  unreachable=0  failed=0

The operational consequence is the one to carry into a design review: a patching playbook whose first play silences monitoring and whose last play re-enables it will, on a single failure, leave the entire fleet unmonitored. Anything that must happen regardless of outcome belongs in a block/always, never in a later play.

Choosing between the two thresholds

max_fail_percentage: 0 and any_errors_fatal: true both stop on the first failure. They differ in reach:

	`max_fail_percentage: 0`	`any_errors_fatal: true`
Stops the current play	Yes	Yes
Stops later plays in the playbook	No	Yes
Recoverable by a `rescue` block	No	Yes
Reads clearly in review	Yes	Yes

For most rolling changes, max_fail_percentage: 0 is the better default precisely because it lets your cleanup plays run. Reach for any_errors_fatal when a partial application is genuinely worse than no application — schema migrations, security posture changes, anything where the hosts talk to each other — and then make sure nothing downstream of it restores a safety property.

Neither of them sees unreachable hosts

Both keywords react to task failures. A host that cannot be connected to never returns a task result, so neither threshold counts it.

Verified on 2.21.3 with any_errors_fatal: true, serial: 3 and one documentation-range address in batch one: the batch containing the unreachable host was cut, and then later batches ran to completion and the subsequent play ran too, exiting 4. Compare the failed-host run above, where batch two never started.

`force_handlers`, and the hosts left drained

When a play aborts, notified handlers that have not yet run are discarded. That is usually right — you do not want a service restart firing on a fleet you just decided to stop changing.

It is wrong in one specific shape, and it is the shape rolling deployments have. If the play drains a host from the load balancer, changes it, and returns it to service via a handler, then aborting between those steps leaves the host drained. The abort protected the fleet from the change and left the canary out of rotation.

Service impact possibleforce_handlers so an abort does not strand a drained host— With force_handlers: true, handlers notified before the abort still run on hosts that already reached them. A real rollout play; the handler restarts a service.

- name: Application rollout
hosts: appservers
become: true
serial: [1, 11, '25%']
max_fail_percentage: 0
force_handlers: true
tasks:
  - name: Deploy configuration
    ansible.builtin.template:
      src: app.conf.j2
      dest: /etc/app/app.conf
      mode: '0640'
      validate: '/usr/sbin/appd -t -c %s'
    notify: Restart app

handlers:
  - name: Restart app
    ansible.builtin.systemd_service:
      name: appd
      state: restarted

force_handlers is not a general safety improvement — it means handlers run on a play you decided to abort, which is sometimes precisely what you did not want. The right question is the same one as everywhere else in this part: if this play stops here, what state is each host left in, and does a handler need to run to make that state coherent?

The general answer for cleanup work is block/always, which runs regardless of outcome and is not subject to the notification mechanism at all. force_handlers is for the case where the coherence-restoring action is genuinely a handler and you cannot restructure it.

Under the hood

Why is max_fail_percentage evaluated between batches rather than continuously?

The linear strategy runs one task across the whole batch, collects all results, then advances. Failure accounting therefore has a natural evaluation point: the end of a task across a batch. The percentage is computed from the number of hosts that have failed against the number that started the batch, and compared with a strictly-greater-than test — which is where “must be exceeded” comes from, literally, as an operator in the source.

That structure explains the per-batch scoping too. The failed-host count is reset when a new batch begins, because a batch is a fresh play iteration over a new slice of hosts. There is nowhere for a run-level total to accumulate, which is why no keyword expresses “stop after N failures across the whole run”.

any_errors_fatal hooks a different point: it sets a fatal condition that ends the play iterator. Since the iterator is what walks the batches, ending it ends the batching, and hosts in later batches are never enumerated for execution — which is why they produce no recap line. The recap reports hosts that participated, and they did not. The playbook executor then checks between plays whether the previous play ended fatally, which is the mechanism behind the stop-the-whole-playbook behaviour.

Unreachability is recorded on the connection path, before any task result exists, and removes the host from the active list for the run. Neither threshold is consulted. The asymmetry is a consequence of where the two conditions are recorded rather than a deliberate policy about connectivity — which is exactly why it is worth knowing, since nothing in either keyword’s name would lead you to predict it.

What happens if I set serial, max_fail_percentage and any_errors_fatal all at once?

any_errors_fatal wins, because it triggers on the first failure and nothing can be stricter than that. Whatever percentage you set, the play is already over before the threshold could be exceeded.

Setting both is not an error and is not useful. It reads as a nuanced policy and expresses only the stricter half.

Where the combination is meaningful is at different scopes: a play with max_fail_percentage: 10 for its ordinary work, containing one block with any_errors_fatal: true around the critical section. That says something real — tolerate a tenth of the batch failing on routine tasks, treat any failure in the migration as fatal — and it is the shape to prefer when a play genuinely has both kinds of work in it.

The design, assembled

For the 300-host scenario, the play that would have limited it:

Service impact possiblescope control applied to the signature scenario— A pre-flight play that connects and refuses to continue if the fleet is not present, then a rollout whose first failure costs one host. A production change play.

- name: Pre-flight — the fleet must actually be there
hosts: appservers
gather_facts: false
tasks:
  - name: Prove every target is reachable
    ansible.builtin.ping:

  - name: Refuse to start if the reachable count is short
    ansible.builtin.assert:
      that: ansible_play_hosts | length == ansible_play_hosts_all | length
      fail_msg: >-
        {{ ansible_play_hosts_all | difference(ansible_play_hosts) | join(', ') }}
        did not answer. Resolve or re-scope before changing anything.
    run_once: true

- name: Rollout
hosts: appservers
become: true
serial: [1, 11, '10%', '25%']
max_fail_percentage: 0
force_handlers: true
tasks:
  - name: Apply the change
    ansible.builtin.include_role:
      name: patch_and_configure

Under that play, the same underlying problems produce a very different night: the 50 unreachable hosts stop the run before it starts, and if they had appeared later, the first task failure ends the rollout inside the canary batch.

Knowledge check

Why this matters

Every lesson before this one deals with the aftermath. This one is the part you control.

The difference between the recap that says twelve hosts and the recap that says three hundred is four keywords, chosen at design time, by someone who asked how big the failure was allowed to get. It is not a difference in skill at writing tasks, and it is not visible in a play that succeeds — a play with no scope control looks identical to a play with good scope control on every night that nothing goes wrong.

That is why it has to be a review standard rather than a judgement call. “What is the first batch size, and what is the abort criterion” are two questions any reviewer can ask about any change play, in ten seconds, without understanding the change. Asking them is the cheapest thing in this course, and the 300-host scenario is what not asking them looks like.

The four controls, and the question each answers

serial is the unit of damage

max_fail_percentage must be exceeded

any_errors_fatal stops the playbook, not just the play