max_fail_percentage, precisely · Ansible

By default, a play keeps going as long as some host is still succeeding. Upstream puts it plainly: “Ansible continues to execute tasks as long as there are hosts that have not yet failed.”

On a rolling deploy that default is dangerous. The change fails on host one, and the play does not care — it proceeds to host two, fails again, proceeds to host three. The mechanism that was supposed to protect you by staging the change happily stages a fleet-wide breakage in comfortable instalments.

max_fail_percentage is the threshold that stops it. It has two rules that are not obvious from reading it, and both of them can be demonstrated in a few seconds.

Rule one: it is evaluated per batch

Upstream states this: “The max_fail_percentage setting applies to each batch when you use it with serial.”

Worth proving rather than accepting, because “per batch” and “per play” produce identical results in the common case of equal batches with evenly spread failures, and the difference only shows up when they are uneven — which is precisely the canary shape you will actually deploy.

The discriminating experiment uses unequal batches and a single failure:

Read-only / Safethe discriminator— Six local connections, no state changed. Batches of 4 then 2; a single host fails, and it is in the second batch. Batch-2 failure rate is 50%; play-wide failure rate is 16.7%. A threshold of 30 sits between them.

- name: Rollout
hosts: batchlab
gather_facts: false
serial:
  - 4
  - 2
max_fail_percentage: 30
tasks:
  - name: Deploy
    ansible.builtin.debug:
      msg: 'DEPLOYED {{ inventory_hostname }}'

  - name: Health check
    ansible.builtin.fail:
      msg: health check failed
    when: inventory_hostname in ['h05']

  - name: Return to service
    ansible.builtin.debug:
      msg: 'INSERVICE {{ inventory_hostname }}'

If the threshold is compared against the batch, one failure out of two is 50%, which exceeds 30, and the run aborts. If it is compared against the play, one failure out of six is 16.7%, which does not exceed 30, and h06 finishes normally.

Read-only / Safeexecuted on ansible-core 2.21.3— h06 never reaches Return to service. The threshold was measured against the batch.

$ ansible-playbook -i inv6.ini discriminator.yml

max_fail_percentage: 30
deployed:  h01 h02 h03 h04 h05 h06
FAILED:    h05
returned:  h01 h02 h03 h04          <-- h06 missing
exit=2

max_fail_percentage: 60   (control: 50% does not exceed 60)
deployed:  h01 h02 h03 h04 h05 h06
FAILED:    h05
returned:  h01 h02 h03 h04 h06      <-- h06 present
exit=2

Per batch, confirmed. The control run with a threshold of 60 is the same playbook with the same single failure, and h06 completes — which establishes that the abort in the first run was caused by the threshold and not by anything else about h05.

Without serial, the whole play is a single batch, and the threshold applies to it. Verified: six hosts, no serial, one failure (16.7%) and max_fail_percentage: 30 runs to completion; three failures (50%) and the same threshold aborts.

Rule two: exceeded, not equalled

The comparison is strictly greater-than. Upstream is explicit, and gives the example in the same breath:

The percentage set must be exceeded, not equaled. For example, if serial were set to 4 and you wanted the task to abort the play when 2 of the systems failed, set the max_fail_percentage at 49 rather than 50.

Reproduced here exactly:

Read-only / Safethe upstream example, executed on 2.21.3— serial: 4, two of the four hosts in the first batch fail. That is exactly 50%.

$ ansible-playbook -i inv6.ini boundary.yml

serial: 4, h01 and h02 fail  ->  2/4 = 50.0%

max_fail_percentage: 49
  deployed:  h01 h02 h03 h04
  FAILED:    h01 h02
  returned:  (none)
  exit=2                        <-- aborted, later batch never ran

max_fail_percentage: 50
  deployed:  h01 h02 h03 h04 h05 h06
  FAILED:    h01 h02
  returned:  h03 h04 h05 h06
  exit=2                        <-- continued, whole fleet processed

50 does not stop a 50% failure rate. This is the single most common misconfiguration of this keyword, and it fails in the most expensive direction: the person who wrote max_fail_percentage: 50 believed they had asked for “stop if half the batch breaks”, and what they actually asked for is “stop if more than half breaks”.

The boundary is equally sharp at other ratios:

Read-only / Safethe 33-versus-34 boundary, executed on 2.21.3— serial: 3 over six hosts, one host fails in batch one. One of three is 33.333...%.

$ ansible-playbook -i inv6.ini boundary33.yml

serial: 3, h01 fails  ->  1/3 = 33.33%

max_fail_percentage: 33
  TASK [Deploy]              ok: h01 h02 h03
  TASK [Health check]        fatal: h01   skipping: h02 h03
  NO MORE HOSTS LEFT
  (Return to service never ran; h04 h05 h06 never appeared)
  exit=2

max_fail_percentage: 34
  TASK [Deploy]              ok: h01 h02 h03
  TASK [Health check]        fatal: h01   skipping: h02 h03
  TASK [Return to service]   ok: h02 h03
  PLAY [Rollout]  (batch 2)
  TASK [Deploy]              ok: h04 h05 h06
  TASK [Return to service]   ok: h04 h05 h06
  exit=2

Choose the threshold from the batch size, not from a feeling

max_fail_percentage reads like a tolerance and behaves like an arithmetic comparison against a specific batch size. Compute it.

For a batch of n hosts, to abort once k of them have failed, set the threshold to the largest integer strictly below 100 × k / n:

Batch size	Abort when this many fail	Failure rate	Set threshold to
4	2	50%	`49`
3	1	33.33%	`33`
5	1	20%	`19`
10	2	20%	`19`
10	1	10%	`9`
any	1	any	`0`

The last row is the one to remember: max_fail_percentage: 0 aborts on the first failure in any batch, because any non-zero failure rate exceeds zero. Verified on 2.21.3 — serial: 3 with one failure and a threshold of 0 aborts, and the same play with no failures completes and exits 0.

Under a canary shape like serial: [1, 5, 25%] the batch size changes between batches, so a single percentage means different absolute failure counts at different points in the rollout. That is usually fine and occasionally not — with a first batch of one host, every threshold below 100 means the same thing there.

What an abort actually leaves behind

This is the part that is not in the documentation and that matters more operationally than either rule above.

When the threshold is exceeded, the play does not finish the current batch. It stops at the end of the failing task, and the healthy hosts in that batch do not run the remaining tasks.

Look again at the 49 run:

max_fail_percentage: 49
  deployed:  h01 h02 h03 h04
  FAILED:    h01 h02
  returned:  (none)

h03 and h04 passed their health check. They are fine. They were deployed to, and they never reached “Return to service”. In a real rolling play, “Return to service” is the task that puts the host back in the load balancer — so those two hosts are drained, healthy, running the new version, and receiving no traffic.

The recap will not tell you this. It reports h03 and h04 as failed=0, because nothing failed on them:

Read-only / Safethe recap of an aborted batch, executed on 2.21.3— serial: 3 with max_fail_percentage: 33 and one failure. h02 and h03 report failed=0 and are nonetheless unfinished. h04, h05 and h06 are absent from the recap entirely.

$ ansible-playbook -i inv6.ini abort.yml

PLAY RECAP *********************************************************************
h01                        : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
h02                        : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
h03                        : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

Three things to read out of that recap, none of which are stated in it:

h02 and h03 are unfinished, not successful. failed=0 means no task failed on them. It does not mean the play completed for them. Compare ok=1 here against ok=2 in the run that completed.
h04, h05 and h06 are missing entirely. A host that was never reached gets no recap line at all. The absence of a line is the only evidence that later batches did not run.
The exit code is 2, the same as any other task failure. Nothing in the exit code distinguishes “one host failed and the rest completed” from “the rollout aborted with a batch half-finished”.

Unreachable hosts do not count

Stated here because it changes what this keyword is for, and covered in full in the unreachable-versus-failed lesson.

max_fail_percentage responds to failed hosts. It does not respond to unreachable ones.

Read-only / Safeone unreachable host, the strictest possible threshold— Six hosts, serial: 3. The first host is 192.0.2.11 — a documentation-range address, unreachable by design. The other five are local connections. max_fail_percentage is 0.

$ ansible-playbook -i inv-unreach.ini rollout.yml

TASK [Touch the host] **********************************************************
ok: [h02]
ok: [h03]
fatal: [u01]: UNREACHABLE! => {"changed": false, "msg": "Task failed: Failed to
connect to the host via ssh: ssh: connect to host 192.0.2.11 port 22: Connection
timed out", "unreachable": true}

TASK [Return to service] *******************************************************
ok: [h02]
ok: [h03]

... all remaining batches ran to completion ...

PLAY RECAP *********************************************************************
h02 : ok=2 changed=0 unreachable=0 failed=0
...
u01 : ok=0 changed=0 unreachable=1 failed=0
exit=4

One host out of three in the batch was unreachable — 33.3%, against a threshold of zero — and the play ran the entire fleet.

The operational reading: max_fail_percentage is not a circuit breaker for a fleet that is falling off the network. A rollout during a partial network outage will proceed through every batch, touching whatever it can reach, no matter how strict the threshold. If you need to stop on connectivity loss, you need a different mechanism.

Under the hood

The check runs at the end of each task, in the linear strategy, and it compares the count of hosts that have failed during the current batch against the number of hosts the batch started with.

Three details explain the observed behaviour:

The comparison is strictly greater-than on a float percentage. With two failures out of four, the computed value is 50.0, and 50.0 > 50 is false. This is why the documented remedy is to set 49. It is not a rounding artefact — there is no rounding — it is the comparison operator.

The denominator is the batch, not the play. The strategy holds a per-batch record of failed hosts and the batch size at entry, which is why the [4, 2] discriminator aborts: the second batch entered with two hosts, one failed, and 50.0 > 30 is true.

The abort is raised at the task boundary, not the batch boundary. The check runs after the failing task has completed across the batch, and what follows is observable in the output: the run prints NO MORE HOSTS LEFT and the play ends without dispatching another task. The surviving hosts in the batch carry the result of the failing task — they were skipping or ok for it — but no subsequent task runs for them. That is the mechanism behind the healthy-but-unfinished hosts, and it is why they appear with failed=0 and a lower ok= count rather than with any distinguishing marker.

Unreachable hosts are tracked in a separate structure from failed hosts, and the percentage calculation reads only the failed one. This is consistent with any_errors_fatal treating the two differently, and with the recap keeping unreachable= and failed= as separate columns throughout.

What happens if I set max_fail_percentage without serial?

It still works, but it becomes a fleet-wide tolerance rather than a rollout brake, and it can no longer stop anything early.

Without serial the play is one batch, so the threshold is evaluated against the whole fleet — and every host runs each task before the threshold is checked. Verified on 2.21.3: six hosts, no serial, three failures and max_fail_percentage: 30 aborts the play, but only after the failing task has run everywhere. The remaining tasks are skipped across the fleet.

That is a meaningful thing to want — “if more than 30% of the fleet fails to install the package, do not proceed to the restart” — and it is categorically not a rolling deployment safeguard, because there was never a point at which only part of the fleet had been touched.

If the intent is “stop before this reaches everything”, the keyword is necessary but not sufficient. It needs serial to have something to stop.

Knowledge check

Why this matters

Two of the rules in this lesson are documented and one is not, and it is the undocumented one that produces the incidents.

The exceeded-not-equalled rule is in the upstream text, and a reader who gets it wrong ends up with a rollout that does not stop. That is bad, and it is at least legible afterwards: the log shows every batch running, and the arithmetic explains itself once somebody looks.

The undocumented behaviour is what an abort leaves behind. Healthy hosts in the failing batch stop where they stood, and the recap reports them with failed=0 — indistinguishable at a glance from hosts that completed. If the task they did not reach was the one returning them to the load balancer, you have lost capacity, and the run that lost it reported the loss nowhere.

Set the threshold from arithmetic, not from a round number. Then write the play so that stopping cannot strand a host — which is the problem the rolling-deployment part exists to solve.