any_errors_fatal and what fatal means · Ansible

max_fail_percentage expresses a tolerance: some failures are acceptable, past a threshold they are not. any_errors_fatal expresses the opposite position — one failure anywhere means the change is wrong, so stop touching things.

Upstream describes it precisely, and the precision is the whole lesson:

If you set any_errors_fatal and a task returns an error, Ansible finishes the fatal task on all hosts in the current batch and then stops executing the play on all hosts.

Two clauses. The first is a courtesy; the second is much bigger than it sounds.

Finishing the fatal task on all hosts in the batch

The failing task is not abandoned partway. Every host in the batch runs it, and the results all appear, before anything stops.

Read-only / Safesix hosts, serial: 2, h03 fails its assertion — executed on ansible-core 2.21.3— Local connections and an assert task, so nothing is changed. h04 is in the same batch as h03 and completes the health check normally, after h03 has already been declared fatal.

$ ansible-playbook -i inv6.ini aef.yml

PLAY [Rollout] *****************************************************************

TASK [Deploy] ******************************************************************
ok: [h01] => { "msg": "DEPLOYED h01" }
ok: [h02] => { "msg": "DEPLOYED h02" }

TASK [Health check] ************************************************************
ok: [h01] => { "msg": "HEALTHY h01" }
ok: [h02] => { "msg": "HEALTHY h02" }

TASK [Return to service] *******************************************************
ok: [h01] => { "msg": "INSERVICE h01" }
ok: [h02] => { "msg": "INSERVICE h02" }

PLAY [Rollout] *****************************************************************

TASK [Deploy] ******************************************************************
ok: [h03] => { "msg": "DEPLOYED h03" }
ok: [h04] => { "msg": "DEPLOYED h04" }

TASK [Health check] ************************************************************
fatal: [h03]: FAILED! => {"assertion": "inventory_hostname not in ['h03']",
                        "changed": false, "evaluated_to": false,
                        "msg": "health check failed"}
ok: [h04] => { "changed": false, "msg": "HEALTHY h04" }

PLAY RECAP *********************************************************************
h01                        : ok=3    changed=0    unreachable=0    failed=0
h02                        : ok=3    changed=0    unreachable=0    failed=0
h03                        : ok=1    changed=0    unreachable=0    failed=1
h04                        : ok=2    changed=0    unreachable=0    failed=0

h04 ran the health check and passed it, after h03 was already fatal. That is the first clause working: the task completes across the batch.

h04 then did not run “Return to service”. That is the second clause, and it produces the same stranded-host problem the previous lesson described for max_fail_percentage: a healthy host, drained, never returned.

Stopping the play on all hosts

Read the recap again. h05 and h06 are not in it.

They were in batch three. Batch three never ran. any_errors_fatal did not merely end the batch that contained the failure — it ended the play, and every host that had not yet been reached was never reached.

That is the intended behaviour and it is the reason to choose this keyword. But it extends further than most people expect.

A later play in the same playbook does not run

This is the behaviour that produces the incident, and it is verified rather than inferred.

The playbook above has a second play:

- name: Re-enable monitoring
  hosts: batchlab
  gather_facts: false
  tasks:
    - name: Re-enable
      ansible.builtin.debug:
        msg: 'MONITORING-ON {{ inventory_hostname }}'

Executed on 2.21.3 with any_errors_fatal: true on the first play and a failure in batch two: the PLAY [Re-enable monitoring] banner never appears. No host runs it. The playbook ends at the point of failure.

Now imagine the realistic version. A patching playbook whose first play silences monitoring, patches and reboots the fleet in batches, and whose final play turns monitoring back on. A single host fails to come back in batch two. any_errors_fatal stops the run — correctly, that is what it is for — and the monitoring re-enable play never executes.

The fleet is now unmonitored. Not the failed host: the whole fleet, including the hosts that were patched successfully and the hosts that were never touched. And the alert that would have told you is the one you just disabled.

Anything that must happen regardless of outcome does not belong in a later play. It belongs in a block/always, or in a separate playbook invocation that runs whatever the first one returned.

Scoping it to a block

any_errors_fatal is settable at play level and at block level. Block level is usually the better instrument, because most playbooks contain one genuinely critical section and a lot of ordinary work around it.

Service impact possiblefatal only where it needs to be— The schema migration is all-or-nothing across the batch. The tasks around it are not. Verified structurally on 2.21.3: a failure inside the block stops the play exactly as play-level any_errors_fatal does.

- name: Apply the release
hosts: appservers
become: true
serial: 2
tasks:
  - name: Stage the release artefact
    ansible.builtin.copy:
      src: 'app-{{ release_version }}.tar.gz'
      dest: /var/tmp/app-release.tar.gz
      mode: '0640'

  - name: Schema-affecting migration
    any_errors_fatal: true
    block:
      - name: Verify the migration is reversible before applying it
        ansible.builtin.command: /usr/local/bin/app-migrate --dry-run
        changed_when: false

      - name: Apply the migration
        ansible.builtin.command: /usr/local/bin/app-migrate --apply
        register: migration
        changed_when: "'applied' in migration.stdout"

  - name: Record the deployed version
    ansible.builtin.copy:
      content: '{{ release_version }}'
      dest: /etc/app/deployed-version
      mode: '0644'

Verified on 2.21.3 with the block form: serial: 2 over six hosts, a failure on h03 inside the block. h01 and h02 completed the play, h04 did not run the task after the block, and h05 and h06 never ran at all — identical to the play-level behaviour, scoped to the block.

Which failures deserve any_errors_fatal

The keyword is right when a partial application is worse than no application.

Schema migrations and data format changes. Half the fleet on the new schema and half on the old is frequently a broken system, and every additional host you migrate makes the reversal harder.
Security changes. A key rotation or a firewall change that succeeds on 60% of the fleet leaves an inconsistent security posture that is worse than either end state, and you want to know immediately.
Anything where the hosts talk to each other. Cluster membership, quorum configuration, replication topology.

It is wrong when partial progress has value: package updates, convergence runs, log rotation configuration, monitoring agents. There, max_fail_percentage lets the fleet get better on the hosts where it can while still stopping if the change is broadly wrong.

The distinguishing question is the same one as for handlers: if this change lands on 40% of the fleet and then stops, is that state better or worse than where you started? Better means tolerate; worse means fatal.

It does not make unreachable hosts fatal

This is the finding that surprised me most while preparing this lesson, and it is worth stating as an exception because the keyword name promises otherwise.

The upstream sentence says “if a task returns an error”. A host that cannot be connected to does not return an error from a task — it never ran one. Ansible marks it UNREACHABLE and, per the documentation, “removes it from the list of active hosts for the run”. That is a different path, and any_errors_fatal does not intercept it.

Read-only / Safeany_errors_fatal with an unreachable host — executed on 2.21.3— serial: 3 over six hosts. u01 is 192.0.2.11, a documentation-range address that is unreachable by design. Compare with the failed-host run: there, batch two never started.

$ ansible-playbook -i inv-unreach.ini rollout.yml

PLAY [Rollout] *****************************************************************
TASK [Touch the host] **********************************************************
ok: [h02]
ok: [h03]
fatal: [u01]: UNREACHABLE! => {"changed": false, "msg": "Task failed: Failed to
connect to the host via ssh: ssh: connect to host 192.0.2.11 port 22: Connection
timed out", "unreachable": true}

PLAY [Rollout] *****************************************************************
TASK [Touch the host] **********************************************************
ok: [h04]
ok: [h05]
ok: [h06]
TASK [Return to service] *******************************************************
ok: [h04] => { "msg": "INSERVICE h04" }
ok: [h05] => { "msg": "INSERVICE h05" }
ok: [h06] => { "msg": "INSERVICE h06" }

PLAY [Second play] *************************************************************
TASK [Monitoring] **************************************************************
ok: [h04] => { "msg": "MONITORING h04" }
ok: [h05] => { "msg": "MONITORING h05" }
ok: [h06] => { "msg": "MONITORING h06" }

PLAY RECAP *********************************************************************
h02 : ok=1 changed=0 unreachable=0 failed=0
h03 : ok=1 changed=0 unreachable=0 failed=0
h04 : ok=3 changed=0 unreachable=0 failed=0
h05 : ok=3 changed=0 unreachable=0 failed=0
h06 : ok=3 changed=0 unreachable=0 failed=0
u01 : ok=0 changed=0 unreachable=1 failed=0
exit=4

Compare the two runs directly. Same keyword, same serial, same playbook shape, failure in batch one:

	Host failed a task	Host was unreachable
Rest of the failing batch	stops after the fatal task	stops after the fatal task
Later batches	never run	run to completion
Later play in the playbook	never runs	runs, for the later-batch hosts
Exit code	`2`	`4`

The batch containing the unreachable host was cut — h02 and h03 never returned to service, and they are absent from the second play, so they were dropped from the run. But the rollout itself continued.

any_errors_fatal is not a defence against a fleet that is disappearing. Neither is max_fail_percentage. If the risk you are managing is connectivity rather than correctness, the mechanism is a reachability precondition in an unbatched play before the rollout starts — a subject the next lesson takes up in full.

Under the hood

The play-level and block-level forms are the same mechanism at different scopes. When a task result is a failure and any_errors_fatal is in effect for that task, the strategy records the failure and, once results for the whole batch have been collected, ends the play iteration rather than advancing to the next task.

“Ends the play iteration” is doing the work in that sentence. The iterator is what walks the batches, so ending it ends the batching too — no further slice is taken, and hosts in later batches are never enumerated for execution. They were always in the play host list; they simply never became a batch. That is why they produce no recap line: the recap reports hosts that participated, and they did not.

The playbook executor checks, between plays, whether the previous play ended in a fatal condition, and stops the playbook rather than starting the next play. This is the part with the operational teeth, and it is also the part that distinguishes any_errors_fatal from every other failure keyword — max_fail_percentage aborting a play does not stop the next play from running.

Unreachability is recorded on a separate path, and the documentation describes the effect: the host is removed from the list of active hosts for the run. That removal is not a task failure, so the fatal condition above is never raised, the playbook executor sees no reason to stop, and the next batch is taken normally. The asymmetry is a consequence of where the two conditions are recorded rather than a deliberate policy about connectivity, which is exactly why it is worth knowing: nothing in the keyword name or its documentation would lead you to predict it.

What happens if I combine any_errors_fatal with max_fail_percentage?

any_errors_fatal wins, because it triggers on the first failure and nothing can be stricter than that.

Setting both on the same play is not an error, and it is not useful: whatever threshold you set, any_errors_fatal will already have stopped the play before the percentage could be exceeded. The combination reads as though it expresses a nuanced policy and expresses only the stricter half of it.

Where the combination is meaningful is at different scopes — a play with max_fail_percentage: 19 for its ordinary work, containing one block with any_errors_fatal: true for the critical section. That says something real: tolerate up to a fifth of the batch failing on the routine tasks, but treat any failure in the migration as fatal. The block-scoped example earlier in this lesson is exactly that shape.

Knowledge check

Why this matters

any_errors_fatal is the easiest failure policy to reason about and the easiest to deploy carelessly, because its stopping behaviour is the part people think about and its reach is the part that bites.

Stopping a rollout at the first sign of trouble is almost always the right instinct for a change where consistency matters. But the keyword does not stop the rollout; it stops the playbook. Every cleanup step, every re-enable, every notification that lives in a later play is cancelled by the same mechanism that protected you.

That is a defensible design — Ansible cannot know which of your later plays are cleanup and which are more change — but it means the responsibility is yours. Before you set any_errors_fatal, read the rest of the playbook and ask what is downstream of it. If anything down there restores a safety property that an earlier play removed, it is in the wrong place, and the failure that proves it will be the one you were least able to afford.