Choosing a failure policy · Ansible

You have three failure policies available, and one of them is what you get by writing nothing.

Policy	Stops when	Scope of the stop
(none — the default)	every host in a batch has failed	the play
`max_fail_percentage: N`	more than N% of a batch has failed	the play
`any_errors_fatal: true`	any host fails	the playbook

The rest of this lesson is about choosing between them, but the row that deserves attention first is the one nobody chooses deliberately.

The default is a policy, and it is a bad one

Writing no failure keyword does not mean “no policy”. It means: keep going as long as at least one host is still working.

Verified on ansible-core 2.21.3 — six hosts, serial: 2, no failure keyword, and a health check that fails on both hosts of the first batch:

Read-only / Safewhat the default does when a whole batch fails— Local connections and an assert task; nothing is changed. Both hosts of batch one fail, so no host remains active and the play ends. h03 through h06 never appear.

$ ansible-playbook -i inv6.ini nopolicy.yml

PLAY [batchlab] ****************************************************************

TASK [Health check] ************************************************************
fatal: [h01]: FAILED! => {"assertion": "inventory_hostname not in ['h01','h02']"}
fatal: [h02]: FAILED! => {"assertion": "inventory_hostname not in ['h01','h02']"}

PLAY RECAP *********************************************************************
h01                        : ok=0    changed=0    unreachable=0    failed=1
h02                        : ok=0    changed=0    unreachable=0    failed=1

The play stopped — but only because every host in the batch failed. Change the test so that just one of the two fails, and the rollout proceeds through all six hosts, breaking each batch in turn.

That is the default’s actual shape: it stops only when there is nothing left to break. As a safety mechanism it is the last one to fire, and by then it has done all the damage it can.

The decision

One question separates the two real policies:

If this change lands on part of the fleet and then stops, is that state better or worse than where you started?

Better — partial progress has value. Use max_fail_percentage. A fleet where 80% of hosts got the security patch is genuinely safer than one where none did, and the 20% that failed are a work item rather than an emergency.

Worse — partial application is its own failure. Use any_errors_fatal. A schema migration applied to half the fleet is not “half done”; it is a system where neither version works reliably, and every additional host makes the reversal harder.

flowchart TD
  A["A change to roll across the fleet"] --> B{"Is a partially applied<br/>change worse than<br/>no change at all?"}
  B -->|"Yes — hosts must agree"| C["any_errors_fatal: true"]
  B -->|"No — progress has value"| D["max_fail_percentage: N"]
  C --> E{"Does anything downstream<br/>have to run regardless<br/>of the outcome?"}
  E -->|"Yes"| F["Move it into block/always<br/>— a later play will not run"]
  E -->|"No"| G["Scope it: block level if only<br/>one section is critical"]
  D --> H["Set N from the batch size:<br/>largest integer strictly<br/>below 100 x k / n"]
  B -->|"Neither — cannot tolerate<br/>hosts being unreachable"| I["Neither keyword helps.<br/>Gate reachability in an<br/>unbatched pre-flight play"]

The third branch is the one this part keeps returning to, and it is not a failure policy at all. Both keywords respond to failed hosts. Both were verified to ignore unreachable ones. If the fleet going quiet is the risk you are managing, the control has to run before the rollout does.

The canary ramp

For most production changes the batch shape and the failure policy are chosen together, and the shape is:

Service impact possiblethe default shape for a service-impacting change— One host, then five, then a quarter of the fleet at a time. The threshold is computed for the widest batch, which is the one where a percentage is least forgiving.

- name: Roll the application release across the fleet
hosts: appservers
become: true
serial:
  - 1
  - 5
  - 25%
max_fail_percentage: 0
tasks:
  - name: Deploy and verify
    ansible.builtin.include_tasks: deploy-one-host.yml

Each stage answers a different question, and that is why the shape has three stages rather than one number.

Batch one: one host. Does this change work at all?

Most broken deploys are broken everywhere — a template that renders invalid configuration, a package version that does not exist, a unit file with a typo. One host finds all of those, at the cost of one host. It is also the only batch where you can reasonably expect a human to look at the result before the run continues.

Batch two: five hosts. Does it work on more than a lucky host?

The first host might have been the one with the cached package, the already-correct file, the unusual permissions. Five hosts drawn from the same inventory order are still not a random sample, but they will surface per-host variation that one host cannot.

Batch three onward: 25% at a time. Does it work at scale?

The remaining risk is different in kind — connection pool exhaustion, a shared dependency that cannot take the reconnect storm, a licence server that rate-limits. These only appear with volume, and the ramp is what produces volume in controlled increments.

max_fail_percentage: 0 means something different in each batch

Under a ramp, the batch size changes, so a single threshold means a different absolute failure count at each stage.

With max_fail_percentage: 0, the meaning is stable — abort on the first failure anywhere — which is why it pairs well with a ramp. With any other value, work out what it means in the first batch as well as the last:

`serial` stage	Batch size (200 hosts)	`max_fail_percentage: 19` aborts at
`1`	1	1 failure (100% > 19)
`5`	5	1 failure (20% > 19)
`25%`	50	10 failures (20% > 19)

That table is defensible: tight early, proportional later. But it is a decision, and it should be one you made rather than one that emerged from a round number.

Worked selections

Rolling an application release across 200 web servers. Partial progress is survivable for a short window if the versions are compatible, and intolerable if they are not — so the answer depends on skew, which the rolling part treats in full. Where skew is survivable: serial: [1, 5, 25%] with max_fail_percentage: 0. Stop at the first failure, but keep what has already landed and roll forward or back deliberately.

Rotating the SSH host trust configuration across the fleet. A host that ends up with the new configuration while the controller still presents the old key becomes unmanageable. Partial application is worse than none. any_errors_fatal: true, serial: [1, 5, 10%], and a block/always that restores the previous file if the block fails.

Monthly OS patching across a mixed fleet. Some hosts will fail for uninteresting reasons — a full /boot, a held package, a repository timeout. Stopping the whole window because three hosts out of four hundred had a full disk is worse than proceeding. max_fail_percentage: 9 with serial: "10%", and a report of the failed hosts as the deliverable.

Applying a database schema migration to a replica set. any_errors_fatal: true, and the batch shape barely matters because the first failure stops everything. What matters is that the migration is reversible and that the reversal is written before the forward migration runs.

Deploying a monitoring agent to every host in the estate. Best-effort convergence. max_fail_percentage set high or omitted entirely, serial chosen for controller load rather than for safety, and the value of the run is the list of hosts it could not reach.

Under the hood

The three policies are checked at different points, which is what gives them their different scopes.

The default check is structural rather than a policy at all: the linear strategy advances while the batch has active hosts. When every host in a batch has failed, there is nothing to dispatch the next task to, and the play ends — which is why the default “stops” without any keyword and why it stops so late.

max_fail_percentage is evaluated after each task, against the failed count for the current batch and the batch size at entry. It ends the play. The playbook executor then moves on to the next play normally, which is why a max_fail_percentage abort does not prevent a later cleanup play from running.

any_errors_fatal raises a condition that the playbook executor checks between plays, which is why it stops the whole playbook. This is the only one of the three with reach beyond its own play, and that reach is the property most often discovered during an incident rather than during review.

None of the three consults the unreachable-host record. All three were verified to let a rollout proceed past an unreachable host on 2.21.3.

What happens if the change is fine but the health check is wrong?

The failure policy does its job perfectly and stops a good deploy.

This is the common false positive, and it is worth planning for because the response under time pressure is usually wrong. The tempting move is to rerun with the health check disabled, which converts a working safety mechanism into no safety mechanism at the exact moment you are least sure what is happening.

The correct sequence is the same as for any failed rollout: establish what is actually true on the hosts in the failing batch — is the service answering, is it serving correct responses — and only then decide. If the service is genuinely healthy and the check is wrong, the fix is to the check, and the rollout resumes with --limit scoped to the hosts that have not yet been done.

A health check that produces false positives will be disabled by somebody eventually, and after that it protects nothing. That is an argument for checks that assert something specific enough to be trustworthy, which is the subject the rolling part opens with.

Knowledge check

Why this matters

The failure policy is a line of YAML that encodes a judgement about what kind of broken you can live with. It is worth writing down for that reason alone: a reviewer reading any_errors_fatal: true on a migration play learns something about the change that no amount of reading the tasks would tell them.

The policy that gets deployed most often is the one nobody wrote, and it is the worst of the three. A play with serial and no failure keyword looks careful. It has the shape of a controlled rollout, the batches appear in the log, and it will still put a broken change on every host in the fleet — one tidy batch at a time, with a full audit trail of having done so.

Staging and stopping are two mechanisms. Writing only the first one is how a rollout becomes a well-documented outage.