Canary first, with criteria written down · Ansible

Progressive exposure is one host, then a batch, then the fleet, with a decision between each. The decision is the point. Without it, serial: 5 is a slower way of changing everything.

This lesson is about making the decision mechanical, because the alternative — an operator watching output and deciding when to stop — is a control that works during a demonstration and fails at 03:00 on the third hour of a window.

Sizing the exposure

serial accepts a list, and the list is the exposure schedule.

- name: Roll out the release
  hosts: appservers
  serial: [1, 3, 6]
  max_fail_percentage: 10

Read it as: one host, then three, then six at a time until the fleet is done. The first entry is the canary.

The numbers are not universal, and copying them from another team’s playbook is the usual mistake. Three questions size them:

Question	Effect on the canary	Effect on later batches
How long until a problem is visible?	Canary must stay live at least that long	Batches must be spaced by it
What fraction of capacity can be down?	Usually irrelevant — one host	This is the hard ceiling
How long does the whole window allow?	—	Batch size is fleet size ÷ available rounds

The second one is the constraint people skip. A fleet of ten behind a load balancer with serial: 6 takes 60% of capacity out at once. If the service needs 50% to serve peak traffic, serial: 6 is an outage that the play will report as a success.

Criteria a play can evaluate

The change plan holds criteria in prose. The play needs them as conditions. Translating between the two is the work.

Criterion in the plan	Expressed in the play
The service must answer after the change	`uri` with `status_code: 200`, `retries`, `until`
Error rate must stay below 1%	`uri` against the metrics endpoint plus an `assert` on the parsed value
Latency must stay under 400 ms	`uri` with `timeout: 0.4`, or an assert on a measured value
The version must actually be the new one	`assert` comparing a reported version to the intended one
No more than 10% of hosts may fail	`max_fail_percentage: 10`
The window ends at 03:30	An `assert` on the current time, evaluated per batch

Service impact possiblevalidation that can fire an abort— The uri task fails after six attempts if the service does not answer. That failure is what max_fail_percentage counts.

- name: Wait for the service to answer after the change
ansible.builtin.uri:
  url: "http://{{ ansible_host }}:8080/healthz"
  status_code: 200
  timeout: 5
register: health
retries: 6
delay: 10
until: health is succeeded

- name: Confirm the running version is the one we deployed
ansible.builtin.assert:
  that:
    - health.json.version == target_version
  fail_msg: >-
    {{ inventory_hostname }} reports {{ health.json.version | default('unknown') }},
    expected {{ target_version }}
  quiet: true

- name: Refuse to start another batch after the window closes
ansible.builtin.assert:
  that:
    - ansible_date_time.time < window_end_time
  fail_msg: "Window closed at {{ window_end_time }}; not starting another batch"
  quiet: true
run_once: true

The third task is the one almost nobody writes, and time is the abort criterion that fires most often.

What an abort actually looks like

Executed, rather than described. Ten hosts, serial: [1, 3, 6], max_fail_percentage: 10, with the health check failing on web02, web03 and web09.

Read-only / Safethe abort, as it happens— Executed on ansible-core 2.21.3 against a ten-host inventory using read-only tasks. The canary passed; the first real batch did not.

$ ansible-playbook -i inventories/prod rollout.yml; echo "exit=$?"

PLAY [Canary then batches, with an abort criterion] ****************************

TASK [Post-change health check] ************************************************
skipping: [web01]

TASK [Mark host as validated] **************************************************
ok: [web01] => {
  "msg": "web01 validated"
}

PLAY [Canary then batches, with an abort criterion] ****************************

TASK [Post-change health check] ************************************************
fatal: [web02]: FAILED! => {"changed": false, "msg": "health check failed"}
fatal: [web03]: FAILED! => {"changed": false, "msg": "health check failed"}
skipping: [web04]

NO MORE HOSTS LEFT *************************************************************
NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
web01                      : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
web02                      : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
web03                      : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
web04                      : ok=0    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

exit=2

Four things in that output are worth reading carefully.

The play header appears once per batch. PLAY [...] printed twice because serial runs the play once per batch. This is why run_once means once per batch rather than once per play, and it is a common source of surprise.

Two failures out of three is 66%, which exceeds 10%, so the play ended. The evaluation is per batch, not against the fleet. Two hosts out of ten is 20% of the fleet, and the fleet number is never computed.

web04 shows skipped=1 and never reached the second task. It was in the failing batch, evaluated the health check, and the play ended before the next task. Its state is “partially through the batch”, which is the population lesson 7 is about.

web05 through web10 have no recap line at all. They were never entered. The recap has four lines for a ten-host fleet, and the six missing hosts are still on the old release — invisible in the output that the operator is reading.

Canary size and the percentage floor

With serial: 1, a single failure is 100% of the batch. Any max_fail_percentage below 100 therefore aborts on a failed canary, which is the intent.

Verified by execution: the same play with the failure induced on web01 instead stopped immediately, produced a one-line recap, and exited 2. The canary is a hard gate by construction, and no arithmetic is needed to make it one.

The arithmetic does matter for later batches, and it bites in a direction people do not expect.

Batch size	`max_fail_percentage: 10` tolerates	Effective behaviour
1	0 failures	Any failure aborts
3	0 failures	1 failure is 33%, aborts
6	0 failures	1 failure is 16%, aborts
11	1 failure	1 failure is 9%, continues
20	2 failures	2 failures is 10% — see below

max_fail_percentage is a strictly-greater-than threshold

The play aborts when the percentage of failed hosts is greater than the configured value. Exactly equal does not abort.

Verified by execution on ansible-core 2.21.3, twenty hosts, serial: 10, max_fail_percentage: 10:

One failure in the first batch (10%): the play continued, the second batch ran, and nineteen hosts reached the task after the failure.
Two failures in the first batch (20%): NO MORE HOSTS LEFT, and zero hosts reached the following task — including the eight hosts in that batch that had not failed.

Teams that intend “no more than 10%” and get “up to and including 10%” are usually not harmed by the difference, but a plan that states an exact-percentage criterion should either use a value that cannot be hit exactly, or state the tolerance the way the play will actually enforce it.

The practical consequence of the whole table: for small batches max_fail_percentage is a fancy way of writing “abort on any failure”, and there is nothing wrong with that as long as everyone knows it is what the play does.

Why max_fail_percentage is per batch, and what that means for the plan

Ansible evaluates the failure percentage at the end of each batch, against the hosts in that batch. There is no running total across the play.

The consequence is a scenario a fleet-level reading of the criterion does not predict. A fleet of 100 with serial: 10 and max_fail_percentage: 10:

Batch 1: one host fails. 1/10 = 10%, not greater than 10, continue.
Batch 2: one host fails. 10%, continue.
… and so on.

Ten batches later the play has completed, exit code 2, with ten failed hosts — 10% of the fleet — and it never aborted, because no single batch ever exceeded the threshold. Every batch was individually within tolerance and the aggregate was not.

If the criterion is genuinely about the fleet, the play cannot enforce it and something outside the play must: a validation play run between batches that counts previously failed hosts, or an orchestration layer that stops calling ansible-playbook.

There is a second consequence for the last batch. serial: 10 against 104 hosts produces a final batch of 4, in which one failure is 25%. The tail of a rollout is measured more strictly than the middle of it, purely as an artefact of division. Uneven final batches make the effective criterion tighten as the run finishes, which is harmless but produces “why did it abort at the very end” questions that are easier to answer if you saw it coming.

What happens if the canary passes and the fleet still fails?

It happens, and it is not a failure of the method.

A canary proves that the change works on one host under the conditions that host experienced during the soak. Three things scale differently:

Shared dependencies. One host reconnecting to a database is invisible. Two hundred hosts reconnecting within thirty seconds is a connection storm the canary could not have shown.
Load per host. With one host changed, the load balancer sends it 1/200th of the traffic. It gets busy at 100%.
Host heterogeneity. The canary is one configuration. Fleets accumulate variants — different kernel versions, hand-edited files, hosts built by a different generation of the role.

Two responses. First, choose the canary deliberately rather than taking whatever serial: 1 picks, which is the first host in inventory order: pick a host that carries real traffic and is representative, and consider running the canary phase on two hosts from different variants rather than one. Second, keep the batch gates after the canary, because the canary passing is not permission to go to the fleet — it is permission to go to the next batch.

For the storm class specifically, throttle and a delay between batches are the mechanisms, and they belong in the plan as an explicit number rather than as whatever the play happens to do.

Knowledge check

Why this matters

The reason to write abort criteria into the play rather than the runbook is not that operators are unreliable. It is that the operator running the window is the person most invested in it working.

They wrote the change, or reviewed it, or stayed up for it. When the canary looks marginal at 02:50, the read is “that is probably unrelated” far more often than it should be — and that is not a character flaw, it is what being three hours into a window does to anyone. The criterion in the play does not have a stake in the outcome.

The secondary benefit is that writing criteria the play can evaluate forces them to be specific. “If the canary looks unhealthy” cannot be written as an assert. Trying to write it is what produces the question what exactly would unhealthy look like, and that question is worth more than the assert it eventually becomes.