Lab: Batch arithmetic — predict the batches, then verify them · Lab · Ansible

Objective

By the end of this lab you will be able to state, for any serial value against any inventory size, exactly which hosts are in which batch — because you predicted twenty-eight batches and verified every one. You will also have reproduced the max_fail_percentage boundary from both sides and established the thing most people get wrong about it: it is evaluated per batch, not across the play.

Architecture

Twenty hosts, all resolving to the controller. No managed nodes are required, because batch composition is a scheduling property and has nothing to do with what the tasks do.

inventory: node01 .. node20   (ansible_connection: local)
play:      serial: <the value under test>
task:      debug the current batch membership, run_once per batch

Requirements

A controller with ansible-core 2.21.x. Every batch listing and recap below was captured from 2.21.3; the arithmetic has been stable for many releases but the exact output formatting has not.
No managed nodes, no SSH, no privilege escalation. The only module used is debug and assert.
Somewhere to write predictions down before running. As with the targeting lab, this is the exercise.

Scenario

Your change policy says production rollouts go one host first, then five, then twenty percent at a time. Somebody has written serial: [1, 5, '20%'] into the playbook, and somebody else has asked what that actually does against the 200-host production group.

Neither of them knows. Nor does the person who wrote the max_fail_percentage: 33 on the line below it, which was chosen because “a third seemed reasonable”.

Tasks

Task 1: Build the harness

WORKDIR="$HOME/ansible-serial-lab"
mkdir -p "$WORKDIR"
cd "$WORKDIR"

ansible-config dump --only-changed

inventory20.yml:

app:
  hosts:
    node[01:20]:
  vars:
    ansible_connection: local

batches.yml — the harness. ansible_play_batch is the list of hosts in the current batch, and run_once makes it print once per batch:

- name: Report batch composition
  hosts: app
  gather_facts: false
  serial: "{{ serial_value }}"
  tasks:
    - name: Show this batch
      ansible.builtin.debug:
        msg: "{{ ansible_play_batch | join(',') }}"
      run_once: true

Task 2: Predict, then verify, five values

Fill in the Predicted column completely before running anything.

`serial`	Predicted batch sizes	Actual
`3`
`7`
`[1, 5, '20%']`
`['10%', '50%', '100%']`
`25`

Then run each:

cd "$HOME/ansible-serial-lab"

for v in "3" "7" "[1, 5, '20%']" "['10%', '50%', '100%']" "25"; do
  echo "--- serial: $v"
  ansible-playbook -i inventory20.yml batches.yml \
    --extra-vars "serial_value=$v" | grep '"msg"'
done

The verified results, captured on ansible-core 2.21.3:

Read-only / Safecontroller— serial: 3 over 20 hosts

$ ansible-playbook -i inventory20.yml batches.yml --extra-vars 'serial_value=3' | grep msg

    "msg": "node01,node02,node03"
  "msg": "node04,node05,node06"
  "msg": "node07,node08,node09"
  "msg": "node10,node11,node12"
  "msg": "node13,node14,node15"
  "msg": "node16,node17,node18"
  "msg": "node19,node20"

Seven batches: six of three and a remainder of two. No rounding, no balancing — the last batch is simply what is left.

Read-only / Safecontroller— The list form. Note what the last element does.

$ ansible-playbook -i inventory20.yml batches.yml --extra-vars "serial_value=[1, 5, '20%']" | grep msg

    "msg": "node01"
  "msg": "node02,node03,node04,node05,node06"
  "msg": "node07,node08,node09,node10"
  "msg": "node11,node12,node13,node14"
  "msg": "node15,node16,node17,node18"
  "msg": "node19,node20"

Read-only / Safecontroller— All percentages

$ ansible-playbook -i inventory20.yml batches.yml --extra-vars "serial_value=['10%', '50%', '100%']" | grep msg

    "msg": "node01,node02"
  "msg": "node03,node04,node05,node06,node07,node08,node09,node10,node11,node12"
  "msg": "node13,node14,node15,node16,node17,node18,node19,node20"

2, then 10, then the remaining 8 — because 100% of 20 is 20 and only 8 are left.

Complete the table with 7 (batches of 7, 7, 6) and 25 (one batch of 20; a serial larger than the host count is a single batch, not an error).

Task 3: Reproduce the failure-policy boundary

Nine hosts, serial: 3, and a failure on exactly one host in the first batch. One of three is 33.33%.

inventory9.yml:

app:
  hosts:
    node[1:9]:
  vars:
    ansible_connection: local

failpolicy.yml:

- name: Failure policy demonstration
  hosts: app
  gather_facts: false
  serial: 3
  max_fail_percentage: "{{ mfp }}"
  tasks:
    - name: Fail on node1 only
      ansible.builtin.assert:
        that: inventory_hostname != 'node1'
        fail_msg: "induced failure on {{ inventory_hostname }}"

Predict both results, then run them:

for m in 33 34; do
  echo "=== max_fail_percentage: $m"
  ansible-playbook -i inventory9.yml failpolicy.yml \
    --extra-vars "mfp=$m" | grep -E 'NO MORE HOSTS|^node'
done

Read-only / Safecontroller— 33.33% failed. The threshold is 33.

$ ansible-playbook -i inventory9.yml failpolicy.yml --extra-vars 'mfp=33'

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
node1                      : ok=0    changed=0    unreachable=0    failed=1
node2                      : ok=1    changed=0    unreachable=0    failed=0
node3                      : ok=1    changed=0    unreachable=0    failed=0

Read-only / Safecontroller— Same failure. The threshold is 34. The play continues.

$ ansible-playbook -i inventory9.yml failpolicy.yml --extra-vars 'mfp=34'

PLAY RECAP *********************************************************************
node1                      : ok=0    changed=0    unreachable=0    failed=1
node2                      : ok=1    changed=0    unreachable=0    failed=0
node3                      : ok=1    changed=0    unreachable=0    failed=0
node4                      : ok=1    changed=0    unreachable=0    failed=0
node5                      : ok=1    changed=0    unreachable=0    failed=0
node6                      : ok=1    changed=0    unreachable=0    failed=0
node7                      : ok=1    changed=0    unreachable=0    failed=0
node8                      : ok=1    changed=0    unreachable=0    failed=0
node9                      : ok=1    changed=0    unreachable=0    failed=0

The comparison is strictly greater than: the play aborts when the failure percentage exceeds the threshold. 33.33 > 33 aborts; 33.33 > 34 does not.

Task 4: Establish that it is per batch

This is the part that matters operationally, and it is where the mental model most often goes wrong.

Change the induced failure so the whole first batch fails:

    - name: Fail on the first three
      ansible.builtin.assert:
        that: inventory_hostname not in ['node1', 'node2', 'node3']

Run with max_fail_percentage: 34 — a threshold that just tolerated one failure a moment ago:

Read-only / Safecontroller— Three of nine hosts is 33%. The play still aborts.

$ ansible-playbook -i inventory9.yml failpolicy.yml --extra-vars 'mfp=34'

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
node1                      : ok=0    changed=0    unreachable=0    failed=1
node2                      : ok=0    changed=0    unreachable=0    failed=1
node3                      : ok=0    changed=0    unreachable=0    failed=1

max_fail_percentage is a percentage of the batch

Three failures out of nine hosts is 33% of the play — which is under the threshold of 34. The play aborted anyway, because the percentage is computed against the current batch, and three of three is 100%.

Consequences worth writing down:

With serial: 1, any single failure is 100% of its batch, so any max_fail_percentage under 100 aborts on the first failure. A canary batch of one plus a percentage threshold is a very strict policy, and that is usually what you want — but it is not what “tolerate 20%” sounds like.
The tolerance you get depends on the batch size, which under a list-form serial changes between batches. serial: [1, 5, '20%'] with max_fail_percentage: 20 tolerates 0 failures in batch 1, 1 in batch 2, and 0 in the 20% batches — because 1/4 is 25% and 25 > 20.
Without serial at all, the whole play is one batch, and max_fail_percentage becomes a fleet-wide tolerance.

Task 5: The recap is not the fleet

Look again at the aborted runs. node4 through node9 appear nowhere in the recap — not as ok, not as failed, not as skipped.

ansible-playbook -i inventory9.yml failpolicy.yml --extra-vars 'mfp=34' \
  | grep -c '^node'

Three lines for a nine-host inventory.

Hosts that were never attempted do not appear at all

The recap lists hosts the play reached. A host in a batch that never ran is simply absent, and there is no not attempted column.

The operational consequence, on the morning after an aborted rollout: you cannot answer “which hosts got the change” from the recap alone. You have to compare the recap’s host list against the inventory:

ansible-playbook -i inventory9.yml failpolicy.yml --extra-vars 'mfp=34' \
  > run.txt 2>&1 || true

# Hosts the run touched
sed -n '/PLAY RECAP/,$p' run.txt | grep -oE '^node[0-9]+' | sort > touched.txt

# Hosts the inventory contains
ansible -i inventory9.yml app --list-hosts | tail -n +2 | tr -d ' ' | sort > all.txt

echo "NOT ATTEMPTED:"
comm -13 touched.txt all.txt

That four-line comparison is the artefact an aborted rollout needs, and it is not produced for you.

Task 6: Three policies, one failing play

Run the same three-host-failure play three ways and put the recaps side by side in policies.md.

No policy at all — remove max_fail_percentage:

ansible-playbook -i inventory9.yml nopolicy.yml | tail -12

The three hosts fail; the play continues to the remaining batches, because without a policy a failed host is simply removed from the play and the rest carry on.

max_fail_percentage: 34 — as above: aborts at the end of batch 1.

any_errors_fatal: true — the strictest:

- name: Failure policy demonstration
  hosts: app
  gather_facts: false
  serial: 3
  any_errors_fatal: true
  tasks:
    - name: Fail on node1 only
      ansible.builtin.assert:
        that: inventory_hostname != 'node1'

One failure anywhere in the batch ends the play for every host in that batch immediately — including hosts in the same batch that had not yet reached the failing task.

Policy	Batch 1 outcome	Batches 2–3
none	failed hosts dropped, others continue	run
`max_fail_percentage: 34`	evaluated at batch end	do not run
`any_errors_fatal: true`	batch stops at the failing task	do not run

Task 7: Answer the scenario’s question

In policies.md, answer the two questions the scenario opened with, for the 200-host production group:

What does serial: [1, 5, '20%'] do? (1, then 5, then batches of 40 — 40 being 20% of 200, and the last element repeating for the remaining 194 hosts, giving batches of 40, 40, 40, 40, 34.)
What does max_fail_percentage: 33 tolerate in each of those batches? (Batch 1: 0 failures, since 1/1 is 100%. Batch 2: 1 failure, since 1/5 is 20% and 20 is not greater than 33; 2/5 is 40% and aborts. Batches of 40: up to 13, since 13/40 is 32.5% and 14/40 is 35%.)

Verify at least the second batch prediction by building a 200-host inventory and running the harness:

cat > inventory200.yml <<'YAML'
app:
  hosts:
    node[001:200]:
  vars:
    ansible_connection: local
YAML

ansible-playbook -i inventory200.yml batches.yml \
  --extra-vars "serial_value=[1, 5, '20%']" \
  | grep -o '"msg": "[^"]*"' | awk -F, '{print NF" hosts"}'

Validation

serial: 3 over 20 hosts produces seven batches sized 3,3,3,3,3,3,2.
serial: 7 produces three batches sized 7,7,6.
serial: [1, 5, '20%'] produces six batches sized 1,5,4,4,4,2 — the last element repeating and 20% being 20% of 20.
serial: ['10%', '50%', '100%'] produces three batches sized 2,10,8.
serial: 25 against 20 hosts produces one batch of 20 and no error.
With one failure in a batch of three: max_fail_percentage: 33 aborts, 34 continues. The comparison is strictly greater than.
With three failures in a batch of three, max_fail_percentage: 34 aborts, proving the percentage is per batch.
An aborted run’s recap lists only the hosts that were attempted; the comm comparison names the rest.
policies.md contains three recaps and the 200-host arithmetic.

Expected Outcome

ansible-serial-lab/
├── batches.yml
├── failpolicy.yml
├── inventory9.yml
├── inventory20.yml
├── inventory200.yml
├── nopolicy.yml
├── policies.md
└── predictions.md

You can compute batch composition for any serial form without running it, and you know that max_fail_percentage is a per-batch tolerance whose effective strictness is determined by the batch size you chose.

Troubleshooting

serial_value is not applied. Templating a play keyword works only from sources available at parse time. --extra-vars works; group_vars does not. Confirm with --list-tasks, which shows the play’s resolved keywords in some versions, or simply observe that all hosts ran in one batch.

--extra-vars "serial_value=[1, 5, '20%']" produces a string, not a list. Depending on your shell quoting, Ansible may receive the literal characters. Use the JSON form to be certain: --extra-vars '{"serial_value": [1, 5, "20%"]}'.

Every batch prints twice. run_once was omitted, so the debug ran per host. That is also a valid way to see batch membership; the output is just longer.

max_fail_percentage appears to do nothing. It has no effect without serial, other than as a whole-play tolerance evaluated once at the end. It is also silently ignored if written as a string in some positions — check ansible-playbook --syntax-check and confirm the value is a number.

The percentage produces a batch size of zero. serial: '1%' against a 50-host group is 0.5, which Ansible rounds up to 1 — it never produces an empty batch. Verify rather than assume for the size you actually use.

A batch runs hosts in an order you did not expect. Batches are taken from the play’s host list in inventory order, but hosts within a batch run in parallel up to forks, so their output interleaves non-deterministically. Batch membership is deterministic; ordering inside a batch is not.

Cleanup

Nothing outside the working directory was written, nothing was installed, and no host was contacted — every “host” in every inventory resolved to the controller via ansible_connection: local, and every task was debug or assert.

Step 1. Confirm that rather than assuming it:

cd "$HOME/ansible-serial-lab"

# No config was created or inherited
ansible --version | grep 'config file'

# Nothing was written outside this directory
find "$HOME" -maxdepth 1 -newer inventory20.yml -not -name 'ansible-serial-lab' 2>/dev/null

Step 2. Keep the two write-ups — the arithmetic is worth having and you will refer to it when someone proposes a serial value:

mkdir -p "$HOME/ansible-lab-deliverables"
cp -a policies.md predictions.md "$HOME/ansible-lab-deliverables/"

Step 3. Remove the working directory by absolute path:

rm -rf deletes the whole lab directory, including anything you added

The next block removes ~/ansible-serial-lab and everything under it, recursively and without confirmation. If you saved notes, an inventory you want to keep, or a playbook you extended, copy it out first — the step above does that for the lab’s own deliverables and knows nothing about yours.

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep, and the shell gives no second chance.

Confirm what you are about to delete before you delete it:

ls -la "$HOME/ansible-serial-lab"

rm -rf "$HOME/ansible-serial-lab"

Under the hood

serial is resolved into a list of batch sizes before the play runs, by Play._get_serialized_batches(). The algorithm is short and explains every result in this lab.

Each element is normalised: an integer is taken as-is; a string ending in % is computed as that percentage of len(all_hosts) — the play’s total host count, fixed at the start — and rounded up to at least 1. The resulting list is then consumed left to right against the remaining hosts, and when it is exhausted the last value is reused until no hosts remain.

That single reuse rule is why [1, 5, '20%'] produces six batches rather than three, and why the final element of a list-form serial is the one that governs most of a large fleet.

max_fail_percentage is evaluated in the strategy’s batch loop, after each batch completes, as (failed_this_batch / hosts_in_this_batch) * 100 > max_fail_percentage. Both the numerator and the denominator are per batch, which is the whole of the Task 4 result. Note also that the comparison is strict, which is why a threshold of 33 aborts on exactly 33.33% and a threshold of 34 does not.

any_errors_fatal is evaluated differently — inside the task loop rather than at batch end — which is why it stops the batch at the failing task rather than letting the rest of the batch finish first. For a play where the tasks after the failure would make things worse, that difference is the whole reason to choose it.

What You Learned

The last element of a list-form serial repeats. [1, 5, '20%'] is six batches over 20 hosts and five over 200, not three.
Percentages are of the play’s total host count, computed once. 20% of 20 is 4 for every remaining batch, not 20% of what is left.
A remainder batch is just what is left, with no balancing, and a serial larger than the inventory is one batch rather than an error.
max_fail_percentage is strictly-greater-than, per batch. 33.33% aborts at 33 and continues at 34, and three failures out of three aborts at 34 despite being 33% of the play.
With serial: 1, any threshold under 100 aborts on the first failure, because one host is 100% of its batch.
The recap lists what was attempted, not what exists. Hosts in batches that never ran are absent entirely, and the comm comparison against --list-hosts is what tells you which.
Three policies, three different failure shapes, and choosing between them is a question about whether a partial batch is worse than no batch.

Lab: Batch arithmetic — predict the batches, then verify them

Objectives

Prerequisites