Waves: ordering a fleet by consequence · Ansible

Up to about a hundred hosts, hosts: all is a defensible line in a playbook. You can read the recap. You know roughly which machines are in there. If it goes wrong you can log into every affected host in an afternoon.

At two thousand hosts none of that is true. The recap is two thousand lines nobody reads. Nobody can enumerate the set from memory. And a change that breaks something breaks it on more machines than the on-call engineer can visit in a week.

The structural answer is not a bigger playbook or more forks. It is to stop treating the fleet as one target and start treating it as an ordered sequence of decisions.

Batches are not waves

These two words get used interchangeably and they are different objects.

A batch is what serial produces. It is a concurrency control inside a single play: Ansible runs the whole play against the first batch of hosts, then the next, then the next, in one ansible-playbook process, with no pause and no human involved.

A wave is a unit of the change programme. It has an entry criterion, a verification step, a person who decides whether to continue, and — critically — a boundary at which the run has actually stopped.

You need both, and for different reasons. serial protects you from changing everything simultaneously within a wave. Waves protect you from continuing after evidence that you should not.

Ordering by consequence

The ordering rule is one sentence: each wave should be one the organisation could survive being wrong about.

That is not alphabetical, and it is not by rack. Both of those orderings are attractive because they are already written down somewhere, and both are uncorrelated with consequence. app-a01 through app-a40 may be the entire payments tier.

A workable ordering for most estates:

Wave	Contents	What it is for
0	The controller’s own test hosts and internal tooling	Proves the change applies at all
1	A representative canary: a handful of hosts spanning every OS version, hardware class and role the change touches	Proves the change is correct, not just applicable
2	Non-customer-facing production: batch workers, internal apps	First real production exposure, low visible consequence
3	One failure domain of customer-facing production	Proves it under real traffic with an intact fallback
4–n	The remaining failure domains, one at a time	Volume, with the ability to stop

The interesting row is wave 1, and it is the one people get wrong.

A canary that proves nothing

The natural way to pick a canary is to take the first three hosts in the group. That produces a canary which is convenient and usually unrepresentative: three machines built the same week from the same image in the same rack.

If the change is sensitive to OS minor version, to whether the host was built before or after a base-image change, to CPU vendor, or to whether a host has the legacy storage layout, a canary drawn from one build batch cannot see it. The change passes wave 1, and finds the variation in wave 3 across four hundred hosts.

A canary is a sampling problem. Enumerate the dimensions the change could plausibly depend on, and take at least one host from each value.

Read-only / Safefind the dimensions your canary must span— Reads cached or freshly gathered facts and counts hosts per OS version. Run the same shape for any dimension the change might depend on.

ansible -i inventory/ all \
-m ansible.builtin.setup \
-a 'gather_subset=!all,!min,distribution' \
--tree /tmp/facts

grep -ho '"ansible_distribution_version": "[^"]*"' /tmp/facts/* \
| sort | uniq -c | sort -rn

If that returns five OS versions, a three-host canary from one of them is a wave 1 that proves the change applies and says nothing about whether it is correct.

The wave map

Drawn out, an estate’s wave plan is a grid: waves across, host classes down, with an explicit gate between each pair of waves.

flowchart LR
  subgraph W0["Wave 0 - tooling"]
    A1["build hosts x6"]
  end
  subgraph W1["Wave 1 - canary"]
    B1["1 host per OS version"]
    B2["1 host per hw class"]
  end
  subgraph W2["Wave 2 - internal prod"]
    C1["batch workers x140"]
    C2["internal apps x60"]
  end
  subgraph W3["Wave 3 - customer, domain A"]
    D1["web x300"]
    D2["api x180"]
  end
  subgraph W4["Wave 4-n - remaining domains"]
    E1["domains B..E x1200"]
  end

  W0 -->|"gate: applies cleanly"| W1
  W1 -->|"gate: correct on every variant"| W2
  W2 -->|"gate: 24h soak, no regressions"| W3
  W3 -->|"gate: traffic-level verification"| W4

The arrows are the point. Each one is a place where a human reads evidence and decides, and where the automation has genuinely stopped — not a place where the next batch starts because the previous one did not throw an error.

Waves are inventory, not playbook logic

Express waves as groups. A wave is a property of the fleet, so it lives where the fleet is described.

Read-only / Safeinventory/waves.yml— Wave membership as inventory data. Hosts appear in their role groups too; wave groups are an orthogonal dimension over the same hosts.

all:
children:
  wave0_tooling:
    hosts:
      build01.example.com:
      build02.example.com:
  wave1_canary:
    hosts:
      web014.example.com:    # bookworm, gen9 hardware
      web221.example.com:    # bookworm, gen11 hardware
      api047.example.com:    # trixie, gen11 hardware
      wrk103.example.com:    # trixie, virtual
  wave2_internal:
    children:
      batch_workers:
      internal_apps:
  wave3_domain_a:
    children:
      web_domain_a:
      api_domain_a:

Then a wave is a --limit, and the blast radius of any run is a question you can answer before you run it:

Read-only / Safehow many hosts is this wave, exactly?— Illustrative output. --list-hosts resolves the pattern against the inventory and prints the set without connecting to anything.

$ ansible-playbook -i inventory/ site.yml --limit wave1_canary --list-hosts

playbook: site.yml

play #1 (all): converge base configuration	TAGS: []
  pattern: ['all']
  hosts (4):
    web014.example.com
    web221.example.com
    api047.example.com
    wrk103.example.com

Illustrative output

Four hosts. That number, read before the run rather than counted in the recap afterwards, is the whole discipline.

What actually happens at a batch boundary

Two behaviours of serial surprise people the first time they meet them at scale, and both were confirmed by running them.

A batch in which every host fails ends the play. With serial: [1, 30%, 100%] over ten hosts, batch one is a single host. If that host is unreachable, the run stops there — batches two and three never execute:

Read-only / Safethe canary batch was unreachable— Executed against ansible-core 2.21.3 with an inventory of ten hosts in 192.0.2.0/24, which is unreachable by design. Exit code was 4.

$ ansible-playbook -i inventory.ini serial.yml

PLAY [serial batching shape] ***************************************************

TASK [touch nothing] ***********************************************************
fatal: [web01]: UNREACHABLE! => {"changed": false, ...}

PLAY RECAP *********************************************************************
web01                      : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

That is the behaviour you want — but note what the recap does not say. There is no line reading “nine hosts were never attempted”. Nine of ten hosts are simply absent, and a monitoring rule that alerts on failed or unreachable counts will report one problem host on a run that achieved nothing.

run_once fires once per batch, not once per play. With the same serial list and a run_once: true task:

Read-only / Saferun_once under serial— Executed against ansible-core 2.21.3, ten hosts, connection: local, serial [1, 30%, 100%]. The batches are 1, 3 and 6 hosts; the run_once task fires in each.

$ ansible-playbook -i inventory.ini serial-runonce.yml

PLAY [serial batching shape] ***************************************************
TASK [report] ******************************************************************
ok: [web01]
TASK [once per batch] **********************************************************
ok: [web01] => {"msg": "run_once fired"}

PLAY [serial batching shape] ***************************************************
TASK [report] ******************************************************************
ok: [web02]
ok: [web03]
ok: [web04]
TASK [once per batch] **********************************************************
ok: [web02] => {"msg": "run_once fired"}

PLAY [serial batching shape] ***************************************************
TASK [report] ******************************************************************
ok: [web05]
ok: [web06]
ok: [web07]
ok: [web08]
ok: [web09]
ok: [web10]
TASK [once per batch] **********************************************************
ok: [web05] => {"msg": "run_once fired"}

Three batches, three firings. If that run_once task posts to a change channel, drains a load balancer pool, or takes a database snapshot, it happens once per batch. Across forty batches on a real fleet that is forty notifications, or forty snapshots.

Under the hood

serial does not create three plays in your file; it makes the task queue manager iterate the same play once per batch. That is why the output shows a fresh PLAY [...] header for each batch and why per-play semantics — run_once, any_errors_fatal evaluation, max_fail_percentage evaluation, and the “no hosts left, end the play” condition — are all evaluated per batch.

The percentages resolve against the size of the play’s host list, not the remaining hosts: over ten hosts, 30% is three. The final 100% took the remaining six because six were left, not because the arithmetic said six.

Knowing that the batch is the unit of play semantics explains most of the surprising behaviour, and it also explains why a wave gate cannot be built out of serial: there is no play-level hook that means “stop and wait”, only “stop because too much broke”.

Which wave is allowed to fail

Every wave plan should name, in writing, the wave whose failure is an expected outcome rather than an incident.

Usually it is wave 1. That is the point of a canary: it exists to absorb the failure. If a canary failure triggers an incident bridge and a post-mortem, engineers will stop putting interesting changes through it, and within a quarter the canary is a formality that only sees changes already known to be safe.

The corollary matters as much. Once a change has cleared wave 2, a failure in wave 3 is an incident, because the change was declared verified and the verification was wrong. Different waves earn different responses, and saying so in advance is what makes the difference credible.

What happens if a wave passes but you cannot tell why?

This is the failure mode that quietly voids the whole scheme.

A wave gate that reads “the playbook completed with no failures” is satisfied by a run that did nothing. Tasks were skipped by a when that evaluated differently in production, a tag filter excluded the change, or the --limit matched fewer hosts than intended.

The gate criterion has to be a positive statement about the hosts, not the absence of an error: the service reports the new version, the rendered config on disk contains the new directive, the metric moved. “Zero failures” and “the change is present” are different claims, and only the second one is evidence.

Count the recap lines against the expected host count as well. A run that touched forty hosts when the wave contains four hundred passes every failure-based gate you can write.

Blast radius, stated per wave

Before any wave runs, three numbers should be on the change record:

Host count for the wave, from --list-hosts, not from memory.
Concurrency inside the wave — the effective serial/forks number, which is how many machines are mid-change simultaneously.
Worst case — how many hosts would be in a broken state if the change is bad and nothing stops the run.

The third is the one that changes designs. A wave of four hundred hosts at serial: 50 has a worst case of four hundred, because nothing in that run will notice a change that breaks the service without failing a task. Adding a verification task that fails the batch turns the worst case into fifty.

Who is permitted to run which of these waves against which environment is a separate control, and an organisational one rather than a technical one. Part XXXIX covers the platform and role-based access side of that question; the point here is that the wave structure is what makes such a control expressible at all.