Break/Fix: the rollout stopped after two of six nodes and two more are missing from the report entirely · Break/Fix · Ansible

Reported symptoms

A rolling deployment of the application across six nodes stops after the second batch. The last thing in the output is NO MORE HOSTS LEFT, twice, and then a recap.

It is 22:40. The deployment window closes at 23:30. The questions in the room, in the order they were asked:

Which nodes have the new version?
Which nodes are serving traffic?
Why does the recap list four nodes when there are six?
Should we just run it again?

Nobody can answer the first three from the run output, and the fourth is about to be answered with “yes” because it is 22:40.

Evidence provided

Read-only / Safethe PLAY header repeats per batch - that is what batching looks like

$ grep -E 'PLAY \[|TASK \[|ok:|changed:|fatal:|NO MORE' logs/rollout-2226.log

PLAY [Deploy the application] **************************************************
TASK [Drain from the load balancer] ********************************************
changed: [node01]
changed: [node02]
TASK [Deploy and verify] *******************************************************
ok: [node01]
ok: [node02]
PLAY [Deploy the application] **************************************************
TASK [Drain from the load balancer] ********************************************
changed: [node03]
changed: [node04]
TASK [Deploy and verify] *******************************************************
fatal: [node03]: FAILED! => {"msg": "health check returned 503 after 6 attempts"}
ok: [node04]
NO MORE HOSTS LEFT *************************************************************
NO MORE HOSTS LEFT *************************************************************

Read-only / Safefour rows - node05 and node06 are not here at all

$ tail -6 logs/rollout-2226.log

PLAY RECAP *********************************************************************
node01                     : ok=6    changed=3    unreachable=0    failed=0
node02                     : ok=6    changed=3    unreachable=0    failed=0
node03                     : ok=3    changed=1    unreachable=0    failed=1
node04                     : ok=4    changed=1    unreachable=0    failed=0

Read-only / Safethe failure policy

$ grep -n -A4 'hosts: app' deploy.yml

3:  hosts: app
4:  serial: 2
5:  max_fail_percentage: 25
6:  gather_facts: true

Read-only / Safethree on the new version, three on the old

$ for n in 01 02 03 04 05 06; do printf 'node%s ' $n; curl -s --max-time 3 http://192.0.2.1$n:8080/version || echo unreachable; done

node01 2.4.0
node02 2.4.0
node03 2.3.1
node04 2.4.0
node05 2.3.1
node06 2.3.1

Read-only / Safetwo nodes drained; only one of them failed

$ curl -s http://lb.example.com:8404/stats;csv | awk -F, '$1=="app_pool" {print $2, $18}'

node01 UP
node02 UP
node03 MAINT
node04 MAINT
node05 UP
node06 UP

Work the evidence before reading on

Nothing here is a malfunction. Every component did what it was configured to do.

Count the hosts in the recap. Count the hosts in the inventory. The difference is the most important number in the incident.
Work out what fraction of a two-host batch one failure represents, then compare it with max_fail_percentage.
node04 reports the new version and is in MAINT at the load balancer. Which step ran and which step did not?

Before continuing: if you rerun the play right now, what happens to node04, and what happens first?

Root cause

1. The failure policy triggered on the first possible failure

serial: 2 divides six hosts into three batches of two. The play runs completely for batch one, then completely for batch two, and so on - which is why the PLAY header appears once per batch in the log.

max_fail_percentage: 25 tells Ansible to abandon the play when more than 25 percent of the hosts in the current batch have failed. In a batch of two, one failure is 50 percent. There is no failure smaller than that available, so the setting means “stop on the first failure”, and it was almost certainly copied from a play with much larger batches where 25 percent tolerates a genuine minority.

The play stopped. That is the configured behaviour, correctly executed.

2. Hosts in unstarted batches are absent, not reported as skipped

node05 and node06 were in batch three. Batch three never began.

They are not listed as skipped, or as unreachable, or as anything else - they are simply not in the recap. Reading the recap tells you about four hosts and says nothing whatever about the other two, and the only way to notice is to already know how many there should have been.

That is the mechanism behind the question nobody could answer at 22:40. The report is not wrong; it is incomplete by construction, and it does not say so.

3. The drain step has no counterpart on failure

node03 was drained, failed its check, and stopped. node04 was drained, deployed successfully, and then the play abandoned it before the return-to-service step.

So capacity is not reduced by one node. It is reduced by two, one of which is perfectly healthy and running the new version. Nothing in the Ansible output says this; the load balancer says it, and only if somebody asks.

An orphaned drained node is the worst state in the incident because it is invisible from every angle except the load balancer: the host is up, the service is running, the version is correct, and it is serving nothing.

Under the hood

Batched execution is straightforward to observe, and worth observing once so that the log is readable under pressure.

A play with serial: 2 over six hosts, where one host fails and max_fail_percentage: 25 is set, produces exactly the shape above: the PLAY header repeated per batch, NO MORE HOSTS LEFT when the threshold is crossed, and a recap containing only the hosts that actually ran. Hosts in unstarted batches never appear.

Two consequences follow, and both matter at 22:40:

The recap host count is the number of hosts that started, not the number the play targeted. Comparing it against --list-hosts is the cheapest way to detect a truncated rollout, and it can be automated.
run_once executes once per batch, not once per play. A migration or a notification placed under run_once in a batched play runs as many times as there are batches, which is a separate and unpleasant surprise waiting in the same file.

The exit code is also worth knowing: a play that aborts this way exits non-zero, so a pipeline notices. What the pipeline cannot tell you is which hosts are in which state, and that is the information the incident turns on.

Rerunning is the instinct and it is the wrong move

At 22:40 with fifty minutes of window left, rerunning looks like the fastest path. Consider what it actually does.

The play starts at batch one: node01 and node02, both already on the new version. They are drained again, redeployed, health-checked and returned. That is real risk and real time spent on nodes that were already correct.

Meanwhile node04 stays drained throughout, because it is in batch two and the play will not reach it for several minutes - during which the fleet is down another node.

And if node03 fails its health check again, which is likely since nothing about it has changed, the play stops in exactly the same place, having achieved nothing and having drained and undrained four nodes along the way.

The correct first action after any aborted rollout is not an action at all. It is to establish the true state of every node, from the nodes and the load balancer, and write it down. That table is the input to every subsequent decision and it cannot be reconstructed later.

Resolution

Do not rerun. Announce that explicitly, because the instinct is strong and somebody is already typing it.
Build the state table before doing anything else. One row per node in the inventory, with columns for version, service health, and load balancer status, gathered from the nodes and the load balancer rather than from the Ansible log.
Deal with the orphaned drained node immediately. node04 is healthy and running the new version; return it to service or take it out deliberately, but make capacity a known number rather than an unknown one.
Find out why node03 failed its health check. This is the only genuinely unknown thing in the incident, and it decides whether rolling forward is safe.
Decide between hold, roll forward and roll back on the evidence and the time remaining. Two versions running simultaneously is only acceptable if the versions are compatible, which is a question about the application, not about Ansible.
Execute the decision with a play limited to the nodes that need it. Nodes already on the target version and in service must not be touched again.
Fix the failure policy afterwards, not during the incident. max_fail_percentage has to be computed against the batch size, and the value should carry a comment showing the arithmetic.
Wrap the drain in a block with an always section that returns the node to service, so a failure between drain and return cannot orphan a node again.

Verification

The state table has one row per inventory host. Six rows for six nodes; a table with four rows is the original failure repeating itself.
Capacity is a known number, confirmed at the load balancer. No node is in a maintenance state that nobody chose.
Every node reports the same version, if the decision was to roll forward or back. Query each node directly rather than inferring from the run.
Traffic is actually being served by every node that is supposed to be serving. A node marked UP that receives no requests is a different fault, and the request counters at the load balancer are what distinguish them.
The failure policy behaves as intended. Rehearse it on staging: force one node in a batch to fail and confirm the play stops or continues as you now intend. This is the check that can fail and it is the one nobody had run.
A drained node is always returned. In that same rehearsal, confirm the always section returns the healthy node in the failing batch to service.
The recap-versus-target comparison is automated. The pipeline compares the number of hosts in the recap against the number reported by --list-hosts and fails when they differ.

Prevention

Compute max_fail_percentage against the batch, and write the arithmetic beside it. One failure in a batch of two is fifty percent regardless of how large the fleet is, and a value copied from a larger-batch play means something entirely different here.
Never let a drain exist without a paired return. A block whose always section returns the node to service is the difference between a failed deployment and a silently degraded fleet:

- name: Deploy with a guaranteed return to service
  block:
    - name: Drain from the load balancer
      ansible.builtin.include_tasks: drain.yml
    - name: Deploy and health-check
      ansible.builtin.include_tasks: deploy.yml
  always:
    - name: Return to service
      ansible.builtin.include_tasks: undrain.yml

Treat the play recap as incomplete. Compare its host count against the targeted count on every batched run, and alert on any difference.
Make the state table a required step after any aborted rollout, before any decision. It cannot be reconstructed from the log afterwards.
Rehearse the failure path. The batch policy is the one setting whose behaviour you only observe on a bad day, which makes staging the only place to observe it cheaply.
Remember that run_once means once per batch. Anything that must happen exactly once in a batched play needs a different mechanism.
Decide in advance what a mixed-version fleet means for this application. If the two versions cannot coexist, a rolling deployment is the wrong pattern and the play should say so.

What if max_fail_percentage had not been set at all?

Then the play would have continued through every batch, deploying to all six nodes, and stopping only for hosts that individually failed.

Whether that is better depends entirely on what the failure means. If node03 failed for a reason specific to that node - a stuck process, a full disk, a stale artefact - then continuing is right, and stopping the rollout on one node’s local problem is an overreaction that leaves the fleet split.

If the failure is a property of the new version, continuing deploys a broken release to the entire fleet as fast as the batches allow. That is exactly what max_fail_percentage exists to prevent, and it is why the setting is right in principle and wrong in value here.

There is a third policy worth knowing. any_errors_fatal stops the play for every host as soon as any host fails, rather than counting a percentage. It is the correct choice when the hosts are not independent - a cluster where a partial change is worse than no change - and it is too blunt for a load-balanced tier where one bad node is routine.

Whichever policy is chosen, the orphaned-drain problem is unaffected. any_errors_fatal and max_fail_percentage both stop the play, and a play that stops between drain and return leaves nodes out of service under either. Only the always section fixes that.

Why this matters

The most dangerous state in this incident is not the failed node. It is the healthy node running the correct version and serving no traffic, because nothing except the load balancer knows about it and nobody thought to ask.

Rolling deployments trade a big risky change for a sequence of small ones, and the trade only holds if you can say what state the fleet is in when the sequence stops. Ansible tells you what it did to the hosts it reached. It does not tell you about hosts it never started, it does not tell you which steps ran and which did not on the host that failed mid-sequence, and it has no view of the load balancer at all.

So the discipline is to reconstruct the fleet state from the fleet before deciding anything, and to build plays whose partial execution leaves a state you can describe. A drain that always returns, a recap count that is compared against the target, and a failure threshold whose arithmetic somebody actually did - none of them are clever, and together they are the difference between a stopped rollout and an unknown one.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. The failure policy triggered on the first possible failure

2. Hosts in unstarted batches are absent, not reported as skipped

3. The drain step has no counterpart on failure

Resolution

Verification

Prevention