When the rollout fails halfway · Ansible

Batch four of eleven failed its health check. The failure policy did its job and stopped the play.

You now have a fleet in three or four different states at once, an exit code that tells you almost nothing about which hosts are in which, and a decision to make. This is the situation the whole part has been building toward, because it is the one that actually happens.

The four states

flowchart LR
  subgraph FLEET["the fleet, immediately after the stop"]
    direction LR
    A["batches 1-3<br/><b>NEW</b><br/>deployed, healthy,<br/>in service"]
    B["batch 4, failed host<br/><b>BROKEN</b><br/>deployed, unhealthy,<br/>drained"]
    C["batch 4, healthy hosts<br/><b>ORPHANED</b><br/>possibly deployed,<br/><b>drained, never returned</b>"]
    D["batches 5-11<br/><b>OLD</b><br/>untouched,<br/>in service"]
  end
  C -.->|"the urgent one"| U["capacity is reduced<br/>and nothing said so"]

The third column is the one that gets missed.

Batches 1–3 are fine. New version, healthy, serving traffic.

The failed host in batch 4 is broken and you know it — the health check said so, it is in the log, it is why the play stopped.

The healthy hosts in batch 4 are the problem. Part XXXI established by execution that when a failure policy triggers, the surviving hosts in the batch do not run the remaining tasks. They were drained. They may or may not have been deployed to. They were never returned to service. And the recap reports them with failed=0:

Read-only / Safethe recap of an aborted batch, from Part XXXI— Executed on ansible-core 2.21.3. h02 and h03 report failed=0 and are nonetheless unfinished. Later-batch hosts have no recap line at all.

$ ansible-playbook -i inv6.ini rollout.yml

PLAY RECAP *********************************************************************
h01                        : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
h02                        : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
h03                        : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

h02 and h03 look like successes. The only tell is ok=1 where a completed host would show ok=2, and nobody reads a recap that closely at 03:00.

Batches 5–11 were never touched and are absent from the recap entirely.

Capacity, not correctness, is the emergency

The instinct after a failed deploy is to investigate why it failed. That is the second thing to do.

The first is to establish that the fleet is carrying its traffic. A failed rollout with orphaned hosts has silently reduced capacity, and the service is still up only because the remaining hosts are absorbing the load. That works until the next traffic peak, at which point a deployment that failed at 03:00 becomes an outage at 09:00 that nobody connects to it.

The named incident: capacity dropped 33% at 03:00 and nobody noticed until the morning peak. The play aborted after draining node 2 and never returned it; the load balancer stayed exactly as the play left it.

Check the load balancer first. Not the Ansible log, not the hosts. The load balancer is the only system that knows which hosts are in service, and it is the one nobody opens.

Establishing the true split

Do not reason about it from the run log. The log tells you what Ansible attempted; the fleet tells you what is true.

Three questions, and they need three different sources.

Read-only / Safethe post-failure census— Read-only throughout. Asks each host what is on disk and what it is serving, and asks the load balancer whether it is in the pool. Run against the whole group, not the batch you think failed.

- name: Establish the real state of the fleet
hosts: appservers
gather_facts: false
tasks:
  - name: What version is on disk
    ansible.builtin.slurp:
      src: /etc/app/deployed-version
    register: marker
    failed_when: false

  - name: What version is the process actually serving
    ansible.builtin.uri:
      url: 'http://{{ ansible_host | default(inventory_hostname) }}:8080/healthz'
      return_content: true
    register: live
    failed_when: false
    changed_when: false

  - name: Is the load balancer sending it traffic
    ansible.builtin.uri:
      url: >-
        https://lb.example.com/api/v1/pools/{{ app_pool }}/members/{{ inventory_hostname }}
      method: GET
      headers:
        Authorization: 'Bearer {{ lb_api_token }}'
      return_content: true
    delegate_to: localhost
    register: member
    failed_when: false
    changed_when: false

  - name: Report all three together
    ansible.builtin.debug:
      msg: >-
        {{ inventory_hostname }}
        disk={{ (marker.content | default('') | b64decode | trim) or 'ABSENT' }}
        serving={{ live.json.version | default('NO-ANSWER') }}
        lb={{ member.json.state | default('UNKNOWN') }}

The combinations and what each means:

disk	serving	lb	Meaning	Urgency
new	new	active	deployed and in service	done
old	old	active	never touched	fine
new	new	disabled	deployed, healthy, orphaned	urgent
old	old	disabled	drained before deploy, orphaned	urgent
new	old	any	written, never restarted	investigate
new	no answer	disabled	deployed and broken	expected, contained
any	no answer	active	broken and serving traffic	most urgent

The last row is the worst case and it is possible: a host whose health check the play never got to, left enabled, serving errors to users. If your play returns hosts to service before checking them, or if a previous run left state behind, this is where it shows up.

Hold, forward, or reverse

Now the decision. All three are legitimate; the criteria are what matter.

Hold

Leave the fleet split. Fix nothing tonight.

Correct when all three hold:

The versions are compatible. Old and new can run simultaneously without breaking each other — the version-skew question. If you have not established this, you cannot hold.
Capacity is intact. No orphaned hosts, or they have been returned.
The split is stable. Nothing will change it while you are asleep — no autoscaling that would launch hosts on an unknown version, no scheduled job that assumes uniformity.

When those hold, holding is frequently the best decision available. Nothing is broken, and diagnosis at 10:00 with the team awake is better than diagnosis at 03:00 alone. The failure this avoids is the one where a tired person makes a hasty change and turns a stable split into an outage.

Roll forward

Finish the rollout.

Correct when the new version is known good and the failure was incidental — a repository timeout, a host with a full disk, a transient network problem — or when the split itself is the danger and completing it is the fastest way out.

The mechanism is --limit scoped to the hosts still on the old version, generated from the census rather than assumed:

Service impact possibleresuming against the remainder— List first, then check, then run. The limit file is the change artefact — review it before the third command.

ansible-playbook site.yml --limit @/tmp/still-on-old.txt --list-hosts

ansible-playbook site.yml --limit @/tmp/still-on-old.txt --check --diff

ansible-playbook site.yml --limit @/tmp/still-on-old.txt

The precondition people skip: fix the thing that failed first. If the failure was a genuine problem with the new version, rolling forward deploys a known-broken release to the rest of the fleet, faster than the original rollout would have because you have now disabled the gate that stopped it.

Roll back

Return the deployed hosts to the old version.

Correct when the new version is the problem.

It is also the option people most overestimate. A rollback is a deployment: same drain, same restart, same health checks, same risk — executed by a more tired person under more time pressure, on a path that is usually less rehearsed than the forward one.

It is only fast if it was built in advance:

The previous artefact is still available. A package repository that keeps one version, or a deploy directory that keeps the last release, is the difference between a five-minute rollback and a rebuild.
Any migration applied by the new version is reversible, and the reverse was written and tested before the forward one ran. This is the most common thing that makes rollback impossible.
The rollback path is the same playbook with a different release_version, not a separate script that has never been run.

Under the hood

Nothing in Ansible models a deployment, which is why none of this information is available from the run.

The play iterator knows about batches while it is iterating and forgets them when the play ends. The recap is assembled by a callback from per-host counters — ok, changed, failed, unreachable, skipped, rescued, ignored — and none of those counters expresses “this host completed the play”. That is why an aborted batch produces failed=0 hosts whose only distinguishing feature is a lower ok count, and why hosts in unreached batches produce no line at all rather than a line saying they were skipped.

There is also no state carried between runs. Ansible is not a deployment system with a database of what it deployed where; every run starts with no knowledge of any previous run. The --limit you construct after a failure is you supplying, by hand, the state that the tool does not keep.

This is a deliberate design position rather than an oversight — Ansible converges hosts toward a described state and does not track history — and it has a direct operational consequence. The source of truth for what is deployed is the fleet, and the source of truth for what is in service is the load balancer. Neither of them is the Ansible log, and building a recovery procedure on the log rather than on the systems themselves is the mistake this lesson exists to prevent.

What happens if hosts are autoscaled while the fleet is split?

New instances launch from whatever the image or bootstrap provides, which is neither of your two versions necessarily, and definitely not under the control of your paused rollout.

This is the case that makes “hold” unsafe. A stable split assumes the set of hosts is stable. In an autoscaling group it is not: a traffic peak launches three instances from the base image, they join the pool running whichever version that image carries, and the split you decided was safe now includes a third state you did not evaluate.

Part XXXI noted that a play resolves its host list once at the start and cannot chase a moving fleet. The consequence here is stronger: if the fleet can grow on its own, the version that new instances receive is a property of the image, not of your playbook, and a rolling deploy that does not also update the image is deploying to a set of hosts that is already being diluted.

Where that is the architecture, holding is rarely right. Either complete the rollout — including the image — or reverse it, so that the fleet has one version and new instances match it.

Knowledge check

Why this matters

Every part of a rolling deployment gets designed except this one.

The batch sizes are argued about. The health check gets iterated on. The failure policy is chosen deliberately. And then the rollout fails at batch four — which is the outcome all of that machinery was built to produce — and there is no procedure for what happens next.

The failure policy stopping the play is a success. It did exactly what you asked. But stopping is not a resolution; it hands you a fleet in four states and a decision that depends on facts you have to go and collect.

Three things make that survivable, and all of them are cheap to build before you need them. A census playbook that reports disk version, served version and load balancer state together. A rollback path that is the same playbook with a different version, tested once. And a written answer to “can these two versions run at the same time”, decided before the rollout rather than during it.

The last one is the subject of the next lesson, and it is the one that determines whether “hold” is even on the table.