Reconciling a fleet after a partial rollback · Ansible

The window aborted, the rollback ran on some hosts, and now somebody has to say what state the fleet is in. The instinctive model is two populations: rolled back, and not.

It is four.

Population	How it got there	What is true of it
Old	Never entered the play	Pre-change state, and nobody looked at it
New	Changed and validated	Post-change state, confirmed
Partial	In the failing batch when the play ended	Some tasks applied, some not — per host
Rolled back	The reverse procedure ran	Claims to be pre-change state

The last two are the work, and the fourth is not as safe as it sounds: “the reverse procedure ran” is a claim about a playbook exit code, not about a host.

Building the host list for each population

Start from the denominator. The recap is not it.

Read-only / Safeestablish the population sets— ansible-inventory and --list-hosts resolve patterns without connecting to anything. Everything here is read-only.

# The denominator: what the window was supposed to touch.
ansible-playbook -i inventories/prod site.yml \
--limit 'appservers:!maintenance' --list-hosts > window-hosts.txt

# The full inventory, for cross-checking that nothing has been forgotten.
ansible-inventory -i inventories/prod --graph appservers

# The recap gives you only the hosts that entered the play. Everything in
# window-hosts.txt with no recap line is population "old".

The set arithmetic is the whole exercise, and it is worth doing in a file rather than in your head at 03:20:

old = window hosts − hosts appearing in the recap
new = hosts with failed=0 that reached the final task
partial = hosts in the recap that did not reach the final task
rolled back = hosts the reverse procedure was run against

Two of those definitions are unsatisfying, and that is honest. “Reached the final task” is not something the recap reports directly — the ok count is a per-host task count, and comparing it against an expected number is fragile because skipped tasks vary by host. This is the argument for the change play writing a completion marker as its last task, which turns population membership into a fact you can query rather than a number you infer.

Make the play tell you where it got to

The last task of a change play should record that the change completed on this host, and the reverse procedure should record the same thing.

- name: Record that the change completed on this host
  ansible.builtin.copy:
    content: |
      change_id: {{ change_window_id }}
      state: applied
      revision: {{ target_revision }}
      at: {{ ansible_date_time.iso8601 }}
    dest: /var/lib/app-change/state.yml
    owner: root
    group: root
    mode: '0644'

After an abort, a read-only play that slurps that file from every host in the window list produces the four populations directly, with no inference from recap arithmetic. A host with no file is old, a host with the previous change ID is old, a host with state: applied is new, and a host with the current change ID and anything else — or a file the play could not read — is partial.

The cost is one task. The alternative is reconstructing populations from scrollback.

Choosing the reference state

The decision that must be made before any reconciliation runs, and the one that gets made by default.

The default is “converge everything to the newest revision”, because that is what running the playbook does. If the newest revision is the change that just failed, the default is to re-apply the failed change to the whole fleet, including the hosts that were successfully rolled back.

Three candidate references, and the criteria for choosing:

Reference	Choose it when	What it costs
Pre-change	The change failed for a reason not yet understood	The rolled-forward hosts must be rolled back too, including any that hit the point of no return
Post-change	The failures were environmental and the change is sound	The old and partial hosts must complete the change, in a window you may not have
A third state	The change was sound but needs a fix	A new change, planned properly, rather than a reconciliation

The third row is the one to take seriously. “Fix it and re-run against the stragglers” during the same window is how a controlled abort turns into an uncontrolled change. The abort criteria fired; the correct response is usually to reconcile to a known state and go home.

Proving convergence without changing anything

Once the reference is chosen, check mode against each population tells you the distance to it.

Read-only / Safemeasure the gap before closing it— --check --diff changes nothing. Run it per population, because the interesting output is different for each one.

# Population "partial" - the important one. Expect a mixed picture.
ansible-playbook -i inventories/prod site.yml \
--limit @partial-hosts.txt --check --diff

# Population "rolled back" - expect changed=0 if the reverse worked.
ansible-playbook -i inventories/prod site.yml \
--limit @rolledback-hosts.txt --check --diff

# Population "old" - expect a uniform diff, identical on every host.
ansible-playbook -i inventories/prod site.yml \
--limit @old-hosts.txt --check --diff

Reading each result:

Rolled back, changed=0: the reverse procedure converged those hosts to what the reference revision declares. That is real evidence and it is bounded by what the role manages — the previous lesson on rollback patterns explains what it does not cover.

Rolled back, changed>0: the reverse did not fully land. Worth stopping on, because the plan currently believes those hosts are safe.

Old, uniform diff: expected and reassuring. Every host in that population shows the same diff, because they are all in the same untouched state.

Old, non-uniform diff: those hosts were not identical before the window either. This is pre-existing drift, discovered by the incident, and it should be recorded as a separate finding rather than fixed silently during a reconciliation.

Partial, anything: read it host by host. This population has no uniform state by definition.

Check mode reports on the world where earlier tasks did not run

The convergence measurement inherits every limitation of check mode, and one of them bites hard here.

A play whose later tasks depend on earlier tasks having run will report nonsense in check mode for exactly the hosts you are most interested in — the partial ones, where some earlier tasks have run and others have not. The reported diff describes a hypothetical host.

The mitigation is to reconcile from evidence rather than from a predicted diff: package_facts for versions, service_facts for service state, stat for file presence and mode, and the change-state file above. Those report what is, not what would change. Part XLIX covers building that evidence properly; here the point is narrower — --check --diff sizes the gap, and facts establish the state.

Why a partial host is not simply 'some tasks ran'

The mental model of a partial host as a prefix of the task list — tasks 1 to 7 applied, 8 onward not — is right for a linear play and wrong in three ways that matter.

Handlers. Handlers run at the end of a play or batch, not at the point of notification. A host that was changed by task 4, notifying a service restart, and then hit the abort at task 6 has the new configuration on disk and the old configuration in the running process. The file says the change applied; the service disagrees. Nothing in the recap distinguishes this from a host where neither happened.

Per-host divergence within the batch. Different hosts in the failing batch reached different points, because a task that failed on one host was skipped on another and succeeded on a third. “The partial population” is a label for a set whose members have nothing in common except when they stopped.

Tasks that are not idempotent on re-entry. Most are. The exceptions are the ones that matter: a command with no creates guard that ran once will run again; an append to a file that is not managed as a whole will append twice; a migration that recorded no marker will re-run.

The consequence for reconciliation: re-running the play against a partial host is not guaranteed to be safe just because the role is idempotent. Idempotence is a property of the tasks, and the partial host is in a state no task was written to expect. The check-mode diff plus fact-based evidence, read per host, is the way to establish which of the three shapes above applies before anything is re-run.

For the common case — a role of template, package and systemd_service tasks with no bare command — re-running is fine, and saying so is as important as the warning. The population that needs individual attention is usually small, and identifying it is what the evidence is for.

The report the change board is owed

Not a narrative. A table with one row per host in the resolved window list, and no host missing.

Field	Content
Host	From the window host list, not the recap
Population	old / new / partial / rolled back
Evidence	The fact that establishes it — package version, change-state file, check-mode result
Action taken	Reconciled to X, or deferred
Outstanding	What is still true and should not be

Three properties make it useful:

It is complete. Every host in the window list appears. A host with no information appears with “unknown” in the population column, which is a finding rather than an omission.

It states evidence, not belief. “Rolled back” is a belief. “acme-app 2.4.1-1 per package_facts, changed=0 in check mode against the pre-change revision” is evidence.

It names what is still outstanding. Hosts deferred, drift discovered, the point-of-no-return hosts that cannot be returned. These become work items, and an item that is not in the report is an item nobody will do.

What happens if the fleet has to stay diverged for a week?

Frequently the answer, and it is a legitimate outcome rather than a failure — but it needs to be managed rather than tolerated.

Four things make a deliberately diverged fleet survivable:

Both states must be supported. If the new and old versions cannot coexist behind the same load balancer, talk to the same database, or be monitored by the same checks, divergence is not an option and the reconciliation is urgent rather than scheduled.
Inventory must record it. A group like pending_change_001, populated from the change-state file rather than maintained by hand, so that anyone running any play knows the populations exist. A divergence that lives only in a report is a divergence the next engineer will not know about.
Automation must not silently close the gap. A half-hourly converge run against the newest revision will roll the whole fleet forward on its own schedule, which is exactly what the abort was supposed to prevent. Either the scheduled run is limited away from the diverged population, or the revision it applies is pinned.
There is a date. Divergence without a date becomes the permanent state of the estate, and a fleet with three generations of a role applied to different thirds of it is how drift stops being fixable.

Point 3 is the one that surprises people, and it is worth checking before anyone goes home: what runs against this fleet automatically, and what revision does it apply?

Knowledge check

Why this matters

This is the lesson nobody writes into a change plan, because a change plan describes a change that works and, failing that, a change that is rolled back. It does not describe the third outcome, which is the most common one: the change partly happened, partly reversed, and the fleet now has more states than the plan has words for.

The cost of not handling it is not paid that night. It is paid three weeks later, when a routine play behaves differently on eleven hosts and nobody can say why — because those eleven are the partial population from a window everyone has stopped thinking about, and the report from that window said “rolled back” without saying which hosts.

The defence is unglamorous and cheap: one task writing a change-state file, a host list built from --list-hosts rather than the recap, and a report with a row for every host including the ones nothing is known about. That is thirty minutes at the end of a bad night, and it is the difference between an incident that closed and an incident that turned into drift.