Retry, roll back, quarantine or investigate · Ansible

You have the four host lists. You know whether the 220 are consistent. You have audited the play for rerun safety. Now something has to happen, and there are exactly four things it can be.

Branch	What it means	What it costs if wrong
Retry	Rerun the change against the hosts that did not get it	Compounds the damage on hosts you did not understand
Roll back	Return the converged hosts to the previous state	A second fleet-wide change, at night, under pressure
Quarantine	Accept the split, isolate the affected hosts, continue tomorrow	The split becomes permanent if the list is not owned
Investigate	Change nothing until the cause is known	Every minute of a live incident where the change was the cause

Nobody chooses well among four options at 03:00 with a half-converged fleet in front of them and a channel asking for updates. The purpose of this lesson is to move the choice earlier: the criteria go into the change plan, before the run, so that at 03:00 the work is matching evidence to criteria rather than inventing them.

Retry

Choose when: the cause is understood, the fix is in place, and the play is rerun-safe against the specific hosts you are targeting.

Evidence required:

A named cause for the failures, not a category. “Package lock contention on hosts whose apt-daily-upgrade.timer fired at 02:31” is a cause. “Transient errors” is not.
The per-task rerun audit, completed.
A --check --diff run against the retarget list showing per-host what remains to be done.
Stuck resources cleared first: half-finished package transactions, held locks, services in a transitional state.

Do not choose when: the failures are heterogeneous. Thirty hosts failing thirty different tasks is not one problem with one retry; it is a fleet that does not match the play’s assumptions, and retrying tests the same assumptions again.

Roll back

Choose when: the converged hosts are demonstrably worse than the unconverged ones, and a reversal path exists that you have tested.

Evidence required:

A specific harm on the 220, observed rather than feared. Error rates up, latency up, a service failing, a dependency broken.
A reversal that is a designed artefact: the previous package versions pinned and available, the previous config in version control, the backup taken by the play itself before it changed anything.
The knowledge that rolling back is itself a fleet-wide change with the same blast radius as the one that just failed.

Do not choose when: the harm is hypothetical. “Half the fleet is on the new config” is a consistency problem, not a harm, and consistency problems are usually better solved by finishing than by reversing — reversing means touching 220 working hosts to fix 80 that are not.

Quarantine

Choose when: the converged hosts are healthy, the affected hosts can be isolated from traffic or from downstream dependencies, and the remaining work can wait for daylight.

This is the branch that is under-chosen, because it feels like not solving the problem. It is frequently the correct answer, and it has one absolute requirement.

Evidence required:

The 220 verified healthy against the actual service, not the recap.
A way to isolate: remove the affected hosts from the load balancer, exclude them from the next change, mark them in the inventory.
A written list with an owner and a date. Without this, quarantine is not a decision, it is forgetting with extra steps.

Read-only / Safemarking a quarantine in the inventory so the next run cannot silently include it— A group whose membership is the incident output. The change plays exclude it by pattern, so the hosts stay out until someone removes them deliberately. Inventory data only — nothing executes.

# inventory/prod/quarantine.yml
all:
children:
  quarantine_2026_08_11:
    hosts:
      app-047: {}
      app-112: {}
      app-203: {}
    vars:
      quarantine_reason: 'Partial patch 2026-08-11, state unverified'
      quarantine_owner: 'platform-oncall'
      quarantine_review_by: '2026-08-14'

Read-only / Safeand the plays that must respect it— The exclusion is in the hosts line, so it applies to every run of this play by default rather than depending on the operator remembering a --limit. Play structure only.

- name: Application configuration
hosts: 'appservers:!quarantine_2026_08_11'
become: true
serial: '10%'

An inventory group is a better quarantine record than a wiki page because it is enforced rather than advisory: the next person to run the play excludes the hosts whether or not they read the incident notes.

Investigate

Choose when: you cannot describe what happened, or when the change itself is a plausible cause of a live problem.

Evidence required: none. This is the branch you choose because you lack evidence, and it is the correct default when the other three criteria are unmet.

Investigating is not passive. It means: stop the run if it is still going, freeze the fleet against further automated change, capture the evidence before it decays, and go look at hosts. What it does not mean is applying changes while thinking.

Who decides

The four branches do not carry the same authority.

Branch	Who can decide	Why
Investigate	Anyone, unilaterally	It changes nothing; it can never be the wrong first move
Quarantine	The operator, with a record	Reversible, low blast radius, but creates an obligation someone must own
Retry	The operator if the cause is named and the play is safe; otherwise escalate	It is a production change with real blast radius
Roll back	Escalate. Always.	It is a second fleet-wide change against healthy hosts, decided under pressure

Rollback deserving an escalation is not bureaucracy. It is the branch where an individual under stress can turn a partial failure affecting 80 hosts into a full outage affecting 300, and the cost of a two-minute phone call is much lower than the cost of being wrong.

The criteria belong in the change plan

Write them before the run, when nothing is on fire. A usable section is short:

Read-only / Safethe decision criteria, as part of the change record— A structured block committed alongside the playbook, so the criteria are reviewed in daylight and available verbatim during the incident. Documentation, not executed.

change: patch-and-config-2026-08-11
blast_radius: 300 hosts, appservers, serial 10%
abort_criteria:
- more than 5% of any batch fails
- any host becomes unreachable after the firewall task
- service health check fails on any converged host

on_partial_failure:
retry_if: >-
  a single named cause accounts for the failures, the play passes the
  rerun audit, and check mode against the retarget list is clean
rollback_if: >-
  converged hosts show a measured regression in error rate or latency
  AND the pinned previous versions are available
quarantine_if: >-
  converged hosts are verified healthy and the affected hosts can be
  taken out of rotation until 2026-08-14
otherwise: investigate

rollback_authority: platform-lead or delegate
rollback_tested_on: staging, 2026-08-04

The last two lines do most of the work. rollback_authority means nobody has to work out who to call. rollback_tested_on means the rollback path is a thing that has been executed rather than a paragraph somebody wrote.

Under the hood

Why does a four-way decision need writing down, when an experienced engineer can reason it out?

Because the reasoning is not the hard part; the conditions are. At 03:00 you have degraded working memory, incomplete information, an audience, and a strong bias toward action — every one of which pushes toward the branch that is fastest to start rather than the branch that is correct.

Written criteria change what the task is. Instead of “decide what to do about a partial fleet failure”, which is an open-ended judgement, the task becomes “check whether the evidence matches these four written conditions”, which is a much smaller cognitive load and produces reviewable reasoning. The person on call does not need to be the person with the best judgement about this specific change; they need to be someone who can read.

It also makes the criteria arguable in advance, at review time, by people who are not tired. “We would roll back if error rates rise” is a sentence someone can push back on in a change review meeting. It is not a sentence anyone can usefully push back on while the fleet is half-converged.

What happens if two branches both look correct?

They usually can be combined, and the combination is often the best answer. The four are not mutually exclusive; they are exclusive only in what you do first.

The common good combination is quarantine then investigate: isolate the affected hosts so nothing downstream depends on them, then take the time to understand the cause properly, then retry in daylight with a named fix. That sequence gets the urgency down without giving up on resolving the split.

The combination to be careful with is retry then investigate. It sounds like the same thing in a different order and it is not: retrying first destroys the evidence you would investigate with. The failed hosts currently record what happened once; after a retry they record what happened twice, with the original failure buried under the second.

If you are going to do both, capture the evidence first — the four host lists, the per-task failure histogram, a sample of hosts left untouched for inspection — and only then retry.

Retry

Roll back

Quarantine

Investigate

Who decides

The criteria belong in the change plan

Knowledge check