Is it safe to run it again? · Ansible

“Just run it again” is the most common response to a partial fleet failure and it is right often enough to be dangerous.

It is right when every task is idempotent, when the target list is correct, and when nothing was interrupted mid-transaction. Those three conditions are checkable in about fifteen minutes. Running without checking them is not a decision, it is a reflex, and the cases where the reflex is wrong are the expensive ones.

Rerun safety is not a property of the run

It is a property of each task, decided when the task was written.

That is worth being blunt about, because it means the answer at 03:00 is already determined. You are not deciding whether the rerun is safe; you are discovering whether the person who wrote the play made it safe. The only thing you can do at 03:00 is find out, and the only place to find out is the playbook.

The three-column audit

Open the play. For every task, fill in three columns:

Task	Safe to repeat?	What a second run does
`template` writing a config	Yes	Rewrites identical content, reports `ok`
`package` with `state: present`	Yes	Reports `ok`, no transaction
`package` with `state: latest`	Mostly	May install a newer version than the first run did
`command` running a migration script	Depends on the script	Unknown until you read the script
`command` appending to a file	No	Appends a second time
`shell` with a redirect creating a file	Yes, if it truncates; no, if it appends	Read the redirect operator
`uri` POSTing to an API	Usually no	Creates a second resource, sends a second notification
`user` with `generate_ssh_key`	Yes	Does not regenerate an existing key
`file` with `state: absent`	Yes	Idempotent by construction
Anything with `creates:` or `removes:`	Yes	That is what the argument is for

The audit takes fifteen minutes for a hundred-task play and it is the only honest input to the decision. If you cannot fill in the third column for a task, that task is a no until you can.

Note the state: latest row. It is idempotent in the sense that it converges, and it is not repeatable in the sense that matters here: a rerun four hours later may pick up a package published in between, so the 30 retried hosts end up on a version the original 220 do not have. That is not a failure — it is a new inconsistency, created by the fix.

Building the rerun target list

You need a --limit file. There is a trap in how people expect to get one.

Read-only / Saferetry files are off by default — read from ansible-config on 2.21.3— ansible-config list prints the shipped defaults for every setting. It reads configuration and executes nothing against any host.

$ ansible-config list | grep -A 3 '^RETRY_FILES_ENABLED'

RETRY_FILES_ENABLED:
default: false
description: This controls whether a failed Ansible playbook should create a .retry
  file.

RETRY_FILES_ENABLED defaults to false. The --limit @site.retry idiom that appears in older material and in a great deal of Stack Overflow does not work out of the box on 2.21, because the file is not written.

You can turn it on. It is worth knowing why the default moved: a .retry file is written next to the playbook, silently, containing a host list from a previous run. Two runs later it is stale, and --limit @site.retry then targets a set of hosts chosen by an incident that happened last Tuesday. Several estates have run a change against the wrong hosts exactly that way.

Build the list from the run artefact instead

The list you want is not “hosts that failed”. It is one of four different lists, and which one depends on what you decided:

Failed hosts only — you have diagnosed the failure and fixed the cause.
Unreachable hosts only — connectivity is restored and these hosts need the change they never got.
Failed plus unreachable — the usual “finish the job” case.
Everything except the confirmed-good — the paranoid case, correct when you no longer trust the 220 either.

Each is one line of text extraction from the log, each is explicit, and each leaves a file you can show someone before you run it. That reviewability is the actual argument against .retry: not that the mechanism is broken, but that a hand-built list can be read, diffed and attached to the change record.

Read-only / Safeproduce and verify the limit file— Text extraction from an existing log, then a read-only confirmation that the pattern resolves to exactly the hosts you intend. --list-hosts parses the inventory and prints the match; it connects to nothing.

log=/var/log/ansible/patch-2026-08-11.log

awk '/failed=[1-9]/     {print $1}' "$log" | sort -u >  retarget.txt
awk '/unreachable=1/    {print $1}' "$log" | sort -u >> retarget.txt
sort -u -o retarget.txt retarget.txt
wc -l retarget.txt          # expect 80

# Prove the limit resolves to exactly those hosts, before running anything
ansible-playbook -i inventory/prod patch.yml \
--limit @retarget.txt --list-hosts

The --list-hosts step is not optional ceremony. A limit file containing a hostname the inventory does not know produces a warning and a smaller host list than you expect, and at 03:00 nobody reads warnings. Count the hosts it prints and compare to the number you wrote down.

What a rerun does to an interrupted host

This is the case the audit table cannot cover, because the question is not “is this task idempotent” but “is this task idempotent when applied to a host on which it was already half-executed”.

Three shapes recur:

The half-finished package transaction. The rerun does not get as far as your task; apt or dnf refuses to do anything until the previous transaction is resolved. This fails loudly and safely, which is the good outcome — you get an error naming the real problem.

The partially written file. Modules that write via a temp file and rename (copy, template, assemble) leave either the old file or the new one, never a half-file, so the rerun simply completes the job. A shell task using a redirect does not have this property and can leave a truncated file that looks valid.

The service that was mid-restart. The rerun evaluates a service that is in a transitional state. Depending on timing, the service module may report ok for a service that is failing to start, because it asked at the moment systemd still considered it starting.

Establish state before you rerun, not after

Check mode is the tool for this, and this is the situation it is genuinely for.

Running the play with --check --diff --limit @retarget.txt asks every supporting module what it would do, and prints the diff. On a host that completed the change before its connection died, you should see nothing to do. On a host that never started, you should see the whole change. On a half-converged host, you see exactly where it stopped — which is the fact you were unable to get from the run log.

The limits are real and you must know them: command, shell, raw and script skip in check mode unless given check_mode: false, so a play built on those returns a check run that proves very little. That is one more consequence of the same design decision as the last lesson, and the check-mode part of this course covers what each module supports.

Read-only / Safecheck mode against the retarget list, before any real rerun— --check makes supporting modules predict rather than act; --diff prints the content they would write. No managed node is modified. Read the changed counts per host: they tell you which population each host is in.

$ ansible-playbook -i inventory/prod patch.yml --limit @retarget.txt --check --diff

PLAY RECAP *********************************************************************
app-047   : ok=12  changed=0  unreachable=0  failed=0   # already converged
app-112   : ok=8   changed=4  unreachable=0  failed=0   # stopped part-way
app-203   : ok=6   changed=9  unreachable=0  failed=0   # never started

Illustrative output

Three hosts, three changed counts, three different situations — and none of them were distinguishable in the original recap.

Under the hood

Why is a .retry file a worse artefact than a text file you wrote yourself, given they contain the same thing?

Because of who chose the contents and when. The retry file is produced as a side effect of a failing run, into a path derived from the playbook name, overwriting whatever was there. Nothing marks it with a timestamp in its name, nothing prevents a later run from replacing it, and nothing in --limit @site.retry at the command line tells a reviewer which run’s failures it holds.

The failure this produces is not exotic. A run fails on Tuesday, writing 80 hosts. Someone fixes the cause and reruns successfully on Wednesday — a successful run does not write a retry file, so Tuesday’s file survives untouched. On Thursday a different failure affects 3 hosts, writing a new file with 3. Someone who remembers the earlier incident runs --limit @site.retry expecting Tuesday’s set and gets Thursday’s, or the reverse if the Thursday run happened to be a different playbook name.

Making the default false does not fix that reasoning error, but it does stop the file appearing unbidden in repositories, which is how it became a habit in the first place. The habit worth building instead is that any --limit file used against production is a file you can show someone, named for the incident, committed or attached to the change record.

What happens if I rerun the whole 300 rather than the 80?

Mostly nothing, and that is exactly why it is tempting — a fully idempotent play against 220 already-converged hosts reports ok everywhere and costs only time.

Three reasons not to:

Time is the resource you are short of. 300 hosts at the same rate as the original run is another 40 minutes, in the middle of an incident, during which you cannot do anything else with the controller.

You re-expose the 220 to risk for no benefit. They are currently correct. Every task that runs against them is another chance for a handler to fire, a service to restart, or a state: latest task to pick up a package version the first run did not install. The blast radius of your fix should be the hosts that need fixing.

It destroys the population boundaries. After a full rerun, the run log no longer distinguishes the hosts that needed work from the hosts that did not, and your evidence about what happened at 02:47 is one generation older.

The exception is the paranoid case, where the consistency check across the 220 came back with more than one distinct answer. Then the 220 are not a confirmed-good population, and re-converging everything is the right call — made deliberately, on evidence, not by default.

The decision, condensed

Audit the play, task by task. Any task you cannot classify is a blocker.
Build the target list explicitly from the run artefact. Verify it with --list-hosts.
Establish per-host state with --check --diff against that list.
Resolve stuck resources — package transactions, locks, stale PID files — on the hosts that have them, before the rerun.
Rerun with serial and a small first batch, even though the play already ran once. The conditions changed; treat it as a new change.

Step 5 is the one people skip because the play “already worked on 220 hosts”. It worked at 02:06 on a fleet in a state that no longer exists.

Knowledge check

Why this matters

The rerun is where a partial failure either resolves or compounds, and the difference is almost entirely preparation.

There is a version of this night where someone reads the play, builds a named host list, runs it in check mode, finds three hosts with stuck package transactions, clears those by hand, and re-converges the remaining seventy-seven in one batched run. It takes ninety minutes and appears in the incident review as a competent recovery.

There is another version where someone types the same command as before without --limit because the play is “idempotent anyway”, and discovers at 04:30 that one task was a command that appends to a file, that it has now appended twice on 220 hosts that were previously fine, and that the blast radius of the fix exceeded the blast radius of the failure.

Both engineers know Ansible equally well. The difference is that one of them treated the rerun as a new change with its own blast radius, and the other treated it as undoing the last one.