Break/Fix: the laptop lid closed during a fleet change and nobody knows which hosts got what · Break/Fix · Ansible

Reported symptoms

An engineer starts a configuration change across 180 hosts from a laptop, over the office wireless, at 16:40 on a Friday. At 16:52 the wireless drops for eleven seconds.

The terminal shows output up to a point and then stops. There is no play recap. Reconnecting to the controller shows no ansible-playbook process running.

The immediate questions:

How many hosts completed?
Which ones?
Is any host in a half-changed state?
Can we just run it again?

Nobody can answer any of them from anything on the controller. The scrollback is gone with the terminal, there is no log file, and there is no retry file.

The proposal on the table at 16:58 is to rerun the play against the full inventory and let it converge.

Evidence provided

Read-only / Safethe run died with the session

$ pgrep -af ansible-playbook

Read-only / Safeno log, no retry file, and one useful default

$ ansible-config dump | grep -E 'RETRY_FILES|LOG_PATH|DEFAULT_STRATEGY'

DEFAULT_LOG_PATH(default) = None
DEFAULT_STRATEGY(default) = linear
RETRY_FILES_ENABLED(default) = False
RETRY_FILES_SAVE_PATH(default) = None

Read-only / Safenothing was recorded anywhere

$ ls -la ~/*.retry /var/log/ansible/ 2>&1 | head -3

ls: cannot access '/home/deploy/*.retry': No such file or directory
ls: cannot access '/var/log/ansible/': No such file or directory

Read-only / Safethe only source of truth left is the fleet itself

$ ansible all -i inventory -b -m ansible.builtin.stat -a 'path=/etc/app/app.conf checksum_algorithm=sha256' -o | awk '{print $1}' | head -3

app001
app002
app003

Read-only / Safe112 done, 67 not, and 180 hosts in the inventory

$ ansible all -i inventory -b -m ansible.builtin.stat -a 'path=/etc/app/app.conf checksum_algorithm=sha256' 2>/dev/null | grep -o '"checksum": "[a-f0-9]*"' | sort | uniq -c

    112 "checksum": "9f2c...new"
67 "checksum": "41ab...old"

Read-only / Safeone host was interrupted mid-task

$ ansible all -i inventory -b -m ansible.builtin.find -a 'paths=/var/tmp patterns=ansible-tmp-* file_type=directory' -o 2>/dev/null | grep -v '"matched": 0'

app144 | SUCCESS => {"matched": 1, "files": [{"path": "/var/tmp/ansible-tmp-1786553521.4-2210-88"}]}

Work the evidence before reading on

112 plus 67 is 179. The inventory has 180 hosts.

Which host is missing from the checksum tally, and what does the temporary directory on app144 tell you about it?
The play used the linear strategy. What does that guarantee about how far apart two hosts can be in the play?
There is no recap and no retry file. Is there any artefact on the controller that records which hosts completed?

Before continuing: before deciding whether to rerun, what property of the playbook do you need to establish, and where would you look to establish it?

Root cause

1. The run died with the terminal

The controller process was a child of the SSH session. When the session went, the process went, and everything the run held in memory went with it.

There is no partial recap because the recap is printed once, at the end. There is no partial log because no log path was configured. There is no resume list because retry files are disabled by default in current ansible-core - a .retry file is something older documentation promises and modern installations do not produce unless asked.

None of that is a malfunction. It is the shape of an interactive run, and it is only visible when a run does not finish.

2. The fleet state exists only on the fleet

This is the part worth internalising, because it changes what the first action should be.

After an interrupted run there is no authoritative record on the controller. The record is the fleet: which hosts have the new file, which have the old, and which are in neither state. It has to be measured, and measuring it is cheap - read-only modules against the specific artefacts the play writes.

The measurement is also the only thing that makes the rerun decision defensible. Without it, “run it again” is a guess about 180 machines.

3. `linear` bounds the problem, and that is genuinely helpful

Under the default linear strategy every host completes a task before any host begins the next. So at any instant the fleet is split between at most two adjacent tasks: those that finished task N and those still on it.

That is why the checksum tally splits cleanly into two groups plus one straggler. The investigation is not “which of 40 tasks did each host reach” but “which side of one task is each host on”, which is a question you can answer with a single read-only command across the fleet.

Under the hood

The straggler is app144, and the leftover directory under /var/tmp is what a host interrupted mid-task looks like.

Ansible creates a temporary directory per task on the target, writes the module payload into it, executes it, and removes it afterwards. A directory that survives means the run was killed between those steps - either before the module executed, or after it executed and before the cleanup.

Which of those it is matters, and the leftover directory does not tell you. The module may have completed its work and reported nothing, or it may never have run at all. For a file write you can distinguish them by checking the file; for a package installation, by checking the package database; for a command, often not at all, which is precisely why unguarded commands make interrupted runs so expensive.

Two smaller details are worth knowing:

Leftover temporary directories accumulate. They are small, but a host that has survived several interrupted runs will collect them, and cleaning them up is part of closing the incident rather than an optional tidy.
With pipelining enabled the payload is streamed to the interpreter rather than written to disk, so there is no temporary directory to find - and correspondingly no evidence that a task was interrupted. The tidier configuration is the one with less forensic trace.

Whether a rerun is safe is a property of the playbook, not of the incident

The question at 16:58 was “can we rerun this”. It is answerable, and it is answerable by reading the play rather than by discussing the outage.

A play can be rerun from an unknown state when every task is idempotent: templates, files, packages, service states, users, and commands guarded by creates or removes. Running such a play twice from any partial state converges.

A play cannot be safely rerun when it contains anything that accumulates or fires once:

An append, a counter, or a lineinfile whose pattern does not match its own output.
A one-shot database migration, especially one under run_once.
A command or shell with no guard, particularly one that restarts, rotates or issues something.
Any task that consumes a token, allocates an address, or increments a version.

Deciding this belongs in the runbook, written down before the incident, because at 16:58 on a Friday it will be answered from memory by whoever is loudest.

Resolution

Do not rerun yet. Say so explicitly, because the proposal is already on the table and it is the natural instinct.
Measure the fleet. Use read-only modules against the specific artefacts the play writes - file checksums, package versions, service states - and build a table with one row per inventory host.
Account for every host. In this incident 179 of 180 fall into two clean groups; the missing one is the host that was interrupted mid-task and it needs individual attention.
Inspect the interrupted host directly. Establish whether the module completed its work before the process died, using the artefact rather than the leftover temporary directory, which cannot tell you.
Clean up leftover temporary directories on any host that has them, so the next investigation is not confused by evidence from this one.
Read the play and decide whether a rerun is safe. Every task idempotent means yes; anything that appends, fires once, or runs unguarded means no, and those tasks need to be excluded or handled individually.
Rerun against the hosts that need it, derived from the measurement rather than from the inventory, from a session that cannot die - a multiplexer on the controller or a scheduler that owns the process.
Converge once more afterwards and require changed=0, which is the evidence that the fleet has settled rather than that another run happened to finish.

Verification

The measurement table is complete. One row per inventory host, each with a definite state. A table with 179 rows for 180 hosts is the incident continuing.
The interrupted host is resolved individually, on the evidence of the artefact rather than on an assumption about how far the task got.
No leftover temporary directories remain. A find across the fleet for the temporary directory pattern returns nothing.
The fleet has converged. A final run reports changed=0 on every host; a run that completes is not the same as a fleet that agrees.
The new logging works under failure. Start a run on a scratch host, kill it deliberately, and confirm the log contains enough per-host detail to reconstruct progress. This is the check that can fail, and a logging setup that has never survived a failure is untested.
The run survives a lost connection. Disconnect from the controller mid-run and confirm the run continues and can be reattached.
The runbook answers the rerun question. It states, for this play, whether a rerun from an unknown state is safe and which tasks are the exceptions.

Prevention

Run fleet changes from a session that cannot disappear. A terminal multiplexer on the controller, or a scheduler that owns the process, turns a dropped connection into a reconnect.
Configure a log path so every run leaves a durable record independent of the terminal, and keep enough per-host detail in it to reconstruct progress from a partial file.
Design plays to be rerunnable, and say so in the runbook. The answer to “is a rerun safe” should be written down before it is needed.
Guard every unguarded command. creates and removes turn a one-shot command into an idempotent task and make the whole play resumable.
Where an action genuinely can only happen once, record its completion on the host so a rerun can detect it. A marker file is unglamorous and it converts the hardest case into an ordinary one.
If a change must be resumable, capture progress deliberately rather than relying on retry files, which are disabled by default and are not a resume mechanism anyone should be depending on.
Prefer smaller batches for changes that write. A serial value bounds how many hosts can be mid-change when something interrupts the run.

What if the play had used the free strategy instead of linear?

The measurement would have been considerably harder, and this is the strongest practical argument for the default.

Under linear, hosts are synchronised at every task, so at any instant the fleet is split across at most two adjacent tasks. One read-only check answers the question for the whole fleet.

Under free, each host proceeds through the play as fast as it can without waiting. When the run is killed, hosts are scattered across the entire play - one on task 3, another on task 27, another finished. There is no single artefact whose presence divides the fleet in two, and reconstructing state means checking every artefact the play writes on every host.

free exists for good reasons and is genuinely faster on a fleet with uneven hosts. The cost is paid exactly here, on the day a run does not finish, and it should be a deliberate trade rather than a performance default.

There is a second consequence worth noting. A rolling deployment depends on the task-level synchronisation that linear provides - drain, deploy, health-check and return only mean anything in order and in batches. free removes that guarantee, so free and a rolling deployment are incompatible, whatever the timing benefit looks like.

Why this matters

The interruption was trivial. Eleven seconds of wireless, on a Friday afternoon, on a laptop. What made it an incident was that the run kept its only record of progress in the memory of a process attached to that terminal.

Two habits remove almost all of the cost. Run from somewhere the process survives you, and log to somewhere the terminal does not own. Both are one-line changes and neither is interesting until the day it matters.

The third habit is harder and more valuable: build plays whose partial execution leaves a state you can describe and recover from. That means idempotent tasks, guarded commands, and a written answer to whether a rerun is safe. An interrupted run is not an exceptional event on a fleet of any size - connections drop, controllers reboot, tokens expire

and a play that can only be reasoned about when it completes is a play that will one day have to be reasoned about anyway, with no evidence and under time pressure.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. The run died with the terminal

2. The fleet state exists only on the fleet

3. linear bounds the problem, and that is genuinely helpful

Resolution

Verification

Prevention

3. `linear` bounds the problem, and that is genuinely helpful