When the connection dies mid-task · Ansible

Every other outcome in this part is a statement about the host. This one is a statement about your knowledge of the host, and it is the only outcome where the honest answer is that you do not have one.

The connection dropped after the module was delivered and before the result came back. Ansible reports the host UNREACHABLE. That word is doing something quite different here from the case in the last lesson.

What Ansible knows, and what it does not

A task runs in stages, and the disconnect can happen at any of them.

Stage	Happened?	If the link drops here
Connection opened	Known	Nothing ran. Host untouched
Temp directory created	Known	An empty directory in `~/.ansible/tmp`
Module payload written to the remote	Known — the write returned	The file is there; nothing executed it
Module executed	Unknown	Anything between nothing and everything
Module wrote its JSON result to stdout	Unknown	The module may have finished perfectly
Result returned to the controller	Did not happen	This is why you are reading this
Temp directory removed	Did not happen	Evidence, and litter

The line in bold is the problem. From the controller’s point of view, “the module never ran” and “the module ran, completed, changed the host and the result was lost in transit” are indistinguishable. Both produce the same connection error, the same unreachable=1, and the same absence of a result.

This is why idempotency is a safety property

Most material on Ansible presents idempotency as good practice — cleaner reruns, honest changed reporting, better diffs. Those are real benefits and they are not the reason it matters.

The reason it matters is this lesson. The only way to recover from a task whose outcome is unknown is to be able to run it again safely, and that is the definition of idempotency.

Work through it. You have a host that dropped during task 17. You want it converged. Your options are:

Inspect it manually and repair by hand. Correct, expensive, and does not scale past a handful of hosts. Also produces a snowflake unless you feed the repair back into the automation.
Run the play again against that host. Free, instant, scales to fifty hosts — if and only if every task in the play is safe to execute against a host that may have already executed it.
Rebuild the host. Correct in an immutable estate, disproportionate in most others.

Option 2 is the one that makes partial failures survivable at fleet scale, and it is available only if you wrote the play a particular way. Idempotency is not a style choice you make for elegance; it is the property that determines whether option 2 exists at 03:00.

The test to apply to every task you write

If this task were interrupted at its worst possible moment and then run again from the start, would the host end up correct?

For ansible.builtin.copy writing a file: yes. The module writes to a temp path and moves it into place atomically, so an interrupted copy leaves either the old file or the new one, and a rerun produces the new one.

For ansible.builtin.lineinfile with a well-anchored regexp: yes. The second run matches the line it already inserted and reports ok.

For ansible.builtin.command running a script that appends to a file: no. A rerun appends twice. There is no way for Ansible to know that, because command cannot inspect the meaning of what it runs.

For ansible.builtin.command running a script that increments a counter, sends a notification, charges a customer, or generates a new key: no, and the consequences differ every time.

The tasks that fail this test are exactly the tasks that make a rerun dangerous, which is the subject of the rerun-safety lesson later in this part. Here the point is narrower and sharper: those tasks are also the ones that make a mid-task disconnect unrecoverable without manual inspection. Non-idempotent tasks do not merely make reruns untidy. They remove your only cheap recovery path.

What the host looks like afterwards

There are physical traces, and they are worth knowing because they let you distinguish “never started” from “was running”.

Read-only / Safeevidence left on a host that was interrupted mid-task— Run against a host once you can reach it again. Every command reads; none of them modify anything. The temp directory is normally removed at the end of a task, so its presence is the tell.

$ ls -la ~/.ansible/tmp/ 2>/dev/null | tail -5 ; sudo ls -la /root/.ansible/tmp/ 2>/dev/null | tail -5

drwx------  2 svc-ansible svc-ansible 4096 Aug 11 02:31 ansible-tmp-1786489621.15-370915-2222077828
-rw-------  1 svc-ansible svc-ansible 82344 Aug 11 02:31 AnsiballZ_apt.py

Illustrative output

A leftover ansible-tmp-* directory with an AnsiballZ_<module>.py inside it names the module that was in flight and timestamps the moment the run stopped. That is a genuinely useful forensic artefact: it tells you which task was interrupted, which the controller-side log cannot, because the controller never learned that the module started.

Read-only / Safechecking a package manager for an interrupted transaction— Three read-only checks. dpkg --audit reports packages that are unpacked but not configured; the dnf history query shows the last transaction and whether it completed. Neither modifies package state.

$ sudo dpkg --audit ; sudo dnf history list --reverse | tail -3

The following packages are only half configured, probably due to problems
configuring them the first time.  The configuration should be retried using
dpkg --configure <package> or the configure menu in dselect:
linux-image-6.8.0-51-generic

ID     | Command line             | Date and time    | Action(s) | Altered
------ | ------------------------ | ---------------- | --------- | -------
  47 | upgrade --assumeyes      | 2026-08-11 02:29 | E, I, U   |     114 EE

Illustrative output

An interrupted package transaction is the worst realistic case, because the package manager holds locks and refuses to proceed until a human resolves the state — which means the next automated run against that host fails too, on a different task, for a reason that has nothing to do with the play.

`async` with `poll: 0`, and its own failure modes

The documented mitigation for long-running tasks is to detach them. async: <seconds> with poll: 0 starts the module on the managed node in the background, returns a job id immediately, and lets a later async_status task collect the result.

Service impact possibledetaching a long operation so a dropped link does not lose it— The upgrade runs on the managed node independently of the controller connection. The second task reconnects later and asks the node what happened. This is a change-making play: it upgrades packages and is at least SERVICE-IMPACT.

- name: Start the upgrade without holding the connection open
ansible.builtin.apt:
  upgrade: dist
  lock_timeout: 300
async: 3600
poll: 0
register: upgrade_job

- name: Collect the result, retrying if the link is flaky
ansible.builtin.async_status:
  jid: '{{ upgrade_job.ansible_job_id }}'
register: upgrade_result
until: upgrade_result.finished
retries: 120
delay: 30

What this genuinely buys you: the operation is no longer coupled to a single continuous SSH session. If the link drops during the hour the upgrade takes, the upgrade carries on, and the async_status task reconnects and finds out how it went. The unknown-outcome window shrinks from “the whole task” to “the moment of launch”.

What async does not fix

It does not make the task idempotent. If the underlying operation is unsafe to run twice, async changes nothing about that. It changes when you find out.

The job id lives on the controller side of the register. If the run dies before async_status executes — the controller crashes, someone presses Ctrl-C, the play aborts — the job id is gone with the run. The job is still running on the managed node, and you now have no handle on it. The record is in ~/.ansible_async/ on the node, so it is recoverable by hand, but nothing in the play will find it for you.

poll: 0 plus serial is a trap. The batch appears to complete in seconds because every host merely launched its job. If your rolling deployment assumes a completed batch before draining the next one, an async task with poll: 0 and no async_status gate silently removes that guarantee and you are now upgrading the whole fleet at once.

async_status on the same connection is still a connection. If the host is genuinely unreachable when the status task runs, you get unreachable again — with the difference that this time the job really is running, and running it again would be running it twice.

Use async for genuinely long operations where a dropped connection is likely: large upgrades, long-running data operations, anything measured in tens of minutes. Do not use it as a general fix for flakiness. A fleet where connections drop often enough to need this everywhere has a network problem that async is hiding.

Under the hood

Why is the disconnect indistinguishable from a no-op?

The SSH connection plugin runs a remote command and reads its exit status, stdout and stderr. For a normal module invocation, the remote command is a Python interpreter executing the AnsiballZ payload, and the module result is a single JSON document on stdout. Ansible parses that document into the task result.

If the TCP connection is lost, the local ssh process exits with status 255 and whatever it had buffered. Ansible sees a connection-level failure with no parseable result and raises the unreachable path. There is no protocol-level acknowledgement of “the module started” that survives the connection, because the only channel for that information is the connection.

This is not a design flaw so much as a consequence of the agentless model. An agent on the managed node could hold state across a disconnect: it would know a job was started, and the controller could ask it later. Ansible deliberately does not install one, so the managed node keeps no independent record of an in-flight synchronous task. The ~/.ansible_async/ job files are the single exception, and they exist precisely because async needed a way to answer this question — which is the clearest evidence that the general case genuinely cannot.

ControlPersist does not help either. It keeps a multiplexed master connection alive for later tasks; it does not resume a session whose transport died mid-command.

What happens if I rerun the play against a host that dropped mid-task?

That depends entirely on the task that was in flight, and the honest process is to work out which one it was before deciding.

The steps, in order:

Find the interrupted task. The leftover ansible-tmp-* directory names the module. The controller log gives you the last task that completed.
Classify that task. Idempotent, or not? A copy, template, file or well-written package task is safe. A command, shell, script or raw task is safe only if you can show its payload is.
Check for a stuck resource. A half-finished package transaction, a held lock, a stale PID file, a partially written file that another process is now reading.
Only then rerun, and rerun --limit-ed to that host, in --check --diff first if the play supports it, so you see what the rerun believes it needs to do before it does it.

Step 4 without steps 1 to 3 is the move that converts a recoverable partial failure into a genuine incident, and it is the most common mistake in this whole part because it is the one that feels like doing something.

Knowledge check

Why this matters

This is the strongest argument this course makes for idempotency, and it is deliberately not an aesthetic one.

Every playbook review that ends with “it works, ship it” is implicitly betting that no connection will ever drop mid-task. At three hundred hosts and forty minutes per run, that bet loses regularly — not because your network is bad, but because forty minutes multiplied by three hundred hosts is a lot of opportunities for a switch to reboot, a firewall to reload, or a bastion to hit its connection limit.

When it loses, the difference between a five-minute recovery and a two-day reconciliation is a property you decided about weeks earlier, while writing tasks, when nothing was on fire and it seemed like a matter of taste. A play made of modules with declarative state is one --limit away from being fixed. A play made of shell commands is a list of things you now have to go and check by hand, one host at a time, starting at 03:00.

What Ansible knows, and what it does not

This is why idempotency is a safety property

What the host looks like afterwards

async with poll: 0, and its own failure modes

Knowledge check

`async` with `poll: 0`, and its own failure modes