Why unreachable hosts are the dangerous ones · Ansible

The thirty failures got the attention. They printed error messages, they have stack traces, they are interesting.

The fifty unreachable hosts printed one line each and then stopped existing as far as the run was concerned. They are the ones that will still be a problem in a month.

An unreachable host is a host you have not changed

That sentence is the whole lesson, and everything else is consequences of it.

The fifty are not a reporting problem. They are fifty production machines running the configuration you decided last night was no longer acceptable. Nothing about the run altered them. Nothing about the run put them on a list. When the run finished and the exit code came back, they went straight back to being ordinary production hosts that nobody is looking at — running the old config, with the old package versions, without the security fix.

The next time anyone notices them is when something depends on the change having landed everywhere. That is not a hypothetical: it is the standard shape of the incident. A certificate rotation misses forty hosts; six weeks later the old certificate expires; forty hosts fail simultaneously and the change record says the rotation completed.

The fifty are three different populations

Before deciding anything, split them.

Off, or genuinely down. The host is powered off, decommissioned, or crashed. Consistent, stale, harmless right now. The action is to find out whether it should exist at all — a stable population of unreachable hosts is usually an inventory that has drifted from reality.

Reachable, but not by you, right now. DNS is broken, a firewall rule changed, the bastion is saturated, your key was rotated out, the host key changed. The host is up and serving traffic. It is running the old config while looking perfectly healthy to everything except Ansible. This is the largest bucket in most real incidents and the most dangerous, because monitoring says the host is fine.

Unreachable because of the change. This is the one that turns a partial failure into an outage. The play reconfigured the firewall, restarted the network, rotated an SSH key, changed sshd_config, or rebooted — and the host is now unreachable as a direct result of a task that succeeded.

The pre-flight check that proves nothing

The natural response to fifty unreachable hosts is to add a reachability check before the next run. The natural way to write it is wrong, and this is worth executing because the failure is completely silent.

Read-only / Safea pre-flight built from debug, against two unreachable hosts — executed on ansible-core 2.21.3— web01 and web02 are 192.0.2.11 and 192.0.2.12, documentation-range addresses that cannot be reached. The play reports every host as ok and exits 0.

$ ansible-playbook -i inv.ini controller-only.yml ; echo exit=$?

PLAY [A pre-flight built from controller-side modules] *************************

TASK [Report in] ***************************************************************
ok: [lh] => {
  "msg": "reporting from lh"
}
ok: [web01] => {
  "msg": "reporting from web01"
}
ok: [web02] => {
  "msg": "reporting from web02"
}

PLAY RECAP *********************************************************************
lh       : ok=1  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0
web01    : ok=1  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0
web02    : ok=1  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0

exit=0

Both remote hosts are unreachable. Both report ok=1, both report unreachable=0, and the run exits 0. The play cheerfully printed “reporting from web01” for a host that does not answer.

The reason is that ansible.builtin.debug is an action plugin that executes on the controller. It never opens a connection, so it can never discover that a connection is impossible. The same is true of ansible.builtin.fail, ansible.builtin.set_fact, and assert where the condition needs no facts.

Read-only / Safethe same inventory, with a task that requires a connection — executed on 2.21.3— ansible.builtin.ping transfers a module to the managed node and runs it there, so it must connect. Same hosts, same inventory, opposite result.

$ ansible-playbook -i inv.ini unreach-only.yml ; echo exit=$?

PLAY [Unreachable only] ********************************************************

TASK [Something that needs a connection] ***************************************
fatal: [web01]: UNREACHABLE! => {"changed": false, "msg": "Task failed: Failed to
connect to the host via ssh: ssh: connect to host 192.0.2.11 port 22: Connection
timed out", "unreachable": true}
fatal: [web02]: UNREACHABLE! => {"changed": false, "msg": "Task failed: Failed to
connect to the host via ssh: ssh: connect to host 192.0.2.12 port 22: Connection
timed out", "unreachable": true}

PLAY RECAP *********************************************************************
web01    : ok=0  changed=0  unreachable=1  failed=0  skipped=0  rescued=0  ignored=0
web02    : ok=0  changed=0  unreachable=1  failed=0  skipped=0  rescued=0  ignored=0

exit=4

Read-only / Safea pre-flight play that actually establishes the target set— Runs before the change play in the same playbook. It connects, so unreachable hosts are detected; it changes nothing on any host. The output is a written list, which is the artefact the rest of this lesson is about.

- name: Establish which hosts are actually reachable
hosts: appservers
gather_facts: false
any_errors_fatal: false
tasks:
  - name: Prove transport, auth and remote Python
    ansible.builtin.ping:
    ignore_unreachable: true
    register: reachability

  - name: Record the hosts that did not answer
    ansible.builtin.copy:
      content: "{{ ansible_play_hosts_all
                   | difference(ansible_play_hosts)
                   | join('\n') }}\n"
      dest: ./artifacts/unreachable-{{ ansible_date_time.iso8601_basic_short
                                        | default('run') }}.txt
      mode: '0644'
    delegate_to: localhost
    run_once: true

Note what this play does with ignore_unreachable: it uses it once, in a play whose only purpose is to measure, and then writes the result down. That is the only defensible use of the keyword, and the next section is about what happens when it escapes into a change play.

`ignore_unreachable` converts an unknown into a silent one

The keyword exists, it is tempting, and at fleet scale it is close to indefensible. Verified on 2.21.3:

Read-only / Safeignore_unreachable: true at play level, against unreachable hosts — executed on 2.21.3— Same two documentation-range addresses. The play has a ping task and then a debug task announcing that the change was applied. Read the recap and the exit code together.

$ ansible-playbook -i inv.ini ignunreach.yml ; echo exit=$?

TASK [Connect and report] ******************************************************
fatal: [web01]: UNREACHABLE! => {"changed": false, "msg": "...Connection timed
out", "unreachable": true}
...ignoring

TASK [Say we are done] *********************************************************
ok: [web01] => {
  "msg": "change applied to web01"
}
ok: [web02] => {
  "msg": "change applied to web02"
}

PLAY RECAP *********************************************************************
lh       : ok=2  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0
web01    : ok=2  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=1
web02    : ok=2  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=1

exit=0

Read that recap carefully. For a host that was never contacted:

unreachable=0. The counter that exists to report this condition reports zero.
ok=2. Higher than the number of tasks that could possibly have run on it.
ignored=1 — the only trace, and it is in the column nobody watches.
The play printed “change applied to web01” for a host it never reached.
The run exited 0.

There is a narrower, legitimate use: a task that is expected to make a host unreachable, such as a reboot, followed by wait_for_connection. There the unreachability is the intended outcome of a specific task and the following task proves recovery. That is a scoped ignore_unreachable on one task, not a play-level policy, and it is the shape to insist on in review.

Under the hood

Why does the debug play report ok for an unreachable host at all?

Ansible plugins are split by where they execute. Modules are Python code transferred to the managed node and run there; action plugins run on the controller and may or may not then invoke a module. Several common ansible.builtin entries are action plugins with no module behind them: debug, fail, set_fact, add_host, include_vars, assert, pause, meta.

For those, the execution path never touches the connection plugin. No connection is attempted, no connection error can be raised, and the result object handed back to the strategy is an ordinary success. The per-host counters are updated from that result exactly as they would be for a real module, which is why the host is credited with ok.

This caught the preparation of this course itself. The first attempt to establish the exit-code table used a play built from fail, observed unreachable=0 across the board, and concluded that task failure took precedence over unreachability. The conclusion was wrong; the run had never contacted anything. The unreachable=0 in the recap was the evidence that exposed it.

The lesson generalises past this keyword: read the recap, do not infer it. A play that produced the numbers you expected for a reason you did not check is not evidence.

The quarantine list is the required output

An unreachable population is not resolved by understanding it. It is resolved by a list that outlives the incident.

The list needs four fields per host, and the fourth is the one that makes it work:

Field	Why
Hostname	Obvious, and must come from the run artefact, not from memory
Error text	The taxonomy: timed out, refused, permission denied, host key mismatch
The change it did not receive	So the reconciliation is specific, not “run the playbook again some day”
Owner and a date	An entry with no owner is a note; an entry with an owner is work

Read-only / Safeturning the run artefact into a quarantine list— Extracts hostname and error text from the run log into a CSV that a human then adds owner and date to. Reads a log file; connects to nothing.

log=/var/log/ansible/patch-2026-08-11.log

printf 'host,error,change,owner,due\n' > quarantine-2026-08-11.csv
grep 'UNREACHABLE!' "$log" \
| sed -E 's/^fatal: \[([^]]+)\].*"msg": "([^"]+)".*/\1,"\2"/' \
| sed 's/$/,patch-2026-08-11,,/' \
>> quarantine-2026-08-11.csv

wc -l quarantine-2026-08-11.csv

What happens if a host is unreachable for three consecutive runs?

It has stopped being a connectivity problem and become an inventory problem.

Three misses means the host has not received three changes. If those changes were cumulative — a package upgrade, then a config change that assumes the new package, then a service change that assumes the config — then the host cannot simply be caught up by rerunning the latest play. The plays were written assuming the previous ones had landed.

This is the argument for tracking consecutive misses rather than per-run unreachability. A host that misses one run needs a retry. A host that has missed three needs a decision: repair it, rebuild it, or remove it from the inventory. All three are fine. Leaving it in the inventory, unreachable, contributing a line to every recap that everyone has learned to scroll past, is not.

Knowledge check

Why this matters

Failures are loud and unreachable hosts are quiet, and operational attention follows noise. That is the bias this lesson exists to correct.

The thirty failed hosts will be dealt with tonight, because they are in front of you and they have error messages. The fifty unreachable hosts will be dealt with tonight only if someone writes them down, because nothing else in the system will ever raise them again. The recap scrolled past. The exit code was consumed by a wrapper. The next run will produce its own fifty, which may or may not be the same fifty.

An unreachable host is not an error. It is a host that did not get the change, and the whole point of running the change was that the old state was no longer acceptable. Leaving it out is the same decision as deciding not to make the change on that host — made silently, by a timeout, at 02:30.