Unreachable is not failed · Ansible

The recap has separate columns for unreachable= and failed=, and the distinction is not cosmetic. It runs all the way through the failure machinery, and the previous two lessons kept bumping into it.

A failed host ran a task and the task returned an error. Something happened on the host, and Ansible knows what.

An unreachable host never ran anything. The connection could not be established, so there is no task result, no changed state, no information about the host at all beyond the fact that it did not answer.

The failure keywords were built for the first case. They do not respond to the second, and this lesson is about what that costs.

Neither keyword sees it

Both results are verified, and both are worth seeing side by side because the keyword names promise otherwise.

Read-only / Safemax_fail_percentage: 0 with one unreachable host in a three-host batch— Executed on ansible-core 2.21.3. u01 is 192.0.2.11, a documentation-range address that is unreachable by design; the other five hosts are local connections. The strictest possible threshold, and 33.3% of the batch did not answer.

$ ansible-playbook -i inv-unreach.ini rollout.yml

TASK [Touch the host] **********************************************************
ok: [h02]
ok: [h03]
fatal: [u01]: UNREACHABLE! => {"changed": false, "msg": "Task failed: Failed to
connect to the host via ssh: ssh: connect to host 192.0.2.11 port 22: Connection
timed out", "unreachable": true}

TASK [Return to service] *******************************************************
ok: [h02]
ok: [h03]

  ... every remaining batch ran to completion ...

PLAY RECAP *********************************************************************
h02 : ok=2 changed=0 unreachable=0 failed=0
h03 : ok=2 changed=0 unreachable=0 failed=0
h04 : ok=2 changed=0 unreachable=0 failed=0
h05 : ok=2 changed=0 unreachable=0 failed=0
h06 : ok=2 changed=0 unreachable=0 failed=0
u01 : ok=0 changed=0 unreachable=1 failed=0
exit=4

max_fail_percentage: 0 means “abort on any failure”. One host in three did not answer, and the rollout completed across the entire fleet.

any_errors_fatal: true does slightly more and still does not stop the run: verified on 2.21.3, it cut the batch containing the unreachable host — h02 and h03 never reached their final task and were dropped from the rest of the playbook — but later batches ran to completion and a subsequent play ran too.

Reading the recap

The recap is the authoritative account, and the two columns answer different questions.

Read-only / Safefour outcomes in one recap— A composite showing what each column means. The exit code reflects the worst outcome present, and unreachable takes precedence over failed.

$ ansible-playbook -i production.ini site.yml

PLAY RECAP *********************************************************************
web01  : ok=12  changed=3  unreachable=0  failed=0  skipped=2  rescued=0  ignored=0
web02  : ok=8   changed=1  unreachable=0  failed=1  skipped=2  rescued=0  ignored=0
web03  : ok=0   changed=0  unreachable=1  failed=0  skipped=0  rescued=0  ignored=0
web04  : ok=5   changed=2  unreachable=1  failed=0  skipped=0  rescued=0  ignored=0

Host	Reading
`web01`	completed the play, made three changes
`web02`	ran, something returned an error, stopped there
`web03`	never answered at all — `ok=0` — so nothing on it was touched
`web04`	ran five tasks, changed two things, then became unreachable

web04 is the row that matters. unreachable=1 with a non-zero ok= and changed= count means the host was reachable, was modified, and then went away — which is exactly what a reboot, a network reconfiguration or a service restart that took out the SSH path looks like.

A host that was changed and then stopped answering is the most urgent line in any recap. It is in an unknown state, the play did not finish its work there, and no failure keyword reacted to it.

Exit code 4 hides a task failure

Established by execution during this course’s environment validation and recorded in the validation notes: when a run contains both task failures and unreachable hosts, the exit code is 4, not 2. Unreachable takes precedence.

Run outcome	Exit code
Success	`0`
Task failure only	`2`
Unreachable host only	`4`
Task failure and unreachable host	`4`

A CI gate written as “fail the build if the exit code is 2” will pass a run that broke a host and lost contact with another. Test for non-zero, not for a specific code.

`ignore_unreachable`

The keyword exists to say “a host not answering is acceptable here”, and what it does is narrower and more useful than it first appears.

Read-only / Safeignore_unreachable: true on the connecting task — executed on 2.21.3— The same inventory. The unreachable host stays in the play and runs the subsequent task, and the recap reclassifies the outcome.

$ ansible-playbook -i inv-unreach.ini ignore.yml

TASK [Touch] *******************************************************************
fatal: [u01]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to
the host via ssh: ...", "unreachable": true, "skip_reason": "..."}
ok: [h02]
ok: [h03]

TASK [After] *******************************************************************
ok: [u01] => { "msg": "AFTER u01" }
ok: [h02] => { "msg": "AFTER h02" }
ok: [h03] => { "msg": "AFTER h03" }

PLAY RECAP *********************************************************************
u01 : ok=2  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=1

Two things changed, and the second is the one to notice.

The host stays in the play. Without the keyword, an unreachable host is removed from the active host list and runs nothing further. With it, the host continues to subsequent tasks — which is why u01 ran the After task.

The recap reclassifies it. unreachable=0, failed=0, ignored=1. The host no longer appears as unreachable at all.

That second point deserves suspicion. ignore_unreachable on a connecting task does not merely tolerate the condition; it erases the evidence of it from the summary. A reader of that recap sees a clean run with one ignored task.

Scope ignore_unreachable to the task, never to the play

There is one good use for this keyword: a task whose whole purpose is to tolerate absence. Probing whether a decommissioned host still answers. Attempting a graceful shutdown on a host that may already be down. Collecting diagnostics on a best-effort basis.

- name: Attempt a graceful drain, tolerating a host that is already gone
  ansible.builtin.command: /usr/local/bin/app-drain
  ignore_unreachable: true
  changed_when: false

Setting it at play level is a different thing entirely: it declares that this play does not care whether it reached any of its hosts. A rolling deploy with play-level ignore_unreachable will report success having touched nothing, because every host that did not answer was counted as ignored rather than missed.

If you find it at play level in a repository, treat it as a defect rather than a style choice.

Gating reachability before the rollout

Since no failure keyword will do it, the control has to run first, in a play that can see the whole fleet.

The removal behaviour that makes unreachable hosts invisible to the failure keywords is also what makes them easy to find: an unreachable host is dropped from ansible_play_hosts but remains in ansible_play_hosts_all. The difference between the two is exactly the set of hosts that did not answer.

Read-only / Safea reachability pre-flight— Read-only: ping connects and returns, changing nothing. Unbatched, so it sees the entire fleet at once. Verified on ansible-core 2.21.3 against an inventory containing one documentation-range address.

- name: Confirm the fleet is reachable before rolling anything
hosts: appservers
gather_facts: false
tasks:
  - name: Contact every host
    ansible.builtin.ping:

  - name: Refuse to roll if any host did not answer
    ansible.builtin.assert:
      that: missing | length == 0
      fail_msg: >-
        Did not answer: {{ missing | join(", ") }}. Establish why before
        rolling a change; a rollout will proceed past unreachable hosts
        and leave them on the old version.
      success_msg: 'All {{ ansible_play_hosts_all | length }} hosts answered'
    vars:
      missing: '{{ ansible_play_hosts_all | difference(ansible_play_hosts) }}'
    run_once: true

- name: Roll the release
hosts: appservers
become: true
serial:
  - 1
  - 5
  - 25%
max_fail_percentage: 0
tasks:
  - name: Deploy and verify
    ansible.builtin.include_tasks: deploy-one-host.yml

Read-only / Safethe pre-flight, executed both ways on 2.21.3— The same play against an inventory with one unreachable documentation-range host, and against one where every host answers.

$ ansible-playbook -i inv-unreach.ini preflight.yml

TASK [Contact every host] ******************************************************
ok: [h02]
ok: [h03]
ok: [h04]
ok: [h05]
ok: [h06]
fatal: [u01]: UNREACHABLE! => {"changed": false, "msg": "Task failed: Failed to
connect to the host via ssh: ssh: connect to host 192.0.2.11 port 22: Connection
timed out", "unreachable": true}

TASK [Refuse to roll if any host did not answer] *******************************
fatal: [h02]: FAILED! => {
  "assertion": "missing | length == 0",
  "msg": "Did not answer: u01"
}

--- control run, all six hosts reachable ---

TASK [Refuse to roll if any host did not answer] *******************************
ok: [h01] => { "msg": "All 6 hosts answered" }

Three things make this work, and each of them is a decision:

No ignore_unreachable on the probe. The default removal is the mechanism the check depends on: a host that answers stays in ansible_play_hosts, a host that does not is dropped, and the set difference is the answer. Adding ignore_unreachable here would keep the unreachable host in the list and the assertion would always pass.

ansible_play_hosts_all as the baseline, because it is the only variable unaffected by hosts dropping out.

A separate, unbatched play. A fleet-wide precondition cannot live inside the rolling play, because there it would only ever see one batch.

Note which host reports the assertion failure in the verified output: h02, not u01. run_once executes on the first active host, and u01 was no longer among them.

Why a blip looks like a fleet-wide failure

The diagnostic confusion this lesson exists to prevent: a transient network problem produces a wall of red UNREACHABLE! lines that looks identical to a catastrophic failure, and the instinct is to assume the change caused it.

Three signals separate the two, all available in the run output:

ok=0 across the affected hosts. A host that was never contacted ran nothing. If the unreachable hosts all have ok=0, the change cannot have caused their state, because it never reached them. Compare web04 above, where ok=5 changed=2 unreachable=1 says the opposite.

The error text. Connection timed out is a network path problem. Permission denied (publickey) is an authentication problem and probably a change you made. Host key verification failed means the host identity changed — a rebuild, or something worse.

The distribution. Unreachable hosts clustered in one rack, one subnet or one availability zone is infrastructure. Unreachable hosts scattered evenly across the fleet, appearing right after a batch that changed the SSH configuration, is you.

Under the hood

Unreachable is not a task result. It is a connection-layer outcome, recorded before any module has been shipped to the host or executed.

Ansible keeps two separate records per play: hosts that have failed, and hosts that are unreachable. The documentation describes the effect for the second — the host is “removed from the list of active hosts for the run” — and that removal is what makes it invisible to the failure machinery. max_fail_percentage computes its ratio from the failed record only. any_errors_fatal is raised by a failing task result, and there is no task result to raise it.

ignore_unreachable intervenes at the point of that removal: the host is kept in the active list and the outcome is recorded as ignored. The recap counter it increments is ignored, which is the same counter ignore_errors uses — which is why an unreachable host and a host with a tolerated task failure become indistinguishable in the summary line.

meta: clear_host_errors operates on the same records from the other direction, restoring hosts that have been removed so that later tasks run against them again. Upstream names it as the way to reactivate hosts marked unreachable. It is a genuine repair tool for a play that expects transient loss — and a way to discard a signal you should have acted on, depending entirely on why the hosts went away.

What happens if a host becomes unreachable because my play rebooted it?

Then unreachable is the expected outcome of a successful task, and the play has to be written to say so.

A reboot task that reports the host unreachable afterwards has not failed — it has worked. The pattern is to use a module that understands this rather than to tolerate the error:

Service impact possiblea reboot that waits for the host to return— The reboot module handles the disconnection and the reconnection itself, and fails only if the host does not come back within the timeout. This is a real reboot and belongs behind a rolling batch.

    - name: Reboot into the new kernel and wait for the host to return
    ansible.builtin.reboot:
      reboot_timeout: 600
      post_reboot_delay: 15
      test_command: systemctl is-system-running --wait

The failure mode to design against is the host that does not come back. Under serial: [1, 5, 25%] with max_fail_percentage: 0, a host that exceeds reboot_timeout produces a genuine task failure — not an unreachable — and that failure does stop the rollout. This is one of the few places where the distinction works in your favour, and it works because the module converts “did not come back” into a task result.

Writing the same thing as a shell: reboot plus a wait_for_connection gets you most of the way there. Writing it as shell: reboot with ignore_unreachable: true and no wait gets you a rollout that reboots the entire fleet without ever confirming a single host returned.

Knowledge check

Why this matters

Every safety mechanism in this part responds to the same signal: a task ran and returned an error. That is a good signal and it covers the failure everyone plans for — the change was wrong, and the host said so.

It does not cover the failure that happens during an incident, which is that hosts stop answering. And the mechanisms do not merely fail to help; they carry on, batch after batch, applying the change to a shrinking fleet while reporting nothing that looks like a problem. The exit code even changes in a way that can defeat a CI gate written by somebody who tested it against a task failure.

The practical instruction is short. Gate reachability before you roll, in a play that can see the whole fleet. Read the unreachable= column of every recap, and treat a host with a non-zero changed= beside it as the first thing you investigate. And test CI gates for non-zero exit codes, because the specific value you were expecting is not the one you will get on the worst night.