Triaging unreachable hosts

The playbooks part of this course established what unreachable means as a result: a distinct outcome from failed, counted in its own recap column, and reported through its own exit code — 4, which takes precedence over the 2 that a task failure produces.

This lesson is about what to do when you have some.

The framing that makes this tractable: unreachable is a statement about the controller’s knowledge, not about the host’s health. It means “I could not establish a working session”. A perfectly healthy machine is unreachable if DNS is wrong. A machine that is on fire is not unreachable until it stops answering.

Read the error text; it is a taxonomy

Ansible passes through the SSH client’s own message, and the messages are precise. This is the highest-value habit in the part, and most people skim straight past it to the recap.

$ ansible-playbook -i inv.ini ping.yml

fatal: [web1.example.com]: UNREACHABLE! => {"changed": false,
"msg": "Task failed: Failed to connect to the host via ssh:
ssh: connect to host 192.0.2.10 port 22: Connection timed out",
"unreachable": true}

That specific string carries information. Compare it against its neighbours:

The three rows in bold are the ones that change the shape of an incident.

Connection refused versus Connection timed out is the single most useful distinction on the list, and they are constantly treated as the same thing. Refused means a TCP stack answered — the machine is up, on the network, and reachable; only sshd is missing. Timed out means nothing came back at all. One is a service problem on a live host; the other is a host or path problem. Confusing them sends you to the wrong team.

Permission denied and Host key verification failed mean the network is working perfectly. Connectivity was established, the protocol negotiated, and the far end made a decision. Anyone who responds to those by investigating firewalls is investigating something they have already proven is fine.

Separating controller-side failures from real ones

Some UNREACHABLE results never involved the network at all. ControlPath too long is the clearest example — the client refuses before sending a packet — but a broken ProxyJump, an unreadable key file, or an unparseable option all produce the same outcome.

The discriminator is fast:

# A controller-side error fails all hosts, and fails immediately -
# there is no ConnectTimeout to wait out.
time ansible -i inventory all -m ansible.builtin.ping --limit 'web1*'

A genuine network timeout takes ConnectTimeout seconds — ten, by default — to give up. A controller-side configuration error comes back instantly, and it comes back for every host, including ones you know are healthy. If your entire inventory went unreachable in under a second, stop looking at the fleet.

Then read the command line and run it yourself:

# 1. What did Ansible actually build?
ansible-playbook -i inventory site.yml --limit web1.example.com -vvvv \
| grep 'SSH: EXEC'

# 2. Run that same command by hand, as the account that runs the automation,
#    with SSH's own verbosity.
ssh -vvv -o ConnectTimeout=10 -o PasswordAuthentication=no web1.example.com 'echo ~'

If step 2 fails the same way, it is not an Ansible problem and reading your playbook is wasted time. If step 2 succeeds, the difference is between your environment and the one Ansible built — a variable resolving to an unexpected ansible_host, a group_vars file supplying an ansible_user you forgot about, an ansible_ssh_common_args from an inventory source you did not know was loaded.

Forty of four hundred: the judgement call

The recap says unreachable=40 out of 400. The run is half done. What now?

The instinct is to look at the number. The number tells you how bad it is; it does not tell you what it is. Correlation tells you what it is, and you can get it in one command.

# Which groups do the unreachable hosts belong to?
for h in $(cat unreachable.txt); do
ansible-inventory -i inventory --host "$h" | grep -o '"group_names".*'
done | sort | uniq -c | sort -rn

# What addresses are they, and do they share a subnet?
ansible-inventory -i inventory --list | \
python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["_meta"]["hostvars"])'

Four correlations, and each points somewhere different:

They share a routing group. Every unreachable host is behind the same bastion, and hosts reached by other paths are fine. This is a path event. Check the bastion before anything else — and if the number is varying between runs with a different subset each time, revisit the MaxStartups signature from the bastion lesson, because that is not an outage at all.

They share a subnet, rack or site. A network event. Your automation has detected it, which is genuinely useful information for somebody, but it is not your problem to fix from here.

They share a role or a build generation. Now it is interesting. All the database hosts, or everything built before a certain date, becoming unreachable together suggests a change that applied to that set — a configuration push, an image rollout, a firewall rule scoped to a role. Ask what else targets that same set.

No correlation at all. Scattered across groups, sites and roles. This is the shape of a controller-side or credential problem, or of a random drop like MaxStartups. Genuinely uncorrelated host failures are rare; scattered results usually mean the common factor is on your side.

Knowledge check