Runbook: Troubleshoot unreachable hosts

“Unreachable” is Ansible telling you it never got far enough to run anything. That is a narrower statement than it sounds, and the whole value of this runbook is in taking it literally: the failure is somewhere between the controller’s network stack and the remote Python interpreter, and there are five distinct layers in that gap.

Diagnose them in order. Skipping to “it must be the key” is how a DNS change costs an hour.

When to use this runbook

• A playbook recap shows unreachable=1 or more.
• A run exits 4.
• A host that was managed yesterday cannot be reached today.
• A newly onboarded host never connected in the first place.

Blast radius

None. Every step in this runbook is read-only until you apply a fix, and the fixes are called out separately.

Step 1: Read the recap properly

ansible-playbook -i inventories/production site.yml --limit web
echo "exit=$?"

PLAY RECAP
web01.example.com : ok=4 changed=1 unreachable=0 failed=1
web02.example.com : ok=0 changed=0 unreachable=1 failed=0

web01 connected fine and a task failed - that is a different runbook. web02 was never reached. Only the second is this runbook’s problem, and the distinction is the first thing to establish because the two have no diagnostic steps in common.

Exit codes, verified on 2.21.3:

Unreachable takes precedence. A CI gate that tests only for 2 treats a run containing both as a pass.

Step 2: Establish the scope

This single question halves the search space.

ansible all -m ping -o | tee reachability.txt
grep -c SUCCESS reachability.txt
grep -c UNREACHABLE reachability.txt

The “everything” case is the one where instinct is least useful. Twenty hosts do not fail simultaneously; one controller does. Check the controller first and you will usually be done in two minutes.

Step 3: Resolve what Ansible is actually connecting to

Do not assume the inventory hostname is the connection target.

ansible-inventory -i inventories/production --host web02.example.com

Read ansible_host, ansible_user, ansible_port, ansible_ssh_common_args and ansible_ssh_private_key_file. A host whose ansible_host points at an address that was re-allocated last month fails in a way that looks exactly like a dead machine.

ansible web02.example.com -m ping -vvvv 2>&1 | grep -E 'ESTABLISH|SSH: EXEC|ansible_host'

-vvvv prints the literal ssh command line Ansible builds, including every option. That line is the ground truth, and it is what you should run by hand in the next step - not an approximation of it.

Step 4: Test the transport by hand, in layers

HOST=192.0.2.12

# 1. Name resolution - only if you are connecting by name
getent hosts web02.example.com

# 2. TCP reachability to the SSH port
timeout 5 bash -c "cat < /dev/null > /dev/tcp/$HOST/22" && echo 'tcp open' || echo 'tcp blocked'

# 3. SSH handshake and authentication, as the automation account
sudo -iu ansible ssh -vv -o BatchMode=yes -o ConnectTimeout=10 "ansible@$HOST" true

# 4. Remote interpreter
sudo -iu ansible ssh -o BatchMode=yes "ansible@$HOST" 'command -v python3; python3 -V'

Each layer fails differently, and the message tells you which one you are on:

Verified message on 2.21.3 for a filtered address, using documentation-range hosts:

web01.example.com | UNREACHABLE!: Task failed: Failed to connect to the host
via ssh: ssh: connect to host 192.0.2.11 port 22: Connection timed out

BatchMode=yes in step 3 is important. Without it, SSH may fall back to a password prompt and you will conclude authentication works when what worked was you typing a password that automation does not have.

Step 5: Group-wide failures - check the shared thing

# Is the bastion itself reachable?
ssh -o BatchMode=yes -o ConnectTimeout=10 ansible@bastion.example.com true

# What proxy configuration is in play?
ansible-inventory -i inventories/production --host web02.example.com \
| grep -iE 'proxy|common_args'

# Does the controller still have the key loaded?
sudo -iu ansible ssh-add -l

Step 6: Single-host failures - check the host

If TCP is open and SSH refuses, the host is up and something on it is wrong.

# Is sshd running and listening where you expect?
systemctl status ssh sshd 2>/dev/null
ss -lntp | grep ':22'

# Full disk breaks authentication in ways that look like a key problem
df -h / /var /home

# Is the automation account usable?
getent passwd ansible
passwd -S ansible

# Does sshd read keys from where you think?
sudo sshd -T -C user=ansible,host=web02.example.com,addr=192.0.2.12 \
| grep -iE 'authorizedkeysfile|authorizedkeyscommand|allowgroups|allowusers'

A full /home is a favourite: sshd cannot write to the account’s directory, key authentication fails, and the message is Permission denied (publickey) - which sends everyone to look at the key.

Step 7: Confirm the fix with something that connects

ansible web02.example.com -m ping -o
echo "exit=$?"

# Then the whole group, and compare the count
ansible web -m ping -o | grep -c SUCCESS
ansible-playbook -i inventories/production site.yml --limit web --list-hosts \
| grep -c 'example.com'

Both counts must match. Verifying one host proves one host; the failure you are recovering from was probably not limited to it.

Step 8: Record which layer failed

The useful incident note is one line: which layer. “web02 was unreachable, cause: DNS record removed during the subnet migration” is something the next person can act on. “web02 was flaky, resolved itself” is how the same hour gets spent again next month.

Rollback

Diagnosis changes nothing. The fixes need cleaning up:

Anything done under pressure is temporary until it exists in the repository. An estate accumulates these silently, and the next rebuild does not reproduce any of them.

Common patterns

Escalation

Escalate when:

• TCP fails from the controller and succeeds from elsewhere. That is a network path problem.
• The host is up but sshd is not, or the disk is full. That is the host owner’s.
• A host key changed and nobody can account for it. That is security’s, immediately.
• The whole fleet went unreachable at once and the controller checks out.