Diagnosing transport failures · Ansible

There is one habit in this part that resolves more problems than everything else combined, and it takes thirty seconds:

Before you change anything in Ansible, reproduce the failure with plain ssh.

Ansible’s ssh connection plugin is a wrapper around the ssh command line. If ssh cannot connect, Ansible cannot connect, and no amount of inventory variables, ansible.cfg edits or retry loops will change that. Half the time spent on “Ansible connection problems” is spent debugging Ansible for a fault that lives entirely in SSH configuration, DNS, firewalling or credentials.

The two-step separation

Step 1: get the exact command Ansible ran

Run the failing task at -vvv and read the SSH: EXEC line. It is the literal command, with every option Ansible assembled.

Read-only / Safethe connection Ansible actually attempted — executed on ansible-core 2.21.3— 192.0.2.11 is a documentation-range address that cannot be reached. The value is the SSH: EXEC line, which is the exact ssh invocation including every option in effect.

$ ansible-playbook -i inv.ini one.yml -vvv

<192.0.2.11> ESTABLISH SSH CONNECTION FOR USER: None
<192.0.2.11> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s
-o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,
gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10
-o 'ControlPath="/home/ops/.ansible/cp/a87e32107b"' -o NumberOfPasswordPrompts=1
192.0.2.11 '/bin/sh -c '"'"'echo ~'"'"''
<192.0.2.11> (255, b'', b'ssh: connect to host 192.0.2.11 port 22: Connection
timed out')

Four things are readable directly from that line and each is a common cause of a surprise:

FOR USER: None — no ansible_user and no remote_user is set, so ssh will use whatever the controller user’s SSH config says. That is a different user on a CI runner than on your laptop.
ConnectTimeout=10 — the Ansible timeout setting, default 10, handed to ssh.
ControlPath=... — the multiplexing socket. A stale one here produces failures that look like network faults.
The target is an address, not a name — because ansible_host was set. If you expected a name to be resolved, this is where you find out it was not.

Step 2: run it yourself, verbosely

Read-only / Safereproduce outside Ansible— Copy the target and options from the SSH: EXEC line, add -vvv, and replace the remote command with something harmless. This opens a connection and runs true on the remote host; it changes nothing.

ssh -vvv -o ConnectTimeout=10 ops@app-047.example.com true

# If that works and Ansible does not, add Ansible's options one at a time
ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s \
-o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey \
-o PasswordAuthentication=no \
ops@app-047.example.com true

The result tells you which half to debug

`ssh` says	The fault is	Where to look
Same failure	Not an Ansible problem	DNS, routing, firewall, `sshd`, credentials, host keys
Connects fine	An Ansible problem	The options Ansible added, the user it chose, the inventory variables

That is the whole value of the habit. It converts an open-ended question into one of two much smaller ones, and it costs one command.

Getting SSH debug output from inside Ansible

When you cannot easily reproduce by hand — the credential is only on the controller, or the failure only happens under load — ask the connection plugin for it. Note that this is a separate option from the Ansible verbosity level, and defaults to 0:

Read-only / Safessh CLI debugging, requested explicitly — executed on 2.21.3— ANSIBLE_SSH_VERBOSITY=3 appends -vvv to the ssh command line. The OpenSSH debug output comes back inside the connection error. Target is a documentation-range address.

$ ANSIBLE_SSH_VERBOSITY=3 ansible-playbook -i inv.ini one.yml -vvv

<192.0.2.11> SSH: EXEC ssh -vvv -C -o ControlMaster=auto ...
<192.0.2.11> (255, b'', b'debug1: OpenSSH_10.2p1, OpenSSL 3.5.5
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname 192.0.2.11 is address
debug1: auto-mux: Trying existing master at "/home/ops/.ansible/cp/a87e32107b"
debug1: Control socket does not exist
debug1: Connecting to 192.0.2.11 [192.0.2.11] port 22.
debug1: connect to address 192.0.2.11 port 22: Connection timed out')

Set it as an inventory variable on the affected host or group (ansible_ssh_verbosity=3) rather than globally. Forty lines per connection attempt across 300 hosts is a log nobody reads.

The settings that matter, with their defaults

Verified against ansible-core 2.21.3 with ansible-doc and ansible-config list:

Setting	Default	What it governs
`ssh_args`	`-C -o ControlMaster=auto -o ControlPersist=60s`	Passed to every SSH CLI tool. Compression and multiplexing
`ssh_common_args`	empty	Extra args, appended to the defaults
`timeout` (`DEFAULT_TIMEOUT`)	`10`	Becomes `ConnectTimeout` on the ssh command line
`PERSISTENT_COMMAND_TIMEOUT`	`30`	Wait for a response over a persistent connection
`PERSISTENT_CONNECT_TIMEOUT`	`30`	How long a persistent connection stays idle before destruction
`HOST_KEY_CHECKING`	`true`	Whether the underlying plugin verifies host keys
`ansible_ssh_verbosity`	`0`	Verbosity passed to the `ssh` CLI itself

Host key verification

HOST_KEY_CHECKING defaults to true, and the frequent advice to set it false is advice to disable a security control that exists to detect exactly the situation you are probably in.

A host key mismatch has three explanations: the host was rebuilt, you are connecting to a different machine than you think, or someone is intercepting the connection. The first is routine and the third is an incident, and turning off checking makes them indistinguishable permanently.

Read-only / Saferesolving a host key change deliberately— ssh-keygen -F queries the known_hosts file and prints the matching entry; ssh-keyscan fetches the key the host is currently presenting. Both read; neither modifies known_hosts. Compare the fingerprints before deciding anything.

# What do we have recorded?
ssh-keygen -F app-047.example.com

# What is the host presenting now?
ssh-keyscan -t ed25519 app-047.example.com 2>/dev/null | ssh-keygen -lf -

# Compare with the key recorded at build time, from the build system,
# not from the host you are currently talking to.

The last comment is the whole discipline. Accepting a new key because the host presented it is not verification; it is agreement. The ansible.builtin.known_hosts module exists so that key distribution can be a managed process rather than a prompt, and the SSH part of this course covers it in full.

The laptop-versus-controller difference is usually the agent

A play works from an engineer’s workstation and fails from the controller. The difference is almost always one of three things, and the first is by far the most common.

Agent forwarding. The workstation has a loaded ssh-agent and the key never leaves it. The controller has no agent, or a different one, or a systemd service with no SSH_AUTH_SOCK in its environment.

Read-only / Safeprove what the agent has, from the machine that is failing— Three read-only checks. ssh-add -l lists loaded identities; the environment variable check shows whether an agent is reachable at all from this context.

echo "SSH_AUTH_SOCK=${SSH_AUTH_SOCK:-none}"
ssh-add -l || echo 'no agent, or no identities loaded'
ssh -vvv -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 ops@app-047.example.com true \
2>&1 | grep -E 'Offering|Authentications that can continue|Server accepts'

A different user. FOR USER: None in the SSH: EXEC line means the user comes from the controller’s SSH config or the login name, which differs between a person’s laptop and a service account.

A different ansible.cfg. Covered in the previous lesson — ansible-config dump --only-changed on both machines, and check the first line for CONFIG_FILE() = None.

Under the hood

Why does a stale control socket produce failures that look like network faults?

ControlMaster=auto with ControlPersist=60s means the first connection to a host opens a master session that stays alive for sixty seconds after its last use, and subsequent connections multiplex over it. That is what makes a run of forty tasks against three hundred hosts finish in a reasonable time: one handshake per host rather than forty.

The socket lives at the ControlPath, under ~/.ansible/cp/ by default, and Ansible hashes the connection parameters into the filename. If the master dies badly — the network drops, the controller is suspended, a process is killed — a socket file can remain that no process is listening on. The next ssh finds it, tries to use it, and fails in a way that has nothing to do with the target host being reachable.

Two consequences worth knowing. First, the fix is to remove the stale socket, and the diagnosis is the auto-mux: Trying existing master line in ssh -vvv output followed by a failure. Second, control paths have a length limit imposed by the Unix domain socket API — around one hundred characters — so a long home directory or a customised ControlPath template can produce a failure that only appears for hosts with longer names, which is a genuinely confusing symptom.

What happens if the bastion is the problem rather than the hosts?

Everything behind it goes unreachable at once, which looks exactly like a fleet-wide outage.

The distinguishing evidence is that hosts reachable by other paths are fine, and that the error appears at the first hop. With ProxyJump, ssh -vvv shows the connection to the jump host before the connection to the target, so the failure point is visible in the ordering.

The specific bastion failure that is worth naming is MaxStartups. sshd limits the number of concurrent unauthenticated connections, by default starting to drop them randomly beyond ten. A run with forks: 50 opens fifty connections to the bastion within a second, and roughly forty of them are refused — producing a random subset of hosts unreachable, differently on every run, which is a very good imitation of a flaky network.

The fix is not more retries. It is fewer simultaneous connections (forks), or a bastion configured for the concurrency you actually use, or connection multiplexing through a single persistent jump session.

Knowledge check

Why this matters

The single most valuable habit in this entire part is thirty seconds long: read the SSH: EXEC line, run it yourself with -vvv, and find out which half of the system is broken before touching either.

Without it, transport failures produce a characteristic kind of wasted evening — inventory variables changed one at a time, timeout raised, ControlPersist disabled, host_key_checking turned off, retries added, each change plausible, none of them addressing a firewall rule that was modified that afternoon. At the end of it the play works because someone eventually noticed the firewall, and the estate is left carrying five configuration changes that nobody can justify and nobody will remove.

Disabling host key checking is the one that lasts longest and costs most. It goes in during an incident, it stays because removing it might break something, and from then on the fleet cannot tell a rebuilt host from a substituted one.