What happens on the wire during a run

Ansible has no network layer.

That sentence is worth sitting with, because a great deal of confused troubleshooting follows from assuming otherwise. There is no Ansible protocol, no Ansible daemon on the far end, and no Ansible socket. When a task runs against a Linux host, Ansible assembles a command line, executes your system ssh binary with it, and reads what comes back.

Everything that is true of ssh on your controller is therefore true of your automation: its version, its ssh_config, its known_hosts, its agent, its ciphers, its ProxyJump, and its bugs. And the reverse holds too — the majority of what gets reported as “Ansible is broken” is an SSH condition that Ansible faithfully reported and nobody read.

This part is about that layer. This lesson establishes what is actually executed, so the rest of the part has something concrete to modify.

What ansible-core ships

Start by narrowing the field. The transport is pluggable, but the set of plugins in core is small:

$ ansible-doc -t connection -l

ansible.builtin.local execute on controller
ansible.builtin.psrp  Run tasks over Microsoft PowerShell Remoting Protocol
ansible.builtin.ssh   connect via SSH client binary
ansible.builtin.winrm Run tasks over Microsoft's WinRM

Four. Two of them are Windows transports and one runs on the controller itself. For a Linux fleet there is exactly one plugin, and its own description states the mechanism plainly: connect via SSH client binary.

DEFAULT_TRANSPORT is ssh by default, so on a Linux fleet you get the right plugin without asking for it.

The command line Ansible actually builds

-vvvv prints it. This is the single most useful diagnostic in the whole part, and it is entirely read-only:

$ ansible-playbook -i inv.ini ping.yml -vvvv

<192.0.2.10> ESTABLISH SSH CONNECTION FOR USER: None
<192.0.2.10> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s
-o KbdInteractiveAuthentication=no
-o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
-o PasswordAuthentication=no -o ConnectTimeout=10
-o 'ControlPath="/home/ebrandi/.ansible/cp/fa537f329b"'
-o NumberOfPasswordPrompts=1 192.0.2.10 '/bin/sh -c '"'"'echo ~'"'"''

The line is wrapped here for the page; Ansible prints it as one line.

Every option on it comes from somewhere you can find and change. Account for them one at a time, because “where did that come from” is a question you will ask under pressure:

That last element surprises people. The first SSH command of a run is not your task. It is echo ~, because Ansible needs to know where to put a temporary directory before it can send a module anywhere. A run that fails at echo ~ has failed before any of your automation was involved, which is a useful thing to be able to say out loud in an incident.

The defaults that matter, and what they actually control

$ ansible-config dump -t connection

ssh:
___
control_path(default) = None
control_path_dir(default) = ~/.ansible/cp
host(default) = inventory_hostname
host_key_checking(default) = True
pipelining(default) = False
reconnection_retries(default) = 0
scp_extra_args(default) =
sftp_extra_args(default) =
ssh_args(default) = -C -o ControlMaster=auto -o ControlPersist=60s
ssh_common_args(default) =
ssh_executable(default) = ssh
ssh_extra_args(default) =
ssh_transfer_method(default) = smart
timeout(default) = 10
use_tty(default) = True

Output trimmed to the settings this part uses.

timeout is a connect timeout, not a task timeout

It becomes ConnectTimeout=10 on the command line. It bounds how long the client waits to establish a TCP connection and complete the handshake. It places no bound whatsoever on how long your task takes once connected — a package upgrade may run for twenty minutes under a timeout of 10.

The confusion is expensive in the wrong direction. Somebody whose long task is being killed raises timeout, nothing improves, and they raise it further, at which point the only thing that has changed is that genuinely dead hosts now take ten times as long to be declared dead. Ten seconds is short for a congested WAN and generous for a datacentre LAN; treat it as a network-topology setting.

reconnection_retries defaults to 0, and retries only one thing

$ ansible-doc -t connection ansible.builtin.ssh

   reconnection_retries  Number of attempts to connect.
                       Ansible retries connections only if it gets
                       an SSH error with a return code of 255.
                       Any errors with return codes other than 255
                       indicate an issue with program execution.

Return code 255 is what ssh returns for its own errors: refused connection, timeout, host key mismatch, bad option. Anything else is the remote command’s exit status and is not a connection problem, so retrying it would just run your task again.

This is a well-drawn boundary, but it has a sharp edge. A misconfigured ControlPath also produces 255 — so does an unparseable ProxyCommand — which means retries will patiently repeat a deterministic controller-side configuration error several times before failing. Retries buy you resilience against a flaky network. They buy you nothing against a broken config, and they multiply the time it takes to find out.

ssh_transfer_method and the OpenSSH 9 scp trap

$ ansible-doc -t connection ansible.builtin.ssh

   ssh_transfer_method  Preferred method to use when transferring
                      files over ssh
      choices:
        piped: Creates an SSH pipe with 'dd' on either side to copy the data.
        scp: Deprecated in OpenSSH. For OpenSSH >=9.0 you must add an
          additional option to enable scp 'scp_extra_args="-O"'.
        sftp: This is the most reliable way to copy things with SSH.
        smart: Tries each method in order (sftp > scp > piped), until one
          succeeds or they all fail.
      default: smart

The default smart tries sftp first, then scp, then piped. Note the order — sftp is not a fallback here, it is the preferred method, and the documentation calls it the most reliable.

The trap belongs to scp. OpenSSH 9.0 changed scp to use the SFTP protocol underneath, and the legacy behaviour needs -O. If you have pinned ssh_transfer_method: scp — usually inherited from an old ansible.cfg written against a target that had no SFTP subsystem — then on a modern client you also need scp_extra_args: "-O". Pinning scp in 2026 without that flag is a slow-motion failure waiting for the day the controller gets upgraded.

If a target genuinely has no SFTP subsystem, piped is the better pin: it needs only dd on both ends and a shell.

Where these settings can be set

Three places, and they are not interchangeable. From ansible-doc, taking pipelining as the example, the plugin declares env, ini and vars entry points:

        set_via:
          env:
          - name: ANSIBLE_PIPELINING
          - name: ANSIBLE_SSH_PIPELINING
          ini:
          - key: pipelining
            section: defaults
          - key: pipelining
            section: connection
          - key: pipelining
            section: ssh_connection
          vars:
          - name: ansible_pipelining
          - name: ansible_ssh_pipelining

The vars entry is the one that matters architecturally, because it is the only one that can differ per host or per group.

• ansible.cfg sets it for every host the controller ever touches.
• ANSIBLE_* sets it for one invocation.
• ansible_* in group_vars/ sets it for the group that needs it.

A fleet is rarely homogeneous at the transport layer. One group is behind a bastion, one group runs an old SSH server, one group is across a WAN with a longer round trip. Those are group properties, so they belong in group_vars/, not in a global config where they apply to hosts that do not need them and quietly mask the fact that the fleet is not uniform.

# inventory/production/group_vars/remote_dc.yml
# 250 ms RTT to this site; the 10 s default connect timeout is marginal
# during the backup window. Reviewed 2026-08-11.
ansible_ssh_timeout: 30

Note the comment. A transport override with no note is a value nobody currently owns, and the next person to read it will either delete it or copy it fleet-wide.

The discipline this part is built on

When a run fails at the connection, take the ssh command line out of the -vvvv output and run it yourself.

$ ssh -vvv -o ConnectTimeout=10 -o PasswordAuthentication=no 192.0.2.10 'echo ~'

Two outcomes, and they lead to completely different afternoons:

It fails the same way. This is not an Ansible problem. It is a network, credential, host key or sshd problem, and every minute spent reading your playbook is wasted. You now have a reproduction that a network engineer can act on without knowing what Ansible is.

It succeeds. Now it is an Ansible problem — a variable resolving to the wrong ansible_host, a group_vars file applying ansible_user you did not expect, an ansible_ssh_common_args from an inventory you forgot was loaded. The difference is between your environment and the environment Ansible built, and that is a much smaller search space.

Knowledge check