Connection reuse and why runs get slow

Ansible’s default ssh_args is not a neutral starting point. It is a performance decision that somebody made on your behalf, and it is load-bearing:

$ ansible-config dump -t connection | grep '^ssh_args'

ssh_args(default) = -C -o ControlMaster=auto -o ControlPersist=60s

-C is compression. The other two are connection reuse, and they are the single biggest connection-level factor in how long a fleet run takes.

What a run costs without reuse

An SSH connection is expensive relative to what a task does. TCP handshake, protocol version exchange, key exchange, host key verification, public key authentication, session setup. On a LAN that is tens of milliseconds; on a WAN, or through a bastion, several hundred; with a slow authentication backend on the far end, more.

Now count how many times a run needs one.

A play with 30 tasks against 200 hosts is 6,000 task executions. Without reuse, that is at least 6,000 connections — more, because file transfers for copy and template open their own sftp invocations, and fact gathering adds its own.

With ControlMaster=auto and ControlPersist=60s, the first connection to each host creates a master connection and leaves a control socket on the controller. Every subsequent ssh, sftp and scp to that host notices the socket and opens a new channel inside the existing session. Roughly 200 connections instead of 6,000, and the 5,800 you avoided were the expensive part.

Two properties follow, and both are worth stating because people consistently get the direction wrong:

The benefit scales with the number of tasks, not the number of hosts. Doubling the fleet doubles the work either way. Doubling the number of tasks in the play doubles the saving. A long role against a modest fleet benefits more than a one-task run against a huge one.

The benefit scales with round-trip time. On a datacentre LAN the handshake is cheap and losing multiplexing costs you tens of percent. Across a WAN or through a bastion it can cost you a factor of five, and that is where the eight-minutes-becomes-fifty scenario lives.

The control path, and the failure it used to cause

$ ansible-config dump -t connection | grep -E '^control_path'

control_path(default) = None
control_path_dir(default) = ~/.ansible/cp

None is not “no control path”. From the plugin documentation:

Since 2.3, if null (default), ansible will generate a unique hash. Use %(directory)s to indicate where to use the control dir path setting. Before 2.3 it defaulted to control_path=%(directory)s/ansible-ssh-%%h-%%p-%%r.

Which is what you see in a real command line — a short hash, not a template:

-o 'ControlPath="/home/ebrandi/.ansible/cp/fa537f329b"'

The old template embedded the hostname, port and username. A Unix domain socket path has a hard length limit, and long fully-qualified hostnames under a long home directory used to blow past it routinely. The hash cannot, because it is fixed length.

The failure has not disappeared, though — it is waiting for anyone who sets control_path back to a template, which people still do because they copied it from a blog post that predates 2.3. Reproduced on 2.21.3:

$ ANSIBLE_SSH_CONTROL_PATH='%(directory)s/ansible-ssh-%%h-%%p-%%r' ansible-playbook -i inv.ini ping.yml -vvvv

<192.0.2.10> (255, b'', b"ControlPath too long
('/tmp/.../scratchpad/sshlab/a-rather-long-directory-name-for-control-sockets/
ansible-ssh-192.0.2.10-22-ebrandi' >= 108 bytes)\r\n")

fatal: [web1.example.com]: UNREACHABLE! => {"changed": false,
"msg": "Task failed: Failed to connect to the host via ssh:
ControlPath too long (... >= 108 bytes)", "unreachable": true}

Path shortened for the page; the limit is exactly as reported.

Three things in that output are worth pulling out.

108 bytes. That is the sun_path field of a Unix domain socket address, and it is a kernel constant, not a tunable. No amount of configuration raises it.

Reported as UNREACHABLE. It reads as a network problem and it is a controller-side configuration error. Nothing was ever sent to the host. This is the central theme of the part: an SSH condition wearing an Ansible error message.

Return code 255. Which is the code reconnection_retries retries on. Set retries, and the controller will patiently repeat a deterministic configuration error several times per host before failing.

The fix is to stop overriding it. If you have a reason to keep a custom control_path, shorten control_path_dir instead — /tmp/.ansible-cp for the automation account, on local disk.

Recognising a run that lost multiplexing

The scenario: a nightly fleet run that took eight minutes now takes fifty. Nothing in the playbook changed. Somebody edited ansible.cfg and replaced the default ssh_args.

$ ANSIBLE_SSH_ARGS='' ansible-playbook -i inv.ini ping.yml -vvvv | grep -m1 'SSH: EXEC'

<192.0.2.10> SSH: EXEC ssh -o KbdInteractiveAuthentication=no
-o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
-o PasswordAuthentication=no -o ConnectTimeout=10
-o NumberOfPasswordPrompts=1 192.0.2.10 '/bin/sh -c '"'"'echo ~'"'"''

Note that ControlPath disappeared too, even though control_path_dir is untouched. Ansible only adds the ControlPath option when ControlPersist is present in ssh_args — reasonably, since a control path with no master would do nothing. The consequence is that emptying or overwriting ssh_args disables multiplexing completely, and neither control_path nor control_path_dir will bring it back.

Diagnose it from timing, not from the diff. You will not always have the diff — the change may be weeks old, in a file nobody thought to look at, or inherited from an environment variable in a CI job. The timing has a distinctive signature:

• Wall-clock time scales with task count, not with the amount of work the tasks do. Adding a trivial debug task to the play makes the run measurably longer, which should be nearly free.
• Per-host time is roughly tasks x round-trip-time above the real work.
• The slowdown is uniform across hosts, unlike a resource problem, which clusters on the loaded ones.
• Remote hosts are hurt far more than local ones, because the constant being multiplied is latency.

Then confirm it directly:

# What are the effective connection settings, including plugin-level overrides?
ansible-config dump --only-changed -t all

# Is anything in the environment overriding them?
env | grep ^ANSIBLE_SSH

# Are sockets actually being created during a run?
ls -l ~/.ansible/cp/

The third is the direct evidence. Run a play and look at the directory while it is going: with multiplexing working you see one socket per host in flight, and they linger for a minute afterwards. An empty directory during a busy run means no reuse is happening, whatever the config says.

Knowledge check