Pipelining, ControlPersist and connection cost · Ansible

Two earlier lessons in this course established the mechanism: how a module is packaged and shipped to a target, and what ControlMaster and ControlPersist remove from that cycle. Neither is repeated here.

This lesson is the performance view of the same machinery — the arithmetic that decides whether connection overhead is your problem, what the settings cost you elsewhere, and how to change one of them without contaminating the measurement.

The arithmetic

Connection overhead is not paid per run or per host. It is paid per task per host, so it scales with the product, and the product is what makes it significant on exactly the plays people complain about.

Play	Tasks	Hosts	Connection operations
A small role	12	20	240
A full site playbook	60	400	24,000
A patching run with health checks	25	1,200	30,000

At 100 ms of overhead per operation — a modest figure for a full SSH handshake on a LAN, and far too optimistic across a WAN — the second row is 2,400 seconds of overhead distributed across the run. Divided by forks: 20, that is two minutes of pure connection setup, before any task does anything.

That is the shape to recognise in a profile: time spread evenly across many tasks, none of which individually looks slow. The previous lessons in this part called this “a connection problem wearing a parallelism costume”, and the arithmetic above is how you check whether that is what you have.

The defaults, verified

Read-only / Safewhat the ssh connection plugin actually defaults to— ansible-core 2.21.3 with no configuration file present. These are the built-in values, not somebody's recommended settings.

$ ansible-config dump -t connection | grep -E '^(pipelining|ssh_args|reconnection_retries|timeout|control_path)'

pipelining(default) = False
reconnection_retries(default) = 0
control_path(default) = None
control_path_dir(default) = ~/.ansible/cp
ssh_args(default) = -C -o ControlMaster=auto -o ControlPersist=60s
timeout(default) = 10

Three of those are worth stating explicitly because they are widely misremembered.

pipelining is false. It is not on by default, and it has never been. Every claim that Ansible pipelines by default is wrong.

ssh_args already contains connection reuse. ControlMaster=auto and ControlPersist=60s are in the shipped default, so multiplexing is on unless something removed it. The most common way something removes it is a well-meaning ssh_args override that sets a proxy option and drops the rest of the string.

reconnection_retries is 0, and the plugin is specific about when it would retry at all:

Read-only / Safewhen Ansible retries a connection— From ansible-doc for the installed ssh connection plugin on 2.21.3.

$ ansible-doc -t connection ansible.builtin.ssh | grep -A5 reconnection_retries

   reconnection_retries  Number of attempts to connect.
                       Ansible retries connections only if it gets
                       an SSH error with a return code of 255.
                       Any errors with return codes other than 255
                       indicate an issue with program execution.

Return code 255 is ssh’s own “something went wrong with the connection” code. Anything else came from the command that ran on the far side, and retrying it would be re-running your task, not re-establishing a connection. That distinction is why the default is 0 and why raising it is a narrow fix for flaky networks rather than a general reliability setting.

An ssh_args override is how connection reuse gets silently disabled

ssh_args is a single string, and setting it replaces the default rather than adding to it.

# ansible.cfg - this DISABLES multiplexing
[ssh_connection]
ssh_args = -o ProxyCommand="ssh -W %h:%p bastion.example.com"

Every task on every host now pays a full handshake, through the bastion, and the only symptom is that the run got slower. Nothing errors.

Keep the default and append:

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s -o ProxyCommand="ssh -W %h:%p bastion.example.com"

The diagnostic for a run that lost multiplexing is a timing signature — the first task on each host being no faster than the twentieth — and it is covered in this course’s SSH part. What belongs here is the reason it matters for performance work: if you are about to measure a pipelining change and multiplexing is already broken, you are measuring the wrong thing. Confirm ssh_args before you tune anything else.

What pipelining requires of your fleet

Pipelining removes the file-transfer step from each task, executing the module over the already-open connection. The plugin describes it as “a very significant performance improvement when enabled”, and then names the condition:

“However this can conflict with privilege escalation (become). For example, when using sudo operations you must first disable requiretty in the sudoers file for the target hosts.”

That is the trade the default is protecting you from, and it is worth being precise about why it is a big one.

requiretty in sudoers means sudo refuses to run without a terminal. It is a hardening measure — its purpose is to make it harder for a compromised non-interactive process to escalate — and on some distributions it has historically been set by default.

So enabling pipelining is not a controller-side performance setting. It is a change to the sudoers policy on every managed node, which means:

It has to be applied to the whole fleet before the controller setting is flipped, or the hosts that still have requiretty start failing.
It is a security-relevant change, which in most organisations means it is reviewed by somebody who is not you.
It fails on the hosts you forgot: the ones built from an older image, the ones a vendor manages, the ones in the region nobody remembered.

The failure mode looks like anything except a configuration setting

When pipelining is enabled and a target still requires a TTY, the task fails with a sudo error about needing a terminal — buried in stderr, under a module that has nothing to do with sudo, on some subset of hosts.

The properties that make it hard to diagnose:

It is partial. Only the hosts that still have requiretty fail, so it looks like those hosts are broken rather than like a fleet-wide setting being wrong.
It is not obviously new. The controller change was one line in ansible.cfg, possibly in a different commit from the run that exposed it.
It only affects escalating tasks. Tasks without become work fine, so the play half-succeeds and the failures cluster in a way that suggests a permissions problem.

The rollout order that avoids it: change sudoers on the fleet first (with Ansible, in a play that does not use pipelining), verify with a read-only check that no host still sets requiretty, and only then enable the controller setting. Enable it per group first if your inventory lets you, using ansible_pipelining as a group variable, so the blast radius of the change is a group rather than everything.

Read-only / Safecheck the precondition before flipping the switch— Reads sudoers configuration and reports. It changes nothing, and it is the step that turns a fleet-wide gamble into a check.

ansible -i inventory/production all -b -m ansible.builtin.command \
-a 'grep -rEl "^[^#]*requiretty" /etc/sudoers /etc/sudoers.d/' \
2>&1 | grep -B1 'rc=0'

Hosts that return a match still require a TTY. Every one of them will fail under pipelining, and finding them costs one read-only run.

Measuring the change honestly

Read-only / Safeone variable, three passes each— --check keeps this read-only. Certify the final number against a real staging run, because check mode skips modules and changes the profile.

for p in false true; do
for pass in 1 2 3; do
  printf 'pipelining=%-6s pass=%s ' "$p" "$pass"
  ANSIBLE_PIPELINING="$p" /usr/bin/time -f '%e s' \
    ansible-playbook -i inventory/staging site.yml --check >/dev/null
done
done

Two rules, both from the measurement lesson and both worth repeating here because this is the setting people break them on:

Three passes. The run-to-run spread measured earlier in this part was around half a second on a four-second run. A single before-and-after pair proves nothing.

One variable. Do not change forks in the same experiment. They fail in different places — a saturated controller versus a fleet rejecting non-TTY sudo — and a combined change leaves you unable to attribute either the improvement or the regression.

What was not measured here, and why

This lesson gives no measured pipelining speedup, and that is deliberate rather than an omission.

The validation environment for this course reaches managed nodes in the documentation address ranges, which are unreachable by design — that is what makes them safe to point a controller at. Pipelining changes what happens over an established SSH connection to a real host with a real sudo, so there is no honest way to measure it without one.

Everything stated above as a default or a requirement was read from ansible-config dump and ansible-doc on the pinned controller, or quoted from the connection plugin documentation. The speedup figure is the one thing you have to produce yourself, on your own network, against your own hosts — which is also the only figure that would be worth acting on.

The lab for this part uses a real VM for exactly this reason: sudoers, requiretty and TTY allocation need a real login stack, and a container will happily give you a misleading answer.

Under the hood

ControlPersist=60s and pipelining are often discussed as alternatives. They are not: they remove different costs and they compose.

ControlPersist keeps the SSH connection open after a task finishes, so the next task on that host reuses it. What it removes is the TCP handshake, the SSH key exchange and the authentication — the expensive part — for every task after the first.

pipelining removes the file transfer within a single task. Without it, running a module means creating a remote temporary directory, copying the module payload into it, executing it, and cleaning up: several operations over the connection. With it, the module is fed to the remote interpreter over the open connection instead.

So the per-task cost has two components, and each setting addresses one. With multiplexing but no pipelining you still pay several round trips per task. With pipelining but no multiplexing you pay a full handshake per task and fewer round trips inside it — which is why pipelining looks disappointing on a fleet whose ssh_args override broke multiplexing, and why checking ssh_args comes first.

The 60-second persistence window has its own edge. A play whose tasks are more than 60 seconds apart on a given host — a long wait_for, a slow package transaction — lets the master connection expire and pays a fresh handshake afterwards. Raising ControlPersist for such a play is reasonable; it also means idle SSH connections and their sockets persist longer on both ends, which interacts with the file-descriptor arithmetic from the forks lesson.

What happens if you set reconnection_retries to a high value to make a flaky fleet look healthy?

The same trade this course flagged for timeout, in a different place.

Retries only apply to SSH return code 255 — genuine connection errors — so raising the value does exactly one thing: it converts a reported unreachable host into a delayed success, when the retry happens to work.

What that costs:

A fork is occupied for every attempt. With timeout: 10 and three retries, one dead host holds a worker for the better part of a minute instead of ten seconds. During an incident, when unreachable hosts are what you have, the diagnostic run becomes something you sit and watch.
You stop learning which hosts are unwell. A host that needs two attempts to accept a connection is telling you something about its load, its disk, or the path to it. The retry suppresses the message and then applies configuration changes to that host anyway.

There is a legitimate use — a genuinely lossy link, a network you do not control, a known-flaky path to one site — and the way to apply it is narrowly, as a group variable on the hosts behind that link, with the reason written next to it:

# inventory/production/group_vars/site_remote.yml
ansible_ssh_retries: 3   # satellite link, packet loss documented in TICKET-1234

A global reconnection_retries = 5 in ansible.cfg is indistinguishable from somebody having made a failure go away.

Knowledge check

Why this matters

Connection tuning is the one area of Ansible performance where the controller-side change is trivial and the prerequisite is not. One line enables pipelining. Making that line safe means a sudoers change on every host that escalates, applied before the switch, verified rather than assumed, and reviewed by whoever owns hardening policy.

Teams that skip the order get a partial outage that presents as a permissions problem on an arbitrary subset of the fleet, days after the commit that caused it.

The rest is arithmetic, and it is worth doing before any of it. Overhead is per task per host. If your play is twelve tasks over twenty hosts, this whole lesson is worth about twenty seconds and you should go and look at fact gathering instead. If it is sixty tasks over four hundred hosts, connection cost may be the largest single item in your profile — and the first thing to check is not pipelining, but whether an ssh_args override quietly turned multiplexing off years ago.