Pipelining and its preconditions · Ansible

The architecture part of this course established what pipelining does: instead of writing the module wrapper to a file on the target and then executing that file, Ansible feeds the wrapper to the remote Python interpreter on standard input. The mkdir, PUT, chmod and rm disappear, and one task becomes one round trip.

This lesson is about the precondition — because “pipelining conflicts with privilege escalation” is repeated everywhere and is, as usually stated, slightly wrong in a way that makes it hard to act on.

What the documentation actually says

Read-only / Safethe setting, in its own words— From ansible-doc -t connection ansible.builtin.ssh on ansible-core 2.21.3.

$ ansible-doc -t connection ansible.builtin.ssh

   pipelining  Pipelining reduces the number of connection operations
             required to execute a module on the remote server, by
             executing many Ansible modules without actual file
             transfers.
             This can result in a very significant performance
             improvement when enabled.
             However this can conflict with privilege escalation
             ('become'). For example, when using sudo operations you
             must first disable 'requiretty' in the sudoers file for
             the target hosts, which is why this feature is disabled
             by default.
      default: false

Two things to take from that. The default is false because of the requiretty interaction, not because pipelining is risky in itself. And the words “for example” are doing real work — sudo is an instance of the problem, not the problem.

The actual mechanism: one line in the connection plugin

The conflict is not between pipelining and sudo. It is between pipelining and a pseudo-terminal, and the decision is made explicitly, here, in ansible/plugins/connection/ssh.py on 2.21.3:

# we can only use tty when we are not pipelining the modules. piping
# data into /usr/bin/python inside a tty automatically invokes the
# python interactive-mode but the modules are not compatible with the
# interactive-mode ("unexpected indent" mainly because of empty lines)

# -tt can cause various issues in some environments so allow the user
# to disable it as a troubleshooting method.
use_tty = self.get_option('use_tty')

args: tuple[str, ...]
if not in_data and sudoable and use_tty:
    args = ('-tt', self.host, cmd)
else:
    args = (self.host, cmd)

in_data is the pipelined payload. So the condition reads: allocate a pseudo-terminal only when there is no payload on stdin, the command is one that might need to escalate, and use_tty is on.

Follow the causal chain, because the order matters:

By default, Ansible passes -tt to ssh, which forces allocation of a pseudo-terminal on the target.
sudo with Defaults requiretty refuses to run without a TTY. With -tt present, there is one, so it runs.
Pipelining puts the module on stdin, so in_data is set, so the condition is false, so -tt is not passed.
sudo now has no TTY and refuses.

The comment explains why step 3 has to work that way, and it has nothing to do with sudo at all: piping a script into an interactive Python inside a TTY makes Python treat it as interactive input, and the module source — which contains blank lines — gets rejected with unexpected indent. The two features are mutually exclusive for a reason internal to Python, and sudo is a bystander that happens to depend on the flag that gets dropped.

Which means requiretty is not really a pipelining problem

use_tty is an independent setting, documented as a troubleshooting control — “add -tt to ssh commands to force tty allocation”, default true. Look at the condition again: setting use_tty: false removes -tt whether or not pipelining is on.

So an estate can hit sudo: sorry, you must have a tty to run sudo from two directions, and the second one is much harder to spot because nobody is looking for it:

Somebody enabled pipelining. Widely known, widely documented, and the first thing anyone suspects.
Somebody set use_tty: false — usually years ago, to work around a terminal-related problem on some appliance — and it has been sitting in ansible.cfg ever since.

If you meet that sudo error and pipelining is off, do not conclude the error is impossible. Check use_tty:

ansible-config dump --only-changed -t all | grep -i use_tty

The useful reframing: the requirement is not “do not use pipelining with sudo”. It is sudo needs a TTY on hosts where requiretty is set, and -tt is the only thing that provides one. Anything that removes -tt breaks those hosts.

Establishing the precondition before you change anything

The architecture part described what happens when a team enables pipelining globally and a third of a mixed-age fleet stops working. The reason that scenario is so common is that people establish the precondition retrospectively, from the failures.

You can establish it in advance, read-only, in one run.

Read-only / Safesurvey requiretty across the fleet

# playbooks/preflight-requiretty.yml
# READ-ONLY: reads sudo policy and reports it. Changes nothing.
# Uses become because sudoers is only readable with privilege.
- name: Survey sudo TTY policy before enabling pipelining
hosts: all
gather_facts: false
become: true
tasks:
  - name: Ask sudo for the effective defaults
    ansible.builtin.command:
      argv: [sudo, -n, sudo, -V]
    register: sudo_v
    changed_when: false
    failed_when: false

  - name: Report hosts whose sudo policy requires a tty
    ansible.builtin.debug:
      msg: "requiretty is in effect"
    when: "'requiretty' in sudo_v.stdout"

That is one approach and it has a weakness worth naming: sudo -V reports the compiled defaults and the settings it can see, and a requiretty that applies only to a specific user or command via a Defaults: specification may not show up. The more reliable survey reads the policy files:

Read-only / Safethe direct read

    - name: Search sudoers and its drop-in directory
    ansible.builtin.shell:
      cmd: grep -RIl '^[[:space:]]*Defaults.*requiretty' /etc/sudoers /etc/sudoers.d/ || true
    register: rt
    changed_when: false

  - name: Hosts that will break under pipelining
    ansible.builtin.debug:
      var: rt.stdout_lines
    when: rt.stdout_lines | length > 0

|| true because grep exits non-zero when it finds nothing, and “nothing found” is the answer you are hoping for rather than a failure. The result is a list of hosts, which turns “a third of the fleet broke” into “these 47 hosts need attention first” — the same information, obtained before the outage instead of during it.

Rolling it out per group

pipelining is settable per host and per group, which makes a staged rollout possible without touching sudo policy under time pressure:

Read-only / Safesix places it can be set, and only two of them scope— From ansible-doc -t connection ansible.builtin.ssh on 2.21.3.

$ ansible-doc -t connection ansible.builtin.ssh

   pipelining
      set_via:
        env:
        - name: ANSIBLE_PIPELINING
        - name: ANSIBLE_SSH_PIPELINING
        ini:
        - key: pipelining
          section: defaults
        - key: pipelining
          section: connection
        - key: pipelining
          section: ssh_connection
        vars:
        - name: ansible_pipelining
        - name: ansible_ssh_pipelining
      default: false

Configuration changeon by default, off where the survey said so

# inventory/production/group_vars/all.yml
ansible_ssh_pipelining: true

# inventory/production/group_vars/legacy_sudoers.yml
# 47 hosts carrying Defaults requiretty from the 2019 hardening baseline.
# Surveyed 2026-08-11. Remove this group as hosts are rebuilt; the
# replacement images do not set it. Tracked as OPS-4471.
ansible_ssh_pipelining: false

The exception group carries what every exception in this course carries: a count, a reason, a date, an owner, and the condition under which it disappears. An exception with an expiry becomes smaller over time. An exception without one becomes permanent and eventually nobody remembers which hosts depended on it.

Note the direction of the default. Pipelining on for the fleet and off for the known exceptions is better than the reverse, because it makes the exception list visible and shrinking. The reverse — off globally, on for a few groups — hides the fact that most of your fleet is paying for a constraint that applies to 47 hosts.

Sequence for enabling it on an estate you did not build

Survey. Produce the list of hosts where requiretty is set. Do not skip this because staging was clean; staging is usually the newest hardware in the estate and therefore the least representative.
Enable for one group that the survey says is clear, ideally one whose runs you can watch. Confirm from -vvv that the PUT operations have gone.
Measure. The gain is largest where latency is highest, so measure on a distant group rather than the one in the same rack.
Widen, group by group, leaving the surveyed exceptions off.
Ticket the exceptions against the rebuild or hardening-baseline work that will remove requiretty. That is the change that retires the group, and it is somebody else’s change.

The whole sequence is reversible at every step by editing one line in group_vars/, which is the property that makes it safe to do on a production estate at all.

Under the hood

There is a security consequence that rarely gets mentioned, and it points forward to the secrets part of this course.

Without pipelining, the module wrapper is written to a file on the managed node before it is executed. That wrapper contains the module’s arguments — which, for a task that configures a database, sets an API credential or writes a configuration template, includes the secret.

The file is created under the remote user’s temporary directory and removed afterwards. In the normal case that window is short and the permissions are restrictive. Two cases make it worse:

A run that is interrupted leaves the file behind. So does ANSIBLE_KEEP_REMOTE_FILES, which is exactly what somebody sets while debugging the task that handles the credential.
become to an unprivileged user historically required loosening permissions on that temporary directory so the target user could read the files, which widens who on that host can read the arguments while the task runs.

With pipelining, the payload goes to the interpreter on stdin and is never written to the target’s filesystem at all. There is no file to leak, no file to leave behind, and no directory permission question.

This does not make pipelining a secrets control — the argument is still visible in the process table on the target for the life of the process, and still passes through controller memory. But when the secrets part enumerates the places a credential escapes, “a temporary file on the managed node” is one of them, and pipelining removes that one.

What happens if you enable pipelining and the run gets slower?

Uncommon, but it happens, and the diagnosis is not obvious.

Pipelining trades round trips for a larger single transfer: the whole module payload now goes in one write instead of a separate file transfer. On a link with high bandwidth and low latency, that trade is close to neutral, and any per-connection overhead you have elsewhere is unchanged.

Check three things before concluding that pipelining is at fault.

Is connection reuse still working? Pipelining and multiplexing address different costs and are complementary — reuse removes handshakes, pipelining removes round trips within a session. If somebody enabled pipelining by writing to ssh_args, they will have discarded ControlPersist at the same time, and the loss will swamp the gain. Look at ~/.ansible/cp/ during a run.

Are the tasks actually modules? Pipelining changes how modules are delivered. A play dominated by copy and template still transfers files, because those transfer content rather than code, and will show little change.

Is the run latency-bound at all? If the wall clock is dominated by a package manager doing real work on the far end, the transport was never the constraint and no transport setting will move it. Measure where the time goes before tuning what you can most easily change.

Knowledge check

Why this matters

Pipelining is the largest performance setting in Ansible that is off by default, and the reason it is off is a compatibility constraint that most current estates no longer have. A lot of fleets are paying a per-task round trip for a sudoers directive that no host in the inventory actually sets.

But the reason to understand the mechanism rather than memorise the advice is the second failure route. “Pipelining conflicts with sudo” is memorisable and leaves you helpless in front of a requiretty failure on a fleet where pipelining is off. Knowing that the real dependency is on the -tt flag, and that two independent settings control it, turns that from a mystery into a two-command check.

That pattern recurs throughout this part. The advice you can recite tells you what usually happens; the mechanism tells you what is happening now.