How a module gets to the target: AnsiballZ and pipelining · Ansible

The previous lesson said “the payload is transferred and executed”. This one opens the payload, because what is inside it explains a set of behaviours that are otherwise arbitrary: why the simplest possible task transfers over a hundred kilobytes, why become and pipelining interact, why keeping remote files is both the best debugging tool available and a way to leave secrets on a hundred hosts.

What is actually shipped

Take ansible.builtin.ping — the smallest useful module there is. Its source is a few dozen lines. Here is what arrives on the managed node.

Read-only / Safethe wrapper left behind by a single ping task— Executed against ansible-core 2.21.3 with ANSIBLE_KEEP_REMOTE_FILES=1, in an isolated temporary directory. The file is the AnsiballZ wrapper for one trivial module.

$ wc -c ~/.ansible/tmp/ansible-tmp-*/AnsiballZ_ping.py

166844 /home/ops/.ansible/tmp/ansible-tmp-1786481678.42-4004527-44275714459835/AnsiballZ_ping.py

163 kilobytes, for ping.

The wrapper is a Python script with a large base64 string in it. Decoded, that string is a zip archive:

Read-only / Safewhat is inside the payload— Read-only inspection of the wrapper captured above, decoded on the controller. Verified on ansible-core 2.21.3.

$ python3 - <<'PY'
import base64, io, zipfile
blob = None
for line in open('AnsiballZ_ping.py'):
  if line.startswith("zip_data='"):
      blob = line[len("zip_data='"):].rstrip().rstrip("'")
raw = base64.b64decode(blob)
z = zipfile.ZipFile(io.BytesIO(raw))
print('zip bytes:', len(raw))
print('entries:', len(z.namelist()))
for n in z.namelist()[:6]:
  print('  ', n)
PY

zip bytes: 115382
entries: 56
 ansible/__init__.py
 ansible/module_utils/__init__.py
 ansible/module_utils/_internal/_ansiballz/_loader.py
 ansible/module_utils/_internal/__init__.py
 ansible/module_utils/_internal/_ansiballz/__init__.py
 ansible/module_utils/_internal/_errors.py

Fifty-six files. The module itself — ansible/modules/ping.py — is one of them. The other fifty-five are module_utils: argument-spec validation, JSON encoding, text conversion, file helpers, distribution detection, error handling.

This is the concrete meaning of agentless. The library the module needs is not installed on the node, so it travels with the module, every time, for every task.

How the wrapper actually runs

The generated file is called an AnsiballZ wrapper, and its structure is worth knowing because it explains the debugging behaviour later in this lesson.

The controller determines which module_utils the module imports, recursively, and builds a zip archive containing the module and all of them.
The archive is base64-encoded and embedded in a Python script template, along with the task’s arguments, also encoded.
That script is what is transferred and executed.
On the node, the wrapper decodes the archive, writes it to a temporary path outside the Ansible temp directory, inserts it into sys.path, and imports the module as __main__ — which is what makes the module’s own if __name__ == '__main__' block run.
The module executes, prints one JSON object to stdout, and exits.
The wrapper removes the extracted archive; the controller removes the temporary directory.

Two details worth carrying forward:

The arguments are inside the wrapper. They are not passed on the command line. This matters for security: a task’s arguments do not appear in the node’s process table, which is why a password passed to a module is not visible in ps output on the managed node the way it would be if it were a shell command argument.
Compression is on. The archive is deflated, verified as DEFAULT_MODULE_COMPRESSION = ZIP_DEFLATED in the configuration. The 115 KB zip expands to considerably more source.

The cost this implies

Per task, per host, without pipelining, the sequence from the previous lesson runs in full: create a temporary directory, transfer a six-figure-byte file, chmod it, execute it, remove the directory.

At three hundred hosts and sixty tasks that is eighteen thousand transfers. This is not a reason to avoid tasks — it is why forks, connection reuse and pipelining exist, and why the performance part of this course insists on measuring before tuning.

Keeping the remote files

ANSIBLE_KEEP_REMOTE_FILES=1 tells Ansible not to clean up the temporary directory. The wrapper stays on the node, and you can read it, decode it, and run it by hand.

Read-only / Safekeep the wrapper on the node— Read-only for the ping module specifically. Note what is missing from the trace: there is no cleanup step, so the wrapper survives for you to inspect.

$ ANSIBLE_KEEP_REMOTE_FILES=1 ansible node1 -m ansible.builtin.ping -vvv

<node1> PUT /home/ops/.ansible/tmp/ansible-local-.../tmpoicft4o4 TO /home/ops/.ansible/tmp/ansible-tmp-1786481678.42-4004527-44275714459835/AnsiballZ_ping.py
<node1> EXEC /bin/sh -c 'chmod u+rwx /home/ops/.ansible/tmp/ansible-tmp-.../AnsiballZ_ping.py'
<node1> EXEC /bin/sh -c '/usr/bin/python3.12 /home/ops/.ansible/tmp/ansible-tmp-.../AnsiballZ_ping.py'
node1 | SUCCESS => {
  "ansible_facts": {
      "discovered_interpreter_python": "/usr/bin/python3.12"
  },
  "changed": false,
  "ping": "pong"
}

Note the absent line: there is no rm -f -r at the end. The directory survives.

This is genuinely the best debugging tool in Ansible for a class of problem that is otherwise very hard: a module that fails on one host and works on thirty-nine. You can reproduce the exact execution on that host, under that interpreter, with those arguments, without Ansible in the loop at all.

The wrapper contains the task arguments, in the clear

Base64 is an encoding, not encryption. Anything you passed to the module is inside that file and is trivially recoverable by anyone who can read it.

If the task was ansible.builtin.user with a password hash, a database module with a connection string, or anything that consumed a vault-decrypted variable, that value is now sitting in a temporary directory on the managed node until somebody removes it. no_log: true does not help: it suppresses controller-side display, and the value still has to reach the module to be used.

Three rules that follow:

Set it as an environment variable for one command, never in ansible.cfg. A permanent setting leaves debris on every host after every run, indefinitely.
Use it on one host, not a fleet.
Delete the directory when you are done, and treat any secret that was in a kept payload as needing rotation if the host is shared.

Pipelining

Pipelining removes the write step. Instead of transferring the wrapper to a file and then executing that file, Ansible opens the remote Python interpreter and feeds the wrapper to it on stdin.

The difference is visible immediately in a verbose run. Without pipelining, one task produces a mkdir, a PUT, a chmod, an EXEC and an rm. With it, verified on ansible-core 2.21.3:

Read-only / Safethe same task with pipelining enabled— Executed on ansible-core 2.21.3. Every filesystem operation has disappeared: no directory is created, nothing is written, nothing is removed.

$ ANSIBLE_PIPELINING=True ansible-playbook -i localhost, local.yml -vvv | grep -E 'PUT|EXEC|ESTABLISH'

<localhost> ESTABLISH LOCAL CONNECTION FOR USER: ops
<localhost> EXEC /bin/sh -c 'echo FOUND; command -v '"'"'python3.14'"'"'; ... ; echo ENDFOUND'
<localhost> EXEC /bin/sh -c /usr/bin/python3.14

Compare that with the non-pipelined trace in the previous lesson. The mkdir, the PUT, the chmod and the rm are all gone. What remains is interpreter discovery and a single EXEC that starts Python with the payload arriving on its standard input.

The gains are real and they compound over a run:

Fewer SSH round trips per task. This is the dominant cost on a high-latency link.
Nothing written to the node’s filesystem. No temporary directory, which sidesteps noexec mounts, full home directories and quota problems.
Less residue. There is no file to be left behind by an interrupted run.

Why it is off by default

Pipelining conflicts with sudo configurations that require a TTY. The default configuration confirms it:

Read-only / Safepipelining is off by default— Inspects configuration only. Verified on ansible-core 2.21.3.

$ ansible-config dump | grep -E '^(ANSIBLE_PIPELINING|DEFAULT_KEEP_REMOTE_FILES)\('

ANSIBLE_PIPELINING(default) = False
DEFAULT_KEEP_REMOTE_FILES(default) = False

The upstream documentation for the setting states the reason directly: it conflicts with privilege escalation, and using it with sudo requires requiretty to be disabled in /etc/sudoers on all managed hosts — which is why it is disabled by default rather than because it is risky in itself.

Many modern distributions ship without requiretty for the relevant entries, which is why enabling pipelining often “just works”. Confirm it rather than assuming, because the failure mode is a run that breaks on whichever subset of your fleet has the older sudoers policy — a partial failure across a mixed-age estate, which is the hardest kind to interpret.

Pipelining enabled, and a third of the fleet stops working

A team enables pipelining = True in ansible.cfg after reading that it improves performance. Testing on the staging fleet — all Ubuntu 24.04, built this year — shows a substantial speed-up and no problems.

Applied to production, roughly a third of hosts begin failing every task that uses become. The error mentions sudo: sorry, you must have a tty to run sudo.

The affected third are older hosts whose /etc/sudoers still carries Defaults requiretty from a hardening baseline applied years ago. The newer hosts do not have it, which is why staging was clean.

Why the obvious diagnosis is wrong: the natural reading is that pipelining is broken or that those hosts have an SSH problem. Neither is true. Pipelining is working exactly as designed; it feeds the payload to Python on stdin, which means there is no TTY, and sudo on those hosts refuses to run without one. The hosts are fine, sudo is fine, and the setting is fine — they are simply incompatible.

The instructive part is that the failure is partial and correlated with host age. That shape — a change that works on the hosts you tested and fails on an older subset — recurs constantly in mixed-age estates, and the diagnostic instinct it should trigger is “what differs between the two groups”, not “what is wrong with the change”.

Two ways out: remove requiretty from the affected hosts, which is a fleet change of its own with its own blast radius; or set pipelining per group rather than globally, leaving it off where sudoers requires a TTY. The second is usually the right first move, because it is reversible and does not touch sudo policy under time pressure.

What this explains

Behaviour	Explanation from this lesson
A trivial task transfers over 100 KB	The module travels with all its `module_utils`
Task count matters more than it looks	Each task builds and ships its own payload
A task fails with `MODULE FAILURE` on a full disk	The payload could not be written to the temp directory
A `noexec` home directory breaks everything	The wrapper is written there and then executed
Module arguments do not appear in `ps` on the node	They are embedded in the wrapper, not passed as argv
Pipelining conflicts with `become`	No TTY is allocated, and `requiretty` sudoers refuse
Keeping remote files leaves secrets on hosts	Base64 is an encoding, not encryption

Knowledge check

Why this matters

This lesson is the mechanism behind claims the rest of the course makes repeatedly.

When the security part says a module’s arguments are safer than a shell command’s, this is why: they are embedded in the payload rather than passed as argv, so they do not appear in the node’s process table. When the performance part says task count matters, this is why: each task is its own payload. When the debugging part reaches for ANSIBLE_KEEP_REMOTE_FILES, this is what it produces and what it leaves behind. When the Python part explains why a node needs a working interpreter and a writable executable temp directory, this is the requirement it is describing.

It is also the fastest route out of a specific and miserable class of problem: a module that fails on one host and works on the rest. Being able to keep the wrapper, read it, and run it by hand under that host’s own interpreter turns an Ansible problem into a Python problem, which is a much smaller thing to debug.