Execution environments as a reproducibility answer · Ansible

Lesson 3 pinned five layers and left one unpinned, because a virtualenv cannot pin it: the operating system underneath.

The system Python’s patch level, the OpenSSH client version, the system’s libssl, the presence of sshpass, the locale — none of these are in requirements.yml or constraints.txt, and all of them can change behaviour. Most of the time they do not matter. The times they do are memorable: an SSH client that changed its default key-exchange algorithms and stopped talking to older network devices, a locale difference that changed how a module parsed command output, a libcrypto that dropped an algorithm a legacy host still required.

An execution environment is a container image containing the whole toolchain — ansible-core, collections, Python dependencies and the OS libraries they need. Rebuilding the controller becomes pulling an image by digest.

What an EE pins that a virtualenv does not

Layer	Pinned virtualenv	Execution environment
`ansible-core`	Yes, by constraints	Yes, baked into the image
Collections	Yes, by `requirements.yml`	Yes, baked into the image
Python libraries	Yes, by constraints	Yes, baked into the image
Python interpreter	Partly — the venv uses whatever `python3` exists	Yes, the image carries it
OS libraries, SSH client, locale	No	Yes
Reproducible by digest	No — a rebuild re-resolves	Yes — the digest is the identity

The last row is the substantive difference. pip install -c constraints.txt re-resolves the dependency graph every time it runs, and while pins make that resolution deterministic in the versions it picks, it still depends on those artefacts being available and on the build host’s OS. Pulling registry.example.com/ansible-ee@sha256:REDACTED is not a resolution. It is a byte-identical retrieval or a failure.

The definition file, version 3

Read-only / Safeexecution-environment.yml - a complete version 3 definition— Read-only: file contents. This belongs in the repository alongside requirements.yml, and it is the specification the image is built from.

---
version: 3

images:
base_image:
  name: quay.io/centos/centos:stream9

dependencies:
ansible_core:
  package_pip: ansible-core==2.21.3
ansible_runner:
  package_pip: ansible-runner
galaxy: requirements.yml
python: constraints.txt
system: bindep.txt

options:
package_manager_path: /usr/bin/dnf

additional_build_steps:
append_final:
  - RUN echo "built $(date -u +%Y-%m-%dT%H:%M:%SZ)" > /etc/ee-build-date

The seven top-level keys of a version: 3 definition are version, images, dependencies, options, additional_build_files, additional_build_steps and build_arg_defaults. Four of them carry the weight:

images.base_image sets what the toolchain runs on — the layer a virtualenv cannot pin. Pin it by digest rather than by tag if you want the reproducibility this whole lesson is arguing for; :stream9 moves, @sha256:… does not.

dependencies is where the files from lesson 3 are consumed directly. galaxy: takes your requirements.yml, python: takes your constraints or requirements file, and system: takes a bindep.txt of OS packages. This is the key property: the EE definition does not replace your pinned files, it consumes them. A team that adopts EEs does not throw away lesson 3’s work; it wraps it.

ansible_core and ansible_runner are declared separately because they are the two components the builder needs to guarantee are present. ansible-runner is the execution API the image exposes, which is how ansible-navigator drives a playbook inside the container.

options carries the build-behaviour switches: package_manager_path, skip_ansible_check, skip_pip_install, container_init, relax_passwd_permissions, workdir, user and tags. package_manager_path is the one you will actually set, because it is how you tell the builder that your base image uses apt rather than dnf.

additional_build_steps injects raw container-build instructions at eight named points — prepend_base, append_base, prepend_galaxy, append_galaxy, prepend_builder, append_builder, prepend_final and append_final. The prefixes correspond to the builder’s multi-stage build: base is the foundation, galaxy is where collections are installed, builder is where Python wheels are compiled, and final is the image you ship.

The remaining two keys are additional_build_files, which copies files from the build context into the image so additional_build_steps can use them, and build_arg_defaults, which sets default build arguments such as proxy settings.

Building and running

Configuration changebuild the image— CONFIGURATION: builds a container image on the build host. Changes no managed node. Requires a container runtime - Podman by default, Docker with --container-runtime docker.

cd /srv/ansible

# Generate the build context and build the image.
ansible-builder build   --file execution-environment.yml   --tag registry.example.com/ansible-ee:2.21.3-1   --container-runtime podman

# Inspect what actually landed in it.
podman run --rm registry.example.com/ansible-ee:2.21.3-1   ansible --version
podman run --rm registry.example.com/ansible-ee:2.21.3-1   ansible-galaxy collection list

Those two inspection commands are the EE equivalent of lesson 3’s verification checks, and they matter for the same reason: a build that printed no errors is not evidence about what is inside the image.

Service impact possiblerun a playbook through the image— SERVICE-IMPACT: this is a real playbook run against real hosts. The container is only the toolchain; --limit and --check apply exactly as they always have, and the blast radius is whatever the play targets.

ansible-navigator run site.yml   --execution-environment-image registry.example.com/ansible-ee:2.21.3-1   --mode stdout   --limit canary   --check

--mode stdout is worth knowing on day one. ansible-navigator defaults to an interactive text UI, which is genuinely useful for exploring a run afterwards and completely wrong for CI, for a wrapper script, or for anyone expecting playbook output on a terminal.

Everything the play needs must be mounted, and nothing is by default

The container starts with your project directory available and nothing else. That is correct behaviour and it catches everyone once.

Your SSH private key is in ~/.ssh on the host, not in the image. Your known_hosts is on the host. Your vault password file is on the host. Your ssh-agent socket is on the host, and its path is in an environment variable the container does not inherit.

ansible-navigator has settings for passing these through — volume mounts and environment-variable pass-through — and they need to be declared in ansible-navigator.yml rather than discovered one failure at a time. The failure mode is a run that authenticates against nothing and reports every host unreachable, which reads exactly like a network problem and is not one.

Under the hood

Where does the code actually execute when you use an execution environment? The answer surprises people who expect the container to be a sandbox around the whole operation.

The controller side runs inside the container: the playbook parser, Jinja templating, lookups, filters, connection plugins, callbacks and strategy plugins. All of that uses the image’s ansible-core, the image’s collections and the image’s Python.

The target side does not change at all. Modules are still shipped to managed nodes as AnsiballZ payloads over SSH and still execute using the managed node’s Python interpreter. The image’s Python is irrelevant to them.

This is worth stating because it bounds what an EE fixes. It pins everything on the controller side of the connection and nothing on the other side. If your fleet has a host with Python 3.6 and a module that needs 3.8, the EE changes nothing about that — Part XLVI’s remote interpreter requirements still apply in full.

The second implication is about the connection itself. The SSH client doing the connecting is the one inside the image, which is precisely why an EE pins behaviour a virtualenv cannot. It also means an ssh_config in your home directory, a ProxyJump your organisation relies on, or a bastion configuration in ~/.ssh/config is not in effect unless you mounted it. Part XX’s bastion patterns need explicit carrying-over, and a run that worked yesterday on the venv and fails today in the image is usually this.

What happens if the registry holding the image is also gone?

Then the EE has moved your single point of failure rather than removing it, and it is worth checking whether you have done that.

The rebuild sequence with an EE is: get a host, install a container runtime, pull the image, run. Step three depends on a registry. If that registry is the internal one, running on the cluster the automation manages, you have the dependency inversion from lesson 2 in a new place.

Three answers, and most teams need the first two:

Keep the definition, not just the image. execution-environment.yml, requirements.yml and constraints.txt are in the repository, so the image is rebuildable from source on any host with a container runtime and network access to the upstream sources. Slower than a pull, and it re-resolves — which is exactly the reproducibility you were trying to avoid — but it is a path that exists.

Keep an exported image with the escrow. podman save produces a tarball; podman load restores it. One file, a few hundred megabytes, stored wherever your escrow material lives. This is the equivalent of lesson 3’s advice to keep the collection tarballs, and it makes the rebuild independent of every registry.

Keep the venv path working. The most under-rated answer. A team whose repository still contains a working constraints.txt and requirements.yml can rebuild a plain pinned controller with no container runtime at all. Adopting EEs does not require deleting that capability, and a rebuild runbook with two documented paths is stronger than one with a better single path.

The general principle, which applies well beyond containers: a reproducibility mechanism that depends on infrastructure has moved the DR question rather than answered it. Ask where the artefact lives and what has to be working for you to get it.

Choosing

Does your fleet include hosts sensitive to SSH client or crypto library versions - old network devices, legacy appliances, hosts with restricted algorithms? If yes, the OS layer matters and an EE earns its cost.
Do multiple people or systems run this automation from different machines? If yes, an image is a much better answer than "everyone install these versions".
Do you already run containers in production, with a registry, a build pipeline and people who understand image lifecycle? If no, an EE adds a new operational surface to your DR plan rather than simplifying it.
Is your controller a single well-managed host with a pinned venv, verified quarterly? If yes, you may already have enough, and lesson 6 will tell you.
Whichever you choose, the answer must be in the repository as a definition, and the drill in lesson 6 must exercise the path you actually intend to use.

Knowledge check

Why this matters

Execution environments are frequently presented as the answer to reproducibility, and they are a good answer to part of it. The framing this lesson insists on is that they are one option with a specific cost, chosen for a specific reason.

The reason is the OS layer. If you have ever debugged a run that worked on one engineer’s machine and failed on another’s, and the difference turned out to be an SSH client version or a crypto library, then you have met the problem an EE solves and no amount of Python pinning would have helped.

The cost is a new operational surface inside your DR plan: a container runtime, a registry, an image lifecycle and a set of mounts that have to be correct before anything works. For a team already running containers, that cost is close to zero. For a team that is not, it is a second system to recover before recovering the first.

What does not change either way: the image pins the toolchain and nothing else. Your inventory, your vault password, your SSH key and your escrow are exactly as much your problem as they were in lessons 2, 3 and 4. An EE makes the toolchain reproducible. The next lesson is about proving the rest of it is.