A controller you can rebuild byte-for-byte enough

Part III built a pinned controller because pinning makes runs predictable. This lesson revisits it for a different reason: pinning is what makes a rebuild reproduce something rather than produce something new.

The distinction matters at 09:00 on rebuild day. An unpinned rebuild gives you a working controller — probably. It gives you the latest ansible-core, the latest collections, and a fleet that has been converged by different versions of both. Whether that combination behaves as the old one did is a question you get to answer live, on production, with an incident already in progress.

“Byte-for-byte enough” is the standard. Not a bit-identical filesystem image, which is neither achievable nor useful, but: every component whose version could change behaviour is declared, and the rebuild is verified to have produced the declared version.

The five layers

Pinning one layer and not the others produces a rebuild that is reproducible in the part nobody was worried about.

The fifth is the one that gets missed, because those libraries are installed as a side effect of something else and never recorded. A dynamic inventory plugin that needs boto3, a filter that needs netaddr, a json_query that needs jmespath: each one works on the old controller and fails on the new one with an import error that names a library nobody has heard of.

# requirements.yml - collections and roles, pinned
---
collections:
- name: ansible.posix
  version: 3.0.0
- name: community.general
  version: 12.0.0
- name: ansible.utils
  version: 6.0.0

roles:
- name: internal.baseline
  src: https://git.example.com/ansible/role-baseline.git
  scm: git
  version: v2.4.1

ansible-core==2.21.3
jmespath==1.0.1        # json_query filter
netaddr==1.3.0         # ipaddr filters
boto3==1.40.0          # aws_ec2 inventory plugin
botocore==1.40.0       # boto3 dependency, pinned so pip cannot drift it

[defaults]
inventory       = inventory/
collections_path = ./collections
roles_path      = ./roles:./galaxy_roles
vault_password_file = ~/.vault/prod
host_key_checking = True

Installing the pinned set

cd /srv/ansible

# 1. The interpreter and ansible-core.
python3 -m venv .venv
./.venv/bin/pip install -c constraints.txt   ansible-core jmespath netaddr boto3

# 2. Collections, into the path ansible.cfg names.
./.venv/bin/ansible-galaxy collection install   -r requirements.yml -p ./collections

# 3. Roles, into the path ansible.cfg names.
./.venv/bin/ansible-galaxy role install   -r requirements.yml -p ./galaxy_roles

Three flags in that block are worth stopping on.

-c constraints.txt applies the pins to everything pip resolves, including transitive dependencies. Listing ansible-core==2.21.3 as an argument pins one package; a constraints file pins the resolution.

-p ./collections installs to a project-scoped path rather than ~/.ansible/collections. That is what makes the collection set part of the project rather than part of somebody’s home directory — and it is the difference between a rebuild that is defined by a file and one that is defined by a host.

-r requirements.yml is mutually exclusive with naming collections on the command line. If you find yourself doing both, the ones on the command line are not recorded anywhere and will not survive the rebuild.

-p warns rather than fails, and the warning is the whole problem

Installing to a path that is not in the configured collections paths is not an error. It succeeds, prints a warning, and produces a controller where the collections are on disk and invisible to every run.

$ ansible-galaxy collection install -r requirements.yml -p ./collections

[WARNING]: The specified collections path '/srv/ansible/collections' is not part of the configured Ansible collections paths '/home/ansible/.ansible/collections:/usr/share/ansible/collections'. The installed collection will not be picked up in an Ansible run, unless within a playbook-adjacent collections directory.

Proving the rebuild, rather than assuming it

An install that printed no errors is not evidence. Three checks turn it into evidence, and all three are read-only.

Check 1: the toolchain is the declared one

$ ./.venv/bin/ansible --version

ansible [core 2.21.3]
config file = /srv/ansible/ansible.cfg
configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /srv/ansible/.venv/lib/python3.12/site-packages/ansible
ansible collection location = /srv/ansible/collections
executable location = /srv/ansible/.venv/bin/ansible
python version = 3.12.3
jinja version = 3.1.6
pyyaml version = 6.0.3 (with libyaml v0.2.5)

config file = None on a rebuilt controller means the project ansible.cfg is not being read, which means every setting the fleet depends on is at its default. That single line has caught more botched rebuilds than any other check in this part.

Check 2: the collections are the declared versions

# What is installed, as name+version pairs.
./.venv/bin/ansible-galaxy collection list --format json | python3 -c 'import json,sys
d=json.load(sys.stdin)
for path in d.values():
  for name, meta in path.items():
      print(name, meta["version"])' | sort > /tmp/installed.txt

# What was declared.
python3 -c 'import yaml
for c in yaml.safe_load(open("requirements.yml"))["collections"]:
  print(c["name"], c["version"])' | sort > /tmp/declared.txt

diff /tmp/declared.txt /tmp/installed.txt && echo "collections match the pins"

A non-empty diff has three shapes and each means something different. A version mismatch means a pin was a range and resolved differently today. An extra installed collection means a dependency was pulled in that requirements.yml does not name — harmless until it is the reason a play works. A missing one means the install did not do what you think it did.

Check 3: the artefacts are what upstream published

# Verify everything named in requirements.yml, in the project path.
./.venv/bin/ansible-galaxy collection verify   -r requirements.yml -p ./collections

# Same check with no network, against the locally stored manifest only.
./.venv/bin/ansible-galaxy collection verify   -r requirements.yml -p ./collections --offline

The distinction between those two runs is worth understanding. Without --offline, verify fetches the canonical manifest hash from the Galaxy server, so it detects a collection whose local manifest was tampered with. With --offline it validates integrity against the manifest already on disk, which catches corruption and accidental local edits but not a manifest that was replaced wholesale. During a rebuild with no network to Galaxy, --offline is what you have; record which one you ran.

Part XXVII’s signature-verification lesson covers --keyring and --required-valid-signature-count for the stronger form.

Knowledge check