Private Galaxy, mirrors and air-gapped install · Ansible

A production controller that runs ansible-galaxy collection install against public Galaxy as part of a deployment has put a third-party website in the critical path of every change it makes.

That is three problems in one sentence:

Availability. Galaxy is down, or slow, or rate-limiting your CI runners, and your change window closes with nothing deployed.
Integrity. Whatever the endpoint serves today is what runs as root on your fleet today. Lesson 6’s verification only helps if you have something to compare against, and a fresh fetch is not it.
Reproducibility. The install is not the same install twice, for all the reasons lesson 4 laid out.

The fix is not exotic. Fetch deliberately, once, into an artefact you keep — then install from the artefact.

The offline bundle

ansible-galaxy collection download resolves the dependency tree and writes tarballs instead of installing them:

Configuration changefetch the tree, install nothing— Downloads every collection in the requirements file, plus its dependencies, as tarballs into a bundle directory. Nothing is installed and no plugin runs.

$ ansible-galaxy collection download -r requirements.yml -p ./bundle

Process download dependency map
Starting collection download process to '/home/opsuser/project/bundle'
Downloading collection 'community.docker:4.7.0' to '/home/opsuser/project/bundle'
Downloading https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/community-docker-4.7.0.tar.gz to /home/opsuser/.ansible/tmp/ansible-local-1971711zm7rmrd/tmpwnwtvy3l/community-docker-4.7.0-v3o6lid7
Collection 'community.docker:4.7.0' was downloaded successfully
Downloading collection 'community.library_inventory_filtering_v1:1.1.5' to '/home/opsuser/project/bundle'
Collection 'community.library_inventory_filtering_v1:1.1.5' was downloaded successfully
Writing requirements.yml file of downloaded collections to '/home/opsuser/project/bundle/requirements.yml'

The last line is the useful part. The bundle is self-describing:

Read-only / Safewhat a bundle contains— Lists the bundle directory. Two artefacts and a generated requirements file naming them.

$ ls -la ./bundle && cat ./bundle/requirements.yml

total 588
-rw-rw-r--  1 opsuser opsuser 557525 Aug 11 22:36 community-docker-4.7.0.tar.gz
-rw-rw-r--  1 opsuser opsuser  36002 Aug 11 22:36 community-library_inventory_filtering_v1-1.1.5.tar.gz
-rw-rw-r--  1 opsuser opsuser    147 Aug 11 22:36 requirements.yml

collections:
- name: community-docker-4.7.0.tar.gz
version: 4.7.0
- name: community-library_inventory_filtering_v1-1.1.5.tar.gz
version: 1.1.5

Note what the generated file contains: filenames, not collection names. That is a relative path, and it produces the most common failure with this workflow.

The bundle only installs from inside its own directory

Copy the bundle to the disconnected machine and run the install from somewhere else, and:

[ERROR]: Neither the collection requirement entry key 'name', nor 'source' point to
a concrete resolvable collection artifact. Also 'name' is not an FQCN. A valid
collection name must be in the format <namespace>.<collection>.

Could not find community-docker-4.7.0.tar.gz.

The error blames the name for not being an FQCN, which sends people looking for a malformed requirements file. The actual cause is that community-docker-4.7.0.tar.gz is a path relative to the working directory, and you are not in it.

Change into the bundle directory first. Put that in the runbook, because the error message will not tell anyone.

Configuration changeinstall with no network at all— Installs from the bundle's own tarballs. Run from inside the bundle directory. Note the absence of any Downloading line.

$ cd ./bundle && ansible-galaxy collection install -r requirements.yml -p ../collections --offline

Starting galaxy collection install process
Process install dependency map
Starting collection install process
Installing 'community.docker:4.7.0' to '/home/opsuser/project/collections/ansible_collections/community/docker'
community.docker:4.7.0 was installed successfully
Installing 'community.library_inventory_filtering_v1:1.1.5' to '/home/opsuser/project/collections/ansible_collections/community/library_inventory_filtering_v1'
community.library_inventory_filtering_v1:1.1.5 was installed successfully

No Downloading lines. Nothing was fetched. This install will produce byte-identical content on a machine with no route to the internet, in five years, from the same bundle.

--offline on install means do not contact any distribution server. It does not cover collections referenced by a git URL or a remote tarball URL — those are still network operations, which is one more reason a bundle should contain only artefacts.

Private Galaxy servers

The bundle is the air-gapped answer. For a connected estate that simply wants control, the answer is a server you run: Automation Hub, a Galaxy-compatible artefact repository, or a pull-through cache in whatever artefact manager you already have for .deb and container images.

The configuration lives in ansible.cfg:

Read-only / Safeansible.cfg

[galaxy]
server_list = internal_hub, community_mirror

[galaxy_server.internal_hub]
url = https://hub.example.com/api/galaxy/content/published/
token = REPLACE_ME

[galaxy_server.community_mirror]
url = https://hub.example.com/api/galaxy/content/community/
token = REPLACE_ME

The mechanics, from the setting’s own documentation:

server_list names sections. Each entry foo corresponds to a [galaxy_server.foo] header holding that server’s details.
Order is resolution order. A collection is looked for in the first server, then the second, and so on.
Setting server_list makes the single-server server setting ignored entirely. There is no implicit fallback to public Galaxy once you have a list — if you want it, it must be in the list, and that is a decision worth making explicitly rather than inheriting.

Why a run-time install is worse than it looks in CI

A pipeline stage that does ansible-galaxy collection install -r requirements.yml before every playbook run seems harmless: it is declarative, it is in the pipeline definition, it is reproducible if the file is pinned.

Three things are still wrong with it.

The install is unverified. Nothing checks signatures unless you configured a keyring, and nothing compares against a known-good artefact — you are trusting whatever the endpoint returns, on every run, including the run during your change window.

It couples change velocity to somebody else’s uptime. An incident where you need to push a fix to 300 hosts is exactly when you do not want the first step to be a download from a public service.

It hides transitive drift. Even with pinned direct dependencies, lesson 4’s >=1.0.0 transitive resolves at install time — so a per-run install re-resolves it per run. The controller’s actual dependency set becomes a function of the clock.

The alternative that fixes all three at once is to stop installing at run time: bake the collections into an image or a bundle, verify once, and have the run consume a fixed environment. That is the execution environment argument, and Part L develops it as a disaster-recovery requirement rather than a convenience.

What happens if the air-gapped side runs a different ansible-core?

Collections are mostly Python and mostly portable, but requires_ansible is a real constraint and the failure is deferred.

ansible-galaxy checks requires_ansible at install time and warns — or in the case of a collection built without the metadata, warns about that instead:

[WARNING]: example_corp.platform:1.0.0 does not have requires_ansible metadata.

What it cannot check is whether the collection works on your core version, only what it claims. A collection tested against 2.15–2.18 and installed on 2.21 may use a module_utils interface that changed, and you find out when a module fails on a managed host rather than at install.

The discipline that avoids this:

Build the bundle on a machine running the same ansible-core version as the target controller, so requires_ansible conflicts surface on the connected side where you can do something about them.
Pin ansible-core in requirements.txt on both sides, and treat the pair of files as one artefact.
Run the estate’s playbooks in --check and --syntax-check on the air-gapped controller before the first real run, because that is the only integration test available there.

The offline bundle

Private Galaxy servers

Knowledge check