Break/Fix: it works on my machine, and on the CI runner it connects to production with no vault and five forks · Break/Fix · Ansible

Reported symptoms

A pipeline that has worked for a year starts failing on the CI runner. The same commit works on every engineer workstation.

The failures do not look like one problem:

The run fails with Attempting to decrypt but no vault secrets found, against vaulted files nobody has touched.
Before it stops, it is roughly ten times slower than the same run on a workstation.
The verbose output shows it connecting to hosts that the repository inventory configuration was supposed to exclude.
Host key checking behaves differently.

Four symptoms, four theories, four people. The vault password file is present and readable on the runner. The inventory is committed. Nothing in the repository changed.

Somebody checks that ansible.cfg exists in the workspace on the runner. It does. Its contents are byte-identical to the copy in the repository. That is where the investigation stops for two hours.

Evidence provided

Read-only / Safeon an engineer workstation

$ ansible-config dump --only-changed

CONFIG_FILE() = /srv/automation/ansible.cfg
DEFAULT_FORKS(/srv/automation/ansible.cfg) = 50
DEFAULT_VAULT_PASSWORD_FILE(/srv/automation/ansible.cfg) = /etc/ansible/vault-pass
DEFAULT_HOST_LIST(/srv/automation/ansible.cfg) = ['/srv/automation/inventory']

Read-only / Safeon the CI runner, in the same commit

$ ansible-config dump --only-changed

[WARNING]: Ansible is being run in a world writable directory (/builds/ops/automation), ignoring it as an ansible.cfg source. For more information see https://docs.ansible.com/ansible/devel/reference_appendices/config.html#cfg-in-world-writable-dir
CONFIG_FILE() = None

Read-only / Safethe file is present and correct, which is what stalled the investigation

$ ls -l ansible.cfg; md5sum ansible.cfg

-rw-r--r-- 1 build build 214 Aug 11 09:02 ansible.cfg
5d41402abc4b2a76b9719d911017c592  ansible.cfg

Read-only / Safethe directory, not the file

$ stat -c '%a %n' .

777 .

Read-only / Safepermissions preserved from the archive, applied to the workspace

$ grep -n -A3 'unpack' .gitlab-ci.yml

  script:
- tar -xf artefacts/dependencies.tar -C .

Read-only / Safeno ANSIBLE_CONFIG to fall back to

$ env | grep -i ansible

Work the evidence before reading on

The file is present, correct and identical. The ansible-config output disagrees with all three of those facts being sufficient.

Compare the two ansible-config dump outputs. One names a configuration file and the other does not. What does CONFIG_FILE() = None mean for every setting the file was providing?
Read the warning line above it. It names a directory, not a file. Check the permissions of that directory rather than of the file.
Four symptoms appeared at once. What single change would produce all four simultaneously?

Before continuing: why would Ansible deliberately refuse to read a configuration file that is present and valid?

Root cause

1. Ansible refuses configuration from a world-writable directory

Ansible discovers ansible.cfg in the current working directory, among other places. It will not load one from a directory that is world-writable, and it says so:

[WARNING]: Ansible is being run in a world writable directory (/builds/ops/automation), ignoring it as an ansible.cfg source.

The reason is a genuine security boundary. A configuration file determines which plugin paths are searched, which Python interpreter runs on targets, where the vault password comes from, and which inventory is used. Honouring one that any local user could rewrite would let any local user redirect an automation run.

The refusal is about the directory, not the file. A perfectly permissioned ansible.cfg inside a world-writable directory is still ignored, because anyone who can write to the directory can replace the file.

2. Every setting reverted at once

The runner has no ANSIBLE_CONFIG environment variable and no user-level or system-level configuration file. So when the repository configuration was skipped, there was nothing to take its place and every setting fell back to its built-in default simultaneously:

Setting	Repository value	Fallback
Vault password file	a path	none, so decryption fails
Inventory	the repository inventory	the built-in default
Forks	50	5
Host key checking	as configured	the built-in default

That is why it presented as four unrelated faults. One cause, four visible effects, and each effect plausible enough on its own to sustain its own theory.

3. The workspace permissions came from a build step

A build step unpacks a dependency archive into the workspace, preserving the permissions recorded in the archive. Those permissions include a world-writable mode on the extraction target.

Nothing about the Ansible configuration changed. The directory it lives in did, and it will change again on the next build unless the build step is fixed.

Under the hood

Ansible looks for a configuration file in a fixed order and uses the first one it finds:

ANSIBLE_CONFIG, if set - a file path or a directory containing an ansible.cfg.
ansible.cfg in the current working directory.
~/.ansible.cfg.
/etc/ansible/ansible.cfg.

Two properties of that list cause most of the confusion in practice.

The first is that it is first-match-wins, not merged. The chosen file is the only file. A setting absent from it does not fall back to the same setting in a later file; it falls back to the built-in default. That is why skipping one file changed four unrelated behaviours at once.

The second is that step 2 depends on the current working directory, which in a pipeline is whatever the previous step left it as. Two jobs running the same commit from different directories can use different configurations, and neither is doing anything wrong.

ansible-config dump --only-changed resolves all of this and prints the source of every non-default value. It is the single most useful diagnostic in this whole class of problem, and it belongs at the top of every pipeline log rather than being reached for after two hours.

Present, valid and not read

The investigation stalled because everybody verified the file and nobody asked whether it was being used. Those are different questions and only one of them was answered.

The pattern recurs constantly outside Ansible: a sudoers drop-in whose filename disqualifies it, a systemd unit in a directory that is not searched, a sysctl.d file that a later file overrides, an environment file the service manager never sources. In every case the content is correct and the system disagrees.

The habit is to verify by rendering rather than by reading. For Ansible configuration that is ansible-config dump --only-changed, which names the source of every value. For a sudoers rule it is sudo -l -U user. For a systemd unit it is systemctl cat. For a service configuration it is whatever the daemon reports as its running configuration.

Reading the file tells you what somebody intended. Only the renderer tells you what the system will do.

Resolution

Establish what the runs under default settings actually did. A run with the wrong inventory and the wrong connection settings may have reached hosts the repository configuration was there to exclude; that is the part with consequences beyond a red build.
Fix the workspace permissions so the directory is not world-writable, and confirm ansible-config dump --only-changed now names the intended file.
Fix the build step that created the permissions. A workspace recreated by the next build will be world-writable again, and the incident will return looking like a flaky pipeline.
Set ANSIBLE_CONFIG explicitly in the CI job to an absolute path. This removes the dependency on the working directory and on its permissions entirely, and it is the durable fix rather than the immediate one.
Add a pipeline step that prints ansible-config dump --only-changed into the log before anything runs, so the effective configuration is recorded with every run.
Add an assertion that the resolved configuration file is the expected one, and fail the build when it is not.
Check the other pipelines and scheduled jobs on the same runner for the same dependency on directory-relative discovery.
Treat the world-writable workspace as a security finding in its own right and raise it separately - any local user on that runner could have replaced the configuration or the playbooks.

Verification

The intended configuration file is in force. ansible-config dump --only-changed on the runner names it, and every setting the repository provides appears with that file as its source.
The check can fail. Make a scratch directory world-writable, run the same command there, and confirm the warning returns and CONFIG_FILE() reports None. Watching the failure is how you know the check means something.
The pipeline assertion goes red. Point it at a directory with the wrong permissions on a scratch branch and confirm the build fails before any play runs.
The individual settings that matter are correct. Confirm the vault identity, the inventory path and the forks value by name; a named configuration file is not proof that every setting arrived, since the file could also be the wrong one.
The permissions survive a fresh build. Trigger a build from clean and confirm the workspace is not world-writable afterwards, which tests the build-step fix rather than the manual one.
The run succeeds end to end with the vault, the correct inventory and the configured concurrency, and takes a duration consistent with 50 forks rather than 5.
No other job on the runner depends on directory-relative discovery. Each one either sets ANSIBLE_CONFIG or asserts the resolved file.

Prevention

Set ANSIBLE_CONFIG explicitly in every pipeline and scheduled job. Directory-relative discovery is convenient at a terminal and inappropriate in automation, where the working directory is decided by whatever ran last.
Print the effective configuration at the start of every run. ansible-config dump --only-changed in the log turns this entire class of problem into a two-second comparison.
Assert the resolved configuration file in CI, so a run with the wrong configuration fails before it connects to anything.
Keep workspace permissions tight, and treat a world-writable checkout as a build failure. Ansible refusing to read it is a symptom; the security exposure is the finding.
Remember that configuration discovery is first-match-wins, not merged. Losing one file does not partially degrade behaviour, it reverts every setting that file provided.
When the same commit behaves differently on two machines, compare effective configuration first. The code is provably identical, so the difference is in the environment, and this is where it usually is.
Be suspicious of several unrelated symptoms appearing together. One cause with four effects is more likely than four causes arriving on the same afternoon.

What if a user-level ansible.cfg had existed on the runner?

Then the failure would have been quieter and considerably harder, because Ansible would have found a configuration file and used it.

ansible-config dump --only-changed would have named ~/.ansible.cfg rather than reporting None, and there would have been no warning to notice. Some settings would have matched the repository by coincidence and others would not, so instead of four symptoms arriving together you would get one or two intermittent ones - a run that uses the right inventory and the wrong vault identity, for instance.

That is why the verification step names individual settings rather than stopping at “a configuration file was found”. The question is not whether Ansible read a configuration file. It is whether it read the one you intended, and whether the values in force are the values you wrote.

The same reasoning applies to environment variables, which sit above all configuration files in precedence. A stray ANSIBLE_INVENTORY or ANSIBLE_VAULT_PASSWORD_FILE exported by an earlier pipeline step overrides the file entirely and appears in ansible-config dump as an environment-sourced value. Printing that output at the top of the log catches it; nothing else will.

Why this matters

A configuration file that is present, valid, committed and ignored is a particularly effective way to lose two hours, because every check somebody thinks to run confirms that the file is fine. The file was always fine. The directory was not.

The specific fact - that Ansible refuses configuration from a world-writable directory - is worth knowing because the refusal is correct and the warning is easy to scroll past. The general habit matters more: verify configuration by asking the system what it resolved, not by reading what you wrote.

And in automation, remove the ambiguity rather than diagnosing it. Directory-relative discovery is a convenience that assumes a human at a terminal who knows where they are. A pipeline does not know where it is, and setting ANSIBLE_CONFIG to an absolute path costs one line and retires the whole question.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. Ansible refuses configuration from a world-writable directory

2. Every setting reverted at once

3. The workspace permissions came from a build step

Resolution

Verification

Prevention