Lab: Find the configuration that is actually in effect

Objective

By the end of this lab you will be able to answer, for any Ansible invocation on any machine, two questions with command output rather than inference: which configuration file is in effect, and where did this particular setting come from. You will also have reproduced the failure mode where a perfectly valid project ansible.cfg is ignored because of a directory permission, and seen exactly what Ansible prints when it happens.

Architecture

Four configuration files in the four locations Ansible searches, all present at once, all setting the same option to different values.

ANSIBLE_CONFIG=/opt/lab/ci.cfg     forks = 50   <- highest precedence
./ansible.cfg                      forks = 25
~/.ansible.cfg                     forks = 10
/etc/ansible/ansible.cfg           forks = 5    <- lowest precedence

Only one of these is ever read. The lab is about finding out which, and about the fact that the other three contribute nothing at all — not even the settings the winner does not mention.

Requirements

A controller with ansible-core 2.21.x. Output below captured from 2.21.3.
Write access to your own home directory and a scratch directory. The /etc/ansible/ansible.cfg layer requires root; Task 3 gives a root-free substitute for readers who do not have it or do not want to touch a shared path.
No SSH, no managed nodes, no connections. Nothing in this lab can lock you out of anything.

Scenario

A colleague reports that a playbook “runs at the wrong parallelism on the build box and the right parallelism on my laptop, and the ansible.cfg is identical in the repo”. Nobody has changed the repo. Your job is to find out what is different, and to build the habit that would have answered the question in ten seconds.

Tasks

Task 1: Capture the starting state

Before creating anything, record what exists. Cleanup depends on this.

WORKDIR="$HOME/ansible-config-lab"
mkdir -p "$WORKDIR/backup"
cd "$WORKDIR"

# Which config, if any, is in effect right now
ansible --version | grep 'config file' | tee backup/pre-lab-configfile.txt

# What it changes from the defaults
ansible-config dump --only-changed > backup/pre-lab-dump.txt

# Preserve any existing user config, with mode and timestamps
if [ -f "$HOME/.ansible.cfg" ]; then
  cp -a "$HOME/.ansible.cfg" backup/user-ansible.cfg
  echo "existing user config saved"
else
  echo "no existing user config" > backup/no-user-config
fi

# And any existing system config, if it is readable
if [ -f /etc/ansible/ansible.cfg ]; then
  cp -a /etc/ansible/ansible.cfg backup/system-ansible.cfg
  echo "existing system config saved"
else
  echo "no existing system config" > backup/no-system-config
fi

ls -la backup/

Task 2: Seed the competing configurations

cd "$HOME/ansible-config-lab"
mkdir -p project alt

Write project/ansible.cfg:

[defaults]
forks = 25
host_key_checking = False
stdout_callback = default

Write alt/ci.cfg:

[defaults]
forks = 50

Write ~/.ansible.cfg:

[defaults]
forks = 10
gathering = explicit

Notice that each file sets forks and each sets one other thing that the others do not. That asymmetry is what exposes the merging question.

Task 3: Establish which one wins

ansible-config dump --only-changed prints every setting that differs from the built-in default, and — crucially — the origin of each in parentheses.

Read-only / Safecontroller — inside project/— Reports the effective config and the origin of every non-default setting

$ cd project && ansible-config dump --only-changed

CONFIG_FILE() = /home/operator/ansible-config-lab/project/ansible.cfg
DEFAULT_FORKS(/home/operator/ansible-config-lab/project/ansible.cfg) = 25
HOST_KEY_CHECKING(/home/operator/ansible-config-lab/project/ansible.cfg) = False

GALAXY_SERVERS:

Read that output carefully, because the interesting thing is what is missing.

gathering = explicit from ~/.ansible.cfg does not appear. The project config did not override it — the project config replaced the entire file. ~/.ansible.cfg was not read at all.

Now show ANSIBLE_CONFIG taking over:

Read-only / Safecontroller — inside project/— ANSIBLE_CONFIG outranks the working-directory config

$ ANSIBLE_CONFIG=../alt/ci.cfg ansible-config dump --only-changed

CONFIG_FILE() = /home/operator/ansible-config-lab/alt/ci.cfg
DEFAULT_FORKS(/home/operator/ansible-config-lab/alt/ci.cfg) = 50

GALAXY_SERVERS:

host_key_checking = False from the project config is gone too. Same mechanism: the environment variable named a different file, so the project file was never read.

Task 4: Reproduce the world-writable skip

There is one more way a config file goes missing, and it is stranger than the precedence rules.

cd "$HOME/ansible-config-lab/project"

# Confirm the config is in effect right now
ansible-config dump --only-changed | head -1

Now make the directory world-writable — not the file:

chmod 777 "$HOME/ansible-config-lab/project"

ansible-config dump --only-changed | head -3

Read-only / Safecontroller— The config file is present, readable, valid — and ignored

$ ansible-config dump --only-changed | head -3

[WARNING]: Ansible is being run in a world writable directory (/home/operator/ansible-config-lab/project), ignoring it as an ansible.cfg source. For more information see https://docs.ansible.com/ansible/devel/reference_appendices/config.html#cfg-in-world-writable-dir
CONFIG_FILE() = None

GALAXY_SERVERS:

CONFIG_FILE() = None. The file is still there, still valid, still readable. Ansible declined to read it because anyone on the machine could have written it — and a config file controls which plugins load, so trusting a world-writable one is a code-execution path.

Restore the permission and confirm it comes back:

chmod 755 "$HOME/ansible-config-lab/project"
ansible-config dump --only-changed | head -2

Task 5: Trace one setting through all four layers

Pick forks and walk it up. Each command below overrides the one above.

cd "$HOME/ansible-config-lab/project"

# 1. From the working-directory config file
ansible-config dump --only-changed | grep FORKS

# 2. From an environment variable
ANSIBLE_FORKS=7 ansible-config dump --only-changed | grep FORKS

# 3. From ANSIBLE_CONFIG naming a different file
ANSIBLE_CONFIG=../alt/ci.cfg ansible-config dump --only-changed | grep FORKS

# 4. Environment variable versus ANSIBLE_CONFIG: which wins?
ANSIBLE_CONFIG=../alt/ci.cfg ANSIBLE_FORKS=7 \
  ansible-config dump --only-changed | grep FORKS

The output annotates its own origin:

Read-only / Safecontroller— The parenthesised origin is the answer to 'where did this come from'

$ ANSIBLE_FORKS=7 ansible-config dump --only-changed | grep FORKS

DEFAULT_FORKS(env: ANSIBLE_FORKS) = 7

(env: ANSIBLE_FORKS) rather than a file path. That parenthesis is the whole diagnostic — you never have to guess.

Record your findings in findings.md:

Setting	Value in effect	Origin reported by `ansible-config`
`DEFAULT_FORKS`
`HOST_KEY_CHECKING`
`DEFAULT_GATHERING`
`CONFIG_FILE`

Task 6: Answer the colleague’s question

Write two sentences in findings.md explaining what is different between the build box and the laptop, and name the single command that would have found it. Then write the check you would add to the pipeline.

cd "$HOME/ansible-config-lab"

cat > preflight-config-check.sh <<'SCRIPT'
#!/usr/bin/env bash
# Print the effective Ansible configuration at the top of every CI run.
set -euo pipefail

echo "=== effective ansible configuration ==="
ansible --version | grep 'config file'
ansible-config dump --only-changed
echo "======================================="
SCRIPT

chmod 0755 preflight-config-check.sh
./preflight-config-check.sh

Validation

cd project && ansible-config dump --only-changed reports CONFIG_FILE() as the project path, DEFAULT_FORKS as 25, and does not mention gathering — proving ~/.ansible.cfg was not read.
ANSIBLE_CONFIG=../alt/ci.cfg ansible-config dump --only-changed reports DEFAULT_FORKS = 50 and does not mention HOST_KEY_CHECKING.
ANSIBLE_FORKS=7 ansible-config dump --only-changed | grep FORKS prints the origin as (env: ANSIBLE_FORKS).
With chmod 777 on the project directory, ansible-config dump --only-changed prints the world-writable warning and CONFIG_FILE() = None. With chmod 755, the config returns.
findings.md has all four rows of the table filled in from output, not from memory.
preflight-config-check.sh runs and prints the effective config.

Expected Outcome

ansible-config-lab/
├── alt/
│   └── ci.cfg
├── backup/
│   ├── pre-lab-configfile.txt
│   ├── pre-lab-dump.txt
│   └── (user-ansible.cfg / system-ansible.cfg, if they existed)
├── findings.md
├── preflight-config-check.sh
└── project/
    └── ansible.cfg

Plus ~/.ansible.cfg, which Cleanup removes or restores. You can name the effective config file and the origin of any setting from one command, and you know that “the ansible.cfg says X” is not a statement about behaviour until you have checked which ansible.cfg.

Troubleshooting

ansible-config dump --only-changed prints nothing but CONFIG_FILE() = None. Either no config file was found, or the one that would have been found is in a world-writable directory. Check for the warning; it appears on stderr, so 2>&1 if you are piping.

Your project config is ignored and the directory is not 777. Check the parent path components as well; the check applies to the current working directory. Also confirm you are running from the directory you think you are — the working-directory config is resolved from $PWD, not from the playbook’s location.

A setting appears in the file but not in --only-changed. Two possibilities. It equals the built-in default, in which case it is not “changed” and is correctly omitted — check with ansible-config dump | grep NAME. Or the key is in the wrong section: forks under [ssh_connection] rather than [defaults] is read as an unknown key and ignored silently.

ansible-config dump shows a setting you cannot find in any file. Look at the origin in parentheses. (env: ...) means an environment variable, which may have been exported by a shell profile, a direnv hook, or a CI job definition rather than by you.

Cleanup

This lab wrote a file into your home directory and changed a directory permission. Both must be reverted explicitly.

Step 1. Restore the directory permission you changed in Task 4, in case you stopped partway through:

chmod 755 "$HOME/ansible-config-lab/project" 2>/dev/null || true

Step 2. Restore ~/.ansible.cfg to whatever it was before the lab. There are two cases and they must not be confused:

cd "$HOME/ansible-config-lab"

if [ -f backup/user-ansible.cfg ]; then
  # There was one before. Put it back exactly, preserving mode and times.
  cp -a backup/user-ansible.cfg "$HOME/.ansible.cfg"
  echo "restored the pre-lab ~/.ansible.cfg"
elif [ -f backup/no-user-config ]; then
  # There was none. Remove the one the lab created.
  rm -f "$HOME/.ansible.cfg"
  echo "removed the lab's ~/.ansible.cfg"
else
  echo "NO CAPTURE FOUND - do not delete ~/.ansible.cfg by hand."
  echo "Inspect it and decide; Task 1 was skipped."
fi

Step 3. If you created /etc/ansible/ansible.cfg for the optional system layer, restore it the same way. If Task 1 recorded that none existed, remove the one you added:

if [ -f backup/system-ansible.cfg ]; then
  sudo cp -a backup/system-ansible.cfg /etc/ansible/ansible.cfg
elif [ -f backup/no-system-config ]; then
  sudo rm -f /etc/ansible/ansible.cfg
fi

Step 4. Confirm you are back where you started. This is the step that turns cleanup into verified cleanup:

cd "$HOME"
ansible --version | grep 'config file'
ansible-config dump --only-changed > /tmp/post-lab-dump.txt
diff -u "$HOME/ansible-config-lab/backup/pre-lab-dump.txt" /tmp/post-lab-dump.txt \
  && echo 'CONFIGURATION RESTORED'

Step 5. Only once step 4 printed CONFIGURATION RESTORED, remove the lab directory:

rm -rf deletes the whole lab directory, including anything you added

The next block removes ~/ansible-config-lab and everything under it, recursively and without confirmation. If you saved notes, an inventory you want to keep, or a playbook you extended, copy it out first — the step above does that for the lab’s own deliverables and knows nothing about yours.

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep, and the shell gives no second chance.

Confirm what you are about to delete before you delete it:

ls -la "$HOME/ansible-config-lab"

mkdir -p "$HOME/ansible-lab-deliverables"
cp -a "$HOME/ansible-config-lab/findings.md" \
      "$HOME/ansible-config-lab/preflight-config-check.sh" \
      "$HOME/ansible-lab-deliverables/"

rm -rf "$HOME/ansible-config-lab"
rm -f /tmp/post-lab-dump.txt

Under the hood

Ansible’s config loader walks a fixed list of candidate paths and stops at the first one that exists and is acceptable:

the file named by ANSIBLE_CONFIG,
ansible.cfg in the current working directory,
~/.ansible.cfg,
/etc/ansible/ansible.cfg.

“Stops at the first” is doing all the work in that sentence. The result is assigned to CONFIG_FILE, which is why ansible-config dump can print a single path rather than a list — there is only ever one.

The world-writable check applies to candidate 2 only, and it tests the directory rather than the file. The reasoning is a supply-chain one: an ansible.cfg can set library, roles_path, action_plugins, callback_plugins and inventory among others, so a writable config is a writable code path. Someone who can drop a file into your working directory can therefore run code as you. Rather than trying to distinguish the safe settings from the dangerous ones, Ansible refuses the whole file and says so.

Individual settings then resolve through their own precedence: command-line flag beats environment variable beats config file beats built-in default. ansible-config dump reports which of those produced each value in the parentheses after the setting name, which makes it the only tool you need for this class of question.

What You Learned

Only one configuration file is ever read. You watched gathering from ~/.ansible.cfg vanish the moment a project ansible.cfg existed, and host_key_checking vanish the moment ANSIBLE_CONFIG was set.
ansible-config dump --only-changed names the origin of every setting. A file path, or (env: ANSIBLE_FORKS). That parenthesis ends every “where is this coming from” conversation.
A world-writable working directory disables the project config entirely, with a warning on stderr and CONFIG_FILE() = None. You reproduced it and restored it.
CI and a laptop diverge because of ANSIBLE_CONFIG, not because of the repository. You can now find that in one command instead of an afternoon.
Cleanup has to distinguish “restore” from “delete”. Your script branches on whether a ~/.ansible.cfg existed before the lab, because deleting one that was already there would be a worse outcome than the lab itself.

Objectives

Prerequisites