AnsibleL · Automation Disaster RecoveryWhat the controller actually held
What belongs in the repository, and what must not
What you'll learn
- List what the repository must contain for a controller rebuild to be possible
- List what must never be committed, and name where each item lives instead
- Apply the two-failure test that decides which side of the line an artefact belongs on
- Audit a repository for the three categories of accidental controller state
Prerequisites
Verified against ansible-core 2.21.x · ansible (community package) 14.x · Python (controller) 3.12+ · ansible-lint 26.x · Molecule 26.x · Ubuntu 24.04 LTS · Debian 12 (Bookworm) · RHEL / Rocky / AlmaLinux 9.x · 2026-08-11
The repository is the recovery plan. Not the backup, not the runbook — the repository, because it is the only artefact that already exists on more than one machine as a matter of routine and is already replicated by a system somebody else operates.
That makes the contents of the repository a DR decision rather than a tidiness decision. Everything in it survives the controller. Everything outside it has to be explicitly arranged for, or it does not.
There is a second requirement pulling the other way, and the two have to be satisfied at the same time:
What must be in the repository
- Playbooks, including the site composition that imports them - a playbook that only ever existed as a file on the controller is a playbook you will rewrite from memory.
- Roles, including their defaults, handlers, templates and files. If a role is consumed from Galaxy or an internal server, its pinned entry in requirements.yml stands in for it.
- Static inventory files, group_vars and host_vars - excluding the plaintext secrets, which are covered below.
- Dynamic-inventory plugin configuration: the .aws_ec2.yml or .proxmox.yml file that tells the plugin what to query. The credentials it uses do not go here.
- The project ansible.cfg. This is the settings file the fleet behaviour depends on, and Part VI showed how much it changes.
- requirements.yml, with versions pinned. Without this the collection set is unrecoverable knowledge.
- Vault-ENCRYPTED variable files and encrypt_string values. These are safe to commit; that is the entire point of encrypting them.
- The controller build definition: the pinned dependency list from lesson 3, or the execution-environment.yml from lesson 5.
- The rebuild runbook itself, so that the instructions for restoring the controller are not stored on the controller.
The last item is the one teams forget, and it has the flavour of a joke until it happens. A rebuild runbook stored on a wiki that authenticates against an LDAP server managed by the automation you cannot run is the same failure in a different suit.
What must never be in the repository
| Never commit | Where it lives instead |
|---|---|
| The vault password, in any form | Escrow (lesson 4), a secret manager, or an operator’s password manager |
A vault_password_file containing the password | Same. Committing the path in ansible.cfg is fine; committing the file is not |
| SSH private keys for the automation account | Escrow, or generated fresh per controller with the public key deployed by an existing play |
| API tokens for dynamic inventory or cloud modules | The environment, a secret manager, or an encrypted vault file |
| Become/sudo passwords in plaintext | A vault file, or eliminated with NOPASSWD for specific commands |
~/.ssh/known_hosts | Regenerated; see the note below on why committing it is tempting and wrong |
| Anything the CI system decrypts and prints | Nothing changes about the file; the leak is the pipeline, and Part XLVIII covers it |
The three categories of accidental controller state
The visible failures are easy. What breaks rebuilds is state that accumulated on the controller without anyone deciding to put it there.
1. Files Git was never told about
cd /srv/ansible
# Everything git does not track, file by file.
git status --short --untracked-files=all
# Files an ignore rule is deliberately hiding. Read every line.
git status --short --ignored
# Local commits that were never pushed anywhere.
git log --branches --not --remotes --onelineThe --ignored listing is the interesting one. A .gitignore entry
for *.local.yml or secrets/ is usually correct and usually also the
place where a genuinely needed file has been quietly excluded. Read the
list and decide, per file, whether it is regenerable, escrowed, or
neither.
2. Configuration that is not the project’s
ansible.cfg has a search order, and only one position in it is inside
the repository.
$ ansible --versionansible [core 2.21.3]
config file = None
configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible collection location = /home/ansible/.ansible/collections:/usr/share/ansible/collections
python version = 3.14.4The search order — ANSIBLE_CONFIG, then ./ansible.cfg, then
~/.ansible.cfg, then /etc/ansible/ansible.cfg — was covered in
Part VI. The DR consequence is one sentence: if the effective config
file is not the one in the repository, the rebuilt controller behaves
differently from the destroyed one, and nothing will tell you.
The settings that bite are the quiet ones. A ~/.ansible.cfg carrying
host_key_checking = False or a raised forks or a roles_path
pointing at a directory outside the project does not fail on the new
controller; it just behaves differently, and the difference shows up as
a run that is slower, or that fails a host-key prompt, or that cannot
find a role.
ansible-config dump --only-changed
# The same, but showing which source set each value.
ansible-config dump --only-changed -vAnything in that output that is not explained by the committed
ansible.cfg is undocumented controller state.
3. Software installed by hand
$ ansible-galaxy collection listCompare that listing against requirements.yml by hand once. The
difference is the set of collections somebody installed during an
incident and never recorded. Lesson 3 makes this comparison
systematic; here it is one of the three audits.
The audit, as a single pass
- git status --short --untracked-files=all - decide, per file: commit it, escrow it, or accept losing it.
- git status --short --ignored - read every ignored path and confirm nothing needed is hidden there.
- git log --branches --not --remotes --oneline - push it, or accept losing it.
- ansible --version | grep "config file" - confirm it names the repository copy.
- ansible-config dump --only-changed - confirm every changed setting is explained by the committed ansible.cfg.
- ansible-galaxy collection list - confirm every collection appears in requirements.yml with a version.
- grep the repository for anything that looks like a credential, then look again at group_vars and host_vars specifically.
- Read the rebuild runbook and confirm no step depends on a system that depends on the controller.
Eight commands, all read-only, and it takes about twenty minutes. It is worth scheduling quarterly, because every item on it drifts.
Knowledge check
Knowledge check · 5 questions
Q1. A dynamic-inventory configuration file for a cloud plugin specifies which regions and tags to query, and reads its API token from an environment variable. How should it be treated?
Q2. Which of these are genuine forms of undocumented controller state that would break a rebuild? Select all that apply.
Q3. A credential accidentally committed and then removed in the following commit is adequately remediated, because the current tree no longer contains it.
Q4. Why does the lesson insist that the rebuild runbook itself lives in the repository rather than on the team wiki?
Q5. What does `ansible-config dump --only-changed` contribute to a DR audit that reading the committed ansible.cfg does not?
Passing score: 75%. Answers are checked in this browser.