Runbook: Install and build an Ansible controller

A controller is not a machine with Ansible installed on it. It is a machine with a specific Ansible, a specific set of collections, a specific checkout of your automation, and credentials that let it change every host in the estate. Get the first three wrong and your playbooks behave differently here than they did in CI. Get the fourth wrong and this box becomes the most dangerous host you own.

This runbook builds one from nothing and stops at the point where it has proven, read-only, that it can see the fleet. It deliberately does not make a change to any managed host.

When to use this runbook

• Building the first controller for a new estate.
• Replacing a controller that is being decommissioned.
• Rebuilding after a controller loss (use this together with the controller recovery runbook, which covers what to do about state that only existed on the lost machine).
• Standing up a second controller for a separate environment.

Blast radius

Building the controller: zero managed hosts. Nothing in Steps 1 to 7 connects to the fleet.

From Step 8 onwards this host can reach every host the inventory names. The moment the private key in Step 7 is in place, an accidental ansible all -m shell -a ... is a fleet-wide command. Treat the credential install as the point where the blast radius changes from nothing to everything.

Inputs

Gather before starting:

• The ansible-core version this estate declares. Not “latest”.
• The repository URL and the production branch name.
• The path convention for the checkout, so that every controller agrees.
• The automation account name.
• The vault password, from the password manager.
• The automation SSH private key, or a decision to generate a new one.

Step 1: Record the target version and check the interpreter

python3 --version
python3 -c 'import sys; print(sys.version_info >= (3, 12))'

ansible-core 2.21 requires Python 3.12 or newer on the controller. Managed nodes are a separate matrix and may run considerably older Python. If the host cannot offer a supported interpreter, stop here: the correct fix is a different host or a newer interpreter package, not an older ansible-core that the rest of the estate does not run.

Write the intended version down before you install anything. The number you type in Step 3 is the thing this whole procedure exists to pin.

Step 2: Create the automation account

sudo useradd --create-home --shell /bin/bash --comment 'Ansible automation' ansible
sudo install -d -o ansible -g ansible -m 0700 /home/ansible/.ssh
sudo install -d -o ansible -g ansible -m 0750 /srv/automation

Do not run automation as root and do not run it as a shared human login. The account exists so that “who ran this” has an answer, and so that the private key has an owner that is not a person.

Step 3: Isolated interpreter, pinned core

sudo -iu ansible
python3 -m venv /home/ansible/.venvs/ansible
/home/ansible/.venvs/ansible/bin/pip install --upgrade pip
/home/ansible/.venvs/ansible/bin/pip install 'ansible-core==2.21.3'

# Put it on PATH for this account only
echo 'export PATH="$HOME/.venvs/ansible/bin:$PATH"' >> /home/ansible/.bashrc

Pin with ==, not ~= and not a bare name. A controller that installed “whatever was current on the day it was built” is a controller whose behaviour you cannot reproduce, and the difference shows up as a playbook that works on one controller and fails on another.

If your estate installs the full ansible package rather than ansible-core, pin that instead - but pin exactly one of them. Both installed in the same environment is a resolution problem you will diagnose at the worst possible moment.

ansible --version

Read every line of that output, not just the version. config file, ansible python module location and python version are all things that will differ between two controllers that “both have Ansible”.

Step 4: Clone the automation repository

git clone --branch main https://git.example.com/infra/automation.git /srv/automation/repo
cd /srv/automation/repo
git log -1 --format='%H %ci %s'

Record that commit hash. It is the answer to “what was this controller running” for every audit question that follows, and it is the first thing to compare when two controllers disagree.

Use a fixed path that every controller shares. A repository checked out under one operator’s home directory is a controller that stops working when that person leaves.

Step 5: Install pinned collections

# requirements.yml lives in the repository and pins exact versions
cat requirements.yml

ansible-galaxy collection install -r requirements.yml -p ./collections

A requirements.yml that pins version: ">=1.0.0" is not pinned. The whole point of building the controller from a requirements file is that the next rebuild produces the same tree, and a floating range breaks that on a schedule you do not control.

ansible-galaxy collection list

Compare that output against requirements.yml line by line. Extra collections are as much of a finding as missing ones: something installed them, and whatever that was is not in your build procedure.

Step 6: Write ansible.cfg and confirm which one is live

[defaults]
inventory = ./inventories/production
collections_path = ./collections
roles_path = ./roles
host_key_checking = True
forks = 20
log_path = /var/log/ansible/ansible.log
interpreter_python = auto_silent

[privilege_escalation]
become = False

The file belongs in the repository, at the repository root, so that the configuration is versioned with the automation it configures.

ansible --version | head -2
ansible-config dump --only-changed

--only-changed prints every setting that differs from the default, and the file each one came from. That second half is the useful part: it is how you prove the config being used is the one in the repository and not a leftover /etc/ansible/ansible.cfg or a stray file in a directory you happened to be standing in.

host_key_checking = True stays on. A controller that accepts unknown host keys silently accepts a machine-in-the-middle for every host it has not seen before, and this is exactly the host where that matters most.

Step 7: Install credentials

This is the step that changes the blast radius. Everything before it was inert.

# Private key for the automation account
install -m 0600 -o ansible -g ansible /dev/null /home/ansible/.ssh/id_ed25519
# paste the key material, then verify what you installed
ssh-keygen -lf /home/ansible/.ssh/id_ed25519

# Vault password file, readable only by the automation account
install -m 0600 -o ansible -g ansible /dev/null /home/ansible/.vault-pass
# write the password into it from the password manager, then:
stat -c '%a %U:%G %n' /home/ansible/.vault-pass /home/ansible/.ssh/id_ed25519

Expected output is 600 ansible:ansible for both. Anything else and the credential is readable by someone who should not have it, or sshd will refuse the key.

Never echo the vault password on the command line. It lands in shell history, in the process table for the duration, and in any audit log that records command lines.

ansible-vault view --vault-password-file /home/ansible/.vault-pass \
inventories/production/group_vars/all/vault.yml

If that fails with Decryption failed (no vault secrets were found that could decrypt), the password is wrong or the file was encrypted under a different vault ID. Resolve it now. A controller that cannot decrypt is a controller that fails on its first real run, at the least convenient moment.

Step 8: The read-only proof sequence

Run these in order. Each one answers a question the next one depends on.

# 1. Which Ansible, which config?
ansible --version

# 2. What does the inventory actually contain?
ansible-inventory --graph

# 3. What would this playbook target?
ansible-playbook site.yml --list-hosts

# 4. Can we reach them? (this one connects)
ansible all -m ping -o

Step 4 is the only one that touches the network, and it is the only one that proves reachability. A “can we see the fleet?” check built from debug proves nothing at all - debug runs on the controller and never opens a connection, so a genuinely unreachable host reports unreachable=0. Use a module that connects.

Expected shape of the recap on a healthy build:

web01.example.com | SUCCESS => {"changed": false, "ping": "pong"}
web02.example.com | SUCCESS => {"changed": false, "ping": "pong"}
db01.example.com  | SUCCESS => {"changed": false, "ping": "pong"}

ansible exits 4 if any host was unreachable, 2 on task failure, and 0 on success. Verified on 2.21.3. Check the exit code, not the colour of the output.

Step 9: One check-mode run against one host

ansible-playbook site.yml --limit staging-web01.example.com --check --diff

Pick a non-production host. What you are testing is the controller, not the playbook, so the smallest possible target is the right one.

Read the diff. On a correctly built controller against an already converged host, the expected result is changed=0. Any change reported here is a divergence between this controller and whatever last managed that host - a different collection version, a different variable file, a different branch. Investigate it before Step 10 rather than discovering it during a fleet run.

Step 10: Register the controller

Record, in the same place the estate records servers:

• Hostname, and which environments it is permitted to target.
• The ansible-core version and the commit hash from Step 4.
• Who may sudo -iu ansible on it.
• The fingerprint of the automation public key it holds.
• Whether it replaced another controller, and whether that one was decommissioned.

An unregistered controller is a machine with fleet-wide credentials that nobody is tracking. That is the finding, not a documentation gap.

Rollback

Nothing on the managed fleet changed, so rollback is removal:

sudo systemctl stop 'ansible-*.timer' 2>/dev/null || true
sudo rm -rf /home/ansible/.venvs/ansible /srv/automation/repo
sudo shred -u /home/ansible/.vault-pass /home/ansible/.ssh/id_ed25519
sudo userdel -r ansible

Use shred -u on the credentials rather than rm. Then confirm the half-built home directory was not caught by a backup job that ran in the meantime; if it was, the vault password is now in the backup system and the rotation runbook applies.

The one thing that does not roll back by deleting this host: if you generated a new key here and its public half was already pushed to managed nodes, those hosts still trust a key that no longer has an owner. That is a fleet change and it needs the SSH credential rotation runbook, not userdel.

Common patterns

Escalation

Escalate when:

• The declared ansible-core version will not install on any available interpreter. Do not silently install a different one.
• The vault password is not in the password manager. Copying it out of another controller’s filesystem is not a build step, it is an undocumented secret transfer.
• A pinned collection cannot be resolved. Do not relax the pin to make the install succeed.
• The estate already has a controller and nobody has decided whether this one replaces it. Two controllers holding the same fleet credentials is a change-control decision.