RunBook AcademyAnsible

← All checklists in Ansible

Before deploymentHost

Checklist: Ansible controller production readiness

20 items ·13 critical ·6 warn ·1 info

Run this before a controller is allowed to hold production credentials for the first time, and again after any change to its Python environment, its collections or its ansible.cfg. Run it on the controller itself, as the account that runs the automation - not as your own user, and not under sudo. Several items check file modes and configuration resolution, and both answers change with the account.

What a failure means

The controller is the one machine in the estate that can reach every other machine with elevated privilege. A weakness here is not a controller problem, it is a fleet problem: whoever holds the controller holds every host in the inventory. Treat every critical item as blocking, and resolve it before the controller is given a production vault password.

The warn items are about whether you will be able to reconstruct what happened afterwards. They do not stop a run, they stop an investigation.

Access this needs

Reading the vault password file mode, the SSH key modes and last output all require the automation account. logrotate and filesystem checks need read access to /etc. Nothing in this checklist writes anything, and nothing in it contacts a managed node.

Where the evidence goes

Capture the full output into the commissioning ticket for the controller, with the date and the account it was run as. The two items with no command - out-of-band access and a tested rebuild - are attested by a named person, not by a command, so record who attested them.

Sign-off

Operator: _________________ Date: ___________
Reviewer: ________________ Date: ___________

Critical13 items

The ansible-core version the controller actually runs is the version pinned in requirements.txt. A controller that drifted one minor version ahead of the pin runs different module code than CI validated, and the difference surfaces as a behaviour change on production hosts rather than as a test failure.
```
have=$(ansible --version | sed -n '1s/.*\[core \(.*\)\].*/\1/p')
want=$(sed -n 's/^ansible-core==//p' requirements.txt)
[ "$have" = "$want" ] || echo "FINDING: controller runs core $have, requirements.txt pins $want"
```
Every pin in requirements.txt is installed at exactly that version in the virtualenv the automation runs from. Prints one line per pin that is missing or at a different version; no output is the pass condition.
```
comm -23 <(grep -vE '^[[:space:]]*(#|$)' requirements.txt | tr 'A-Z' 'a-z' | LC_ALL=C sort) \
         <(pip freeze | tr 'A-Z' 'a-z' | LC_ALL=C sort)
```

Every collection in requirements.yml carries an explicit version, and the controller has that version installed. An unpinned collection resolves to whatever Galaxy published most recently, which means two controllers running the same commit can behave differently.

python3 - <<'PY'
import json, subprocess, yaml
out = subprocess.run(['ansible-galaxy', 'collection', 'list', '--format', 'json'],
                     capture_output=True, text=True).stdout or '{}'
have = {n: v['version'] for p in json.loads(out).values() for n, v in p.items()}
for c in (yaml.safe_load(open('requirements.yml')) or {}).get('collections') or []:
    name, want = (c['name'], c.get('version')) if isinstance(c, dict) else (c, None)
    if want is None:
        print(f'UNPINNED: {name} has no version in requirements.yml')
    elif have.get(name) != want:
        print(f'DRIFT: {name} pinned to {want}, controller has {have.get(name)}')
PY

ansible-config validate exits 0. A misspelled key in ansible.cfg is silently ignored at runtime, so a controller with `host_key_chekcing = True` has no host key checking and no error message. This is the only command that catches it.
```
ansible-config validate
```
Host key checking is enabled. Disabling it is the single most common controller weakening, and it turns a machine-in-the-middle on the management network into fleet-wide code execution. Prints the offending setting and the file it came from; no output is the pass condition.
```
ansible-config dump | awk '/^HOST_KEY_CHECKING/ && !/= True/ {print "FINDING: " $0}'
```

The vault password file is mode 0600 or 0400 and owned by the automation account. Prints the mode when it is anything else; no output is the pass condition.

stat -c '%a %U %n' "$(ansible-config dump | sed -n 's/^DEFAULT_VAULT_PASSWORD_FILE([^)]*) = //p')" 2>/dev/null \
  | awk '$1 != "600" && $1 != "400" {print "FINDING: vault password file is mode " $1 " owned by " $2 ": " $3}'

No vault password, and no path to one, is committed to the repository. A password file inside the working tree is one `git add -A` away from being published.

grep -rInE 'vault[-_ ]?(pass|password)' --include='*.yml' --include='*.yaml' --include='*.cfg' \
  --include='*.sh' --exclude-dir=.git . | grep -v 'vault_password_file[[:space:]]*=[[:space:]]*[~/]'

Every private key the controller uses is mode 0600 or 0400. Prints one line per key that is not; no output is the pass condition.

find ~/.ssh -maxdepth 1 -type f -name 'id_*' ! -name '*.pub' -printf '%m %p\n' | awk '$1 != "600" && $1 != "400" {print "FINDING: private key " $2 " is mode " $1}'

The controller key is used only by the controller. If the same private key is also on a laptop or in a CI secret store, "the controller did it" is no longer a statement anybody can make about an audit log.
Runs are attributable. Either each operator uses their own account and key, or every production run goes through a pipeline that records who triggered it. A shared `ansible` account with a shared key makes every incident review end at "somebody ran it".
```
awk -F: '$3>=1000 && $3<65534 && $7 !~ /nologin|\/false/ {print $1}' /etc/passwd; last -n 20
```
The controller is not a host in its own production inventory. A play that restarts sshd or rewrites a firewall across `all` will otherwise take out the machine running it, halfway through the run.
```
ansible -i inventories/production all --list-hosts | tr -d ' ' \
  | grep -Fx "$(hostname -f)" && echo "FINDING: the controller is a target of its own production inventory"
```
Out-of-band access to the managed nodes exists and has been used at least once. The failure this covers is an automation change that breaks SSH on the fleet, after which the controller can no longer reach anything to fix it.
The controller has been rebuilt from the repository, the pins and the escrowed vault key within the last twelve months, and the rebuild was timed. Until that has been done once, the controller is a pet and the estate has a single point of failure.

Warning6 items

Every setting that differs from the ansible-core default is deliberate and recorded in the repository, with the file it came from. Review the output; an entry nobody can explain is a finding.
```
ansible-config dump --only-changed
```

A log_path is configured, so every run leaves a durable record independent of whoever was watching the terminal. Without it the only evidence a run happened is the operator scrollback.

p=$(ansible-config dump | sed -n 's/^DEFAULT_LOG_PATH([^)]*) = //p')
{ [ -n "$p" ] && [ "$p" != "None" ] && [ -w "$p" ]; } \
  || echo "FINDING: log_path is '$p' - not set, or not writable by this account"

The Ansible log is rotated and the filesystem holding it has headroom. A fleet run against a few hundred hosts at -vv writes hundreds of megabytes, and a controller that fills its own disk mid-run leaves a partial change and no log of it.

p=$(ansible-config dump | sed -n 's/^DEFAULT_LOG_PATH([^)]*) = //p')
grep -rl "$p" /etc/logrotate.d/ 2>/dev/null || echo "FINDING: no logrotate rule references $p"
df --output=pcent "$(dirname "$p")" | tail -1 | tr -dc '0-9' | awk '$1+0 > 80 {print "FINDING: log filesystem is " $1 "% full"}'

The fork count is a deliberate number, sized against controller CPU and memory rather than left at the default of 5. Prints the configured value next to the CPU count so the two can be compared.
```
ansible-config dump | grep '^DEFAULT_FORKS'; echo "controller CPUs: $(nproc)"
```
The controller clock is synchronised. Timestamps in the Ansible log are the primary evidence in a post-incident timeline, and a controller drifting from the hosts it changed makes that timeline unusable.
```
timedatectl show -p NTPSynchronized --value | grep -qx yes || echo "FINDING: controller clock is not NTP-synchronised"
```
The controller Python is a version ansible-core supports, and it is the interpreter the virtualenv actually uses - not the system Python that happens to be first on PATH.
```
ansible --version | grep -E 'python version|executable location'
```

Info1 item

The commissioning baseline is captured and attached to the ticket: exact core version, interpreter path, installed collections and every non-default configuration setting. This is the artefact the disaster-recovery checklist rebuilds against, so it is worth having before it is needed.
```
ansible --version; ansible-galaxy collection list; ansible-config dump --only-changed; pip freeze
```