Whose machine runs production change? · Ansible

Nobody decides where the controller lives. It accumulates.

Somebody installs Ansible on their laptop to try something. It works, so they use it for the next thing. Six months later the laptop is the only machine that can deploy the application, the SSH key that reaches every production server is in that person’s home directory, and the estate has made a governance decision that nobody made.

This lesson is about making it on purpose.

Four options

	Laptop	Shared jump host	CI runner	Automation platform
Who ran it	The owner, unattributed	A service account — attribution needs care	The pipeline, attributed to a commit	The platform, attributed to a job and a user
Reproducible	Only for that person	Yes, if built from the repository	Yes, by construction	Yes
Reviewed before running	Only by convention	Only by convention	Enforced by the merge	Enforced by the job definition
Where the fleet key lives	Next to a browser	On a hardened host	In the CI secret store	In the platform’s credential store
Works during a network incident	Sometimes	If you can reach it	Rarely	Rarely
Setup cost	None	Low	Medium	High

None of these rows is decisive on its own. Read them together and a pattern appears: the options get stronger on governance and weaker on availability as you move right, and the cost rises the whole way.

The laptop

It is the right controller for exactly one thing: exploration. Writing a new role, running --check against staging, debugging an inventory, learning. The feedback loop is short and nothing else can match it.

It is a poor controller for production change, for four reasons that have nothing to do with the engineer’s competence.

Attribution disappears. A run from a laptop leaves a record on the managed hosts — sudo entries, changed files, a service restart — and nothing that connects the change to a reviewed intention. Six weeks later the question is “why is this host different”, and the answer is in somebody’s shell history, if it is anywhere.

Reproducibility is personal. The laptop’s controller environment is the one they built. Their collection versions, their ~/.ansible.cfg, their SSH config with a ProxyJump nobody else has. It works, and it works only there.

Availability is a property of a person’s day. The deployment happens when the laptop is open, awake and on the VPN. This is fine until it is 2 a.m. or the person is on a plane.

The exposure is the point. A laptop runs a browser, a mail client, a chat application and whatever a developer needed last Tuesday. Every one of those is an execution path for content the owner did not write. On that same machine sits a passphraseless key with root on every production host.

The shared jump host

A dedicated machine, a service account, the repository deployed from Git, the venv built from requirements.txt. Everyone runs the same Ansible because there is only one.

This solves reproducibility and most of the exposure problem. It leaves two things open.

Attribution needs deliberate work. If four engineers sudo -u svc-ansible and run playbooks, the managed hosts see one identity. Shell audit on the controller, log_path in ansible.cfg, and a convention of running through a wrapper that records who invoked it are the mitigations — and all three are things you have to add, not things you get.

Review is still a convention. Nothing stops an engineer editing the checkout in place and running it. The filesystem layout from the previous lesson is what makes that hard: the checkout is owned by root and deployed, not writable by the people who use it.

Read-only / Safewhat a shared controller should be able to tell you— Three facts that together identify which controller environment a run used. Capture them at the start of every scheduled run so the log records what was in effect, not what you assume was in effect.

ansible --version | head -2
ansible-galaxy collection list
ansible-config dump --only-changed

Read-only / Safethe third one, on a correctly configured controller— Every line here is a deviation from a default. On a shared controller this output is the difference between your environment and a stock one, which makes it the first thing to compare when two controllers disagree.

$ ansible-config dump --only-changed

CONFIG_FILE() = /opt/estate/ansible.cfg
DEFAULT_HOST_LIST(/opt/estate/ansible.cfg) = ['/opt/estate/inventory/production']
DEFAULT_LOG_PATH(/opt/estate/ansible.cfg) = /var/log/ansible/ansible.log
DEFAULT_VAULT_PASSWORD_FILE(/opt/estate/ansible.cfg) = /etc/ansible/vault-pass
HOST_KEY_CHECKING(/opt/estate/ansible.cfg) = True

GALAXY_SERVERS:

Illustrative output

Note what that output gives you for free: each setting is annotated with where it came from. A value attributed to a file you did not expect is the fastest possible diagnosis of a configuration surprise.

The CI runner

The strongest governance position, because review stops being a convention and becomes a mechanism. Change reaches production by being merged, and the runner has credentials that the humans do not.

What it buys:

Every change is attributable to a commit, an author and an approval.
The environment is built from the repository on every run, so the two-machine test is executed continuously rather than annually.
Credentials live in the CI secret store rather than in a home directory, and can be scoped per pipeline.

What it costs, and these are real:

Feedback is slow. A five-second change becomes a pipeline.
Debugging is awkward. You cannot easily attach to a failing run, and reproducing it locally means reproducing the runner.
It is a dependency during an incident. If the outage involves the network, the CI system, or the identity provider the CI system authenticates against, your remediation tool is behind the failure.

The automation platform

AWX, Ansible Automation Platform, or an equivalent. It adds job templates, a credential store, RBAC, scheduling, a surveyed interface for people who should not have a shell, and a job history that is a real audit trail.

It is the right answer for an organisation where many people need to run a limited set of automations, and where “who may run what against which inventory” is a question with a formal answer. It is a heavyweight answer for a team of four, where the platform becomes another system to patch, back up and recover.

The course does not treat the platform as the destination that everyone should reach. It treats it as one point on the same axis: further right, stronger governance, higher cost.

What most estates should actually do

Two controllers, and the split is by purpose rather than by person.

A personal controller — laptop or a small VM — with the repository checked out, credentials that reach staging only, and no production inventory present. Explore here. Run --check here. Break things here. The blast radius is bounded because production is not in the file.

A production controller — a jump host, a CI runner, or both — with the production inventory, the production credentials, and a checkout nobody edits in place.

The important detail is that the boundary is enforced by which inventory exists on which machine, not by discipline. An engineer on the personal controller cannot accidentally target production, because production is not there to be targeted. This is the same argument the inventory design part makes at fleet scale, arriving early because it is also the answer to where the controller lives.

Under the hood

“It works on my controller” resolves to one of four differences, and all four are printable. It is worth knowing them as a set, because the order you check them in is the difference between a five-minute diagnosis and an afternoon.

The runtime. ansible --version — different ansible-core, or the same core on a different Python.
The collections. ansible-galaxy collection list — a module behaving differently because the collection moved.
The configuration. ansible-config dump --only-changed — the same code obeying different settings, including which inventory is the default and whether host key checking is on.
The environment. SSH config, agent contents, ~/.ansible.cfg, and environment variables. ANSIBLE_* variables beat the config file, so a stray export in a shell profile silently overrides the repository for that one person.

Three of the four are properties of a machine. One is a property of a login session. That last one is why a shared controller with several interactive users is still capable of behaving differently for different people, and why scheduled runs should start from a known environment rather than inheriting a shell.

What happens if the only controller is lost?

The disk fails, the laptop is stolen, the VM is deleted by a cleanup script that did not know what it was.

What you have lost depends entirely on what was only there. If the repository is in Git and the environment builds from requirements.txt and requirements.yml, you have lost a machine and an afternoon. If the inventory was edited in place, if collections were installed by hand, if the vault password existed in exactly one file — you have lost the ability to change your fleet, at the moment you most want it.

The recovery question deserves rehearsing rather than reasoning about, and the rehearsal is the two-machine test from the previous lesson. A team that builds a second controller from the repository once a quarter knows how long recovery takes and what is missing. A team that has never tried is guessing. Rebuilding a lost controller under real conditions — including recovering the credentials, which is the hard part — is the subject of the automation disaster-recovery part later in the course.