Where the controller lives · Ansible

At small scale the controller is wherever you happen to be. At fleet scale it is a named piece of infrastructure with a capacity, a failure domain, a network position, and a role in a pipeline that starts in a Git repository and ends on several thousand machines.

Getting its placement wrong does not produce an error message. It produces runs that take four hours instead of forty minutes, a change window that closes before the change finishes, and — in the worst case — a controller that has just reconfigured the network it reaches the fleet through.

The shape of fleet automation

The controller is one box in a longer chain. Drawing the whole chain first is worth the space, because most placement arguments are really arguments about which box owns which decision.

flowchart TB
  DEV["Engineer<br/>writes a change"] --> GIT["Git repository<br/>roles, playbooks, inventory data"]
  GIT --> CI["CI validation<br/>lint, syntax, check-mode gate, molecule"]
  CI -->|"merge to main"| CTL["Automation controller<br/>schedules and executes runs"]
  CMDB["Inventory source / CMDB<br/>which hosts exist, and what they are"] --> CTL
  VAULT["Secret store<br/>credentials for the run"] --> CTL
  CTL --> W0["Wave 0<br/>tooling"]
  W0 -->|"gate"| W1["Wave 1<br/>canary"]
  W1 -->|"gate"| W2["Wave 2<br/>internal prod"]
  W2 -->|"gate"| W3["Wave 3..n<br/>customer prod"]
  CTL --> LOG["Run artefacts<br/>logs, diffs, recap, commit SHA"]
  LOG --> DEV

Two things fall out of that picture immediately.

The controller is not the source of truth for anything. Code comes from Git, host membership comes from the inventory source, credentials come from the secret store. If the controller is the only place some piece of configuration exists, it has become a snowflake, and rebuilding it becomes an archaeology exercise.

The controller is the only box that touches everything. It can read the secret store and it can reach every managed host. That combination is what makes its placement a security question and not just a performance one.

What latency costs you

A task is not one packet. Even with connection reuse, running a module on a host involves transferring the module payload, executing it, reading the JSON result and cleaning up — several round trips over the SSH channel, per task, per host.

Take a plausible set of numbers: 4 round trips per task, 200 tasks in the play, and a controller 150 ms away from the target region.

Quantity	Value
Latency per task per host	`4 × 150 ms = 0.6 s`
Latency per host for the play	`200 × 0.6 s = 120 s`
Hosts	2,000
Effective concurrency (`forks`)	50
Sequential host slots	`2000 / 50 = 40`
Wall-clock from latency alone	`40 × 120 s ≈ 80 minutes`

Eighty minutes in which nothing computed anything. Move the controller into the region and take the round trip to 2 ms, and the same arithmetic gives roughly one minute.

Read-only / Safemeasure the round trip you are actually paying— A single module execution over a real connection, timed. Compare a local target with a remote one to see the latency component directly.

# one module round trip against a nearby host
time ansible -i inventory/ web014.example.com -m ansible.builtin.ping

# and against a host across the link
time ansible -i inventory/ web811.example.com -m ansible.builtin.ping

Central or regional

Neither answer is universally right, and the trade is not primarily about speed.

A single central controller gives you one place where runs happen, one audit trail, one version of Ansible, one set of collections, and one credential store integration. It also gives you one failure domain: when it is down, nothing is automated anywhere, and when the link to a region degrades, every run into that region degrades with it.

Regional controllers give you short links, a firewall scope per region rather than one flat path to everything, and independent failure domains. They cost you consistency. Three controllers are three chances for “it works on mine” — a different ansible-core minor version, a collection installed by hand eight months ago, an ansible.cfg with a setting nobody remembers adding.

Read-only / Safecontroller identity, for the run record— Everything a future investigation needs to know about what executed. Read-only; none of these connect to a managed host.

ansible --version
ansible-config dump --only-changed
ansible-galaxy collection list
python3 --version

ansible-config dump --only-changed is the important one: it prints only settings that differ from the defaults, which is precisely the list of ways this controller is not a stock controller.

The controller is inside its own blast radius

Here is the part that gets skipped, and it is the one that turns a bad afternoon into a multi-day recovery.

The controller reaches the fleet over the network, through DNS, using SSH, authenticating with keys, trusting a CA bundle. Every one of those is also a thing the fleet’s automation manages.

Which means a run can sever the controller from its fleet:

A play that tightens sshd_config and gets AllowUsers wrong locks the automation account out of every host it succeeded on.
A firewall role that stops permitting the controller’s source range removes the path it needed to fix the firewall role.
A change to the internal resolver, the internal CA, or the NTP source can break authentication or name resolution fleet-wide.
Renewing the automation account’s key without distributing the new public key first breaks the next run and every run after it.

The failure has a distinctive signature: unreachable counts climbing in step with the batches that succeeded. The hosts the change did not reach yet are still fine. That inversion — success causing unreachability — is the tell, and it is why the obvious diagnosis (“network problem”) wastes the first thirty minutes.

Never let the controller be a target of the change that could cut it off

Concrete controls, in order of how much they buy you:

Exclude the controller from fleet-wide plays that touch SSH, firewall, DNS, PKI or the automation account. Manage the controller with a separate, deliberately-run playbook.
Land those changes in waves with a reachability gate. A task at the start of each wave that fails if the previous wave’s hosts have stopped answering turns a fleet-wide lockout into a one-wave lockout.
Have an out-of-band path — console, IPMI, serial, a bastion on a separate credential — and confirm it works before the change window, not during it. An untested out-of-band path is a plan, not a control.
Keep a second, independent credential for the automation account that the run being executed does not rotate.

Under the hood

Nothing in Ansible knows that the controller is special. It is an SSH client with an inventory; the machine it happens to run on has no protected status, and ansible.builtin.setup gathering facts about a host says nothing about whether that host is the one running the play.

That is why the protection has to be structural rather than clever. The inventory_hostname != 'controller01' conditional people reach for first is worse than useless: it depends on the controller appearing in inventory under a name someone remembered to match, it silently does nothing if the name changes, and it puts host-specific logic inside a role — the exact anti-pattern the drift part argues against.

The structural version is that the controller is not in the target group. Blast radius is decided by the inventory and the pattern, and “which hosts can this run reach” is a question you can answer with --list-hosts before you run it. A conditional inside a role is a question you can only answer by reading every role.

Sizing, briefly

Each fork is a controller-side process holding an SSH connection, a Python interpreter’s worth of memory, and file descriptors. The default forks is 5, which is wrong for a fleet in the obvious direction — and raising it consumes controller CPU, RAM and descriptors in a way that starts presenting as target-side failures when you run out.

The controller readiness questions before a fleet run:

CPU and RAM headroom at the fork count you intend to use.
File descriptor limits for the account running the play.
Disk for logs and artefacts. A verbose run over 2,000 hosts produces a log measured in gigabytes, and a controller whose disk fills mid-run fails in confusing ways.
Inventory freshness, and credentials that will not expire mid-window.

Measuring and tuning these properly — finding the fork ceiling empirically rather than guessing — is Part XXXIV’s subject. What belongs here is that they are placement inputs: a controller sized for 5 forks in a region 150 ms away is a different machine from one sized for 100 forks next door.

What happens if the controller dies mid-run?

Every task in flight is abandoned. The SSH connections drop, the module processes on the managed nodes may or may not complete, and there is no resumption record anywhere — the recap that would have told you where it got to is assembled in memory and lost with the process.

The estate is now in a state where some hosts have the change, some do not, and some are mid-change. Nothing in Ansible will tell you which is which.

This is the argument for the partitioning discipline in the next lesson: runs that write down where they got to, and a re-entry that is safe because the plays are idempotent rather than because someone kept notes. It is also an argument for streaming run output to somewhere that is not the controller’s local disk, so the record survives the machine.

The shape of fleet automation

What latency costs you

Central or regional

The controller is inside its own blast radius

Sizing, briefly

Knowledge check