Building a controller worth trusting

The previous lesson established that losing the controller means losing the fleet. The obvious response is to make the controller hard to lose, and the obvious response is mostly right — with the caveat that “hardening” here means something narrower and more achievable than the word usually implies.

You are not trying to build an unbreakable host. You are trying to build one with a small number of ways in, all of which you know about.

The controller’s problem is not that it is weak. It is that it is usually general-purpose, and a general-purpose host has an unbounded number of ways in. Every one of them is fleet-wide.

The properties that matter

Dedicated. The host runs the automation and nothing else. No web server, no monitoring agent that ships arbitrary plugins, no shared service somebody else administers. Every additional service is another component whose compromise is a fleet compromise.

Minimal package set. Fewer packages is fewer CVEs to track and less material for an attacker to use once inside. This is not about disk space; it is about the number of things you must patch on the schedule of your most important host.

No browsing and no development work. A browser is the single largest attack surface any host can carry, and a development toolchain pulls dependencies from public registries on somebody’s whim. Neither belongs on a machine that holds the fleet’s keys.

Encrypted at rest. Full-disk encryption, with an honest understanding of what it buys — see below.

Restricted egress. The controller needs to reach managed nodes, a Git remote and a package or Galaxy source. It does not need the rest of the internet.

Patched on its own schedule. Not with the fleet, and not later than the fleet.

Rebuildable. Everything about it is in configuration management, including its own build. Part L covers rebuilding a lost controller; the same artefact is what lets you rebuild a suspect one, which is a much more common need.

What full-disk encryption actually buys

Be precise here, because the honest answer is narrower than the usual claim and the narrow version is still worth having.

Full-disk encryption protects the controller when it is off. A stolen laptop, a decommissioned disk, a VM image copied out of a hypervisor’s storage, a backup that leaves the building.

It protects nothing while the machine is running and the volume is unlocked, which for a controller is essentially all the time. An attacker with a shell on a running controller reads the vault password file and the SSH key through the same filesystem you do.

So: encrypt it, because the offline cases are real and encryption is cheap. Do not count it as a control against the compromise this part is actually about.

Egress that does not break the tool

The failure mode of a restrictive egress policy is a controller that cannot install a collection at the worst possible moment. Work out the list deliberately.

The controller genuinely needs:

• The managed nodes, on the SSH port, for the environments this controller is responsible for. Not every environment — see lesson 3.
• The Git remote, for the repository.
• A content source: Galaxy, a private Galaxy or Automation Hub, or an internal mirror. An internal mirror is the better answer and Part XXVII covers --offline installs against one.
• A package source for its own operating system updates.
• Log and monitoring destinations, if it ships logs.

It does not need general web access, and denying that is most of the benefit.

Patching, on its own schedule

The controller is patched before the fleet, not with it, and for a reason that is easy to state.

If a vulnerability in a shared component affects both the controller and the managed nodes, the controller is the host where exploitation is worth the most. Patching it last means the highest-value target spends the window exposed. Patching it in the same wave as five hundred servers means it is patched whenever the wave reaches it.

There is a second reason specific to this host: the controller is the machine that patches everything else. A controller compromised through an unpatched component is a controller that can deliver a package to every host in the estate, which converts a single-host vulnerability into a supply-chain event.

Give it its own window, its own change record, and its own verification that the run afterwards still works — because a controller patch that breaks ansible-core or a Python dependency is an outage of your ability to respond to anything else.

The workstation controller

This is the common case and it deserves to be argued rather than dismissed, because the people running it are not being careless — they are being practical, and the practical objections are real.

The argument for it: the controller is where the engineer is. Their editor, their Git credentials, their SSH agent, their notes. Moving automation to a separate host adds friction to every task, and friction is why people stop using the automation and start using SSH directly, which is worse.

The argument against it is one sentence: a workstation is a host that executes untrusted content by design. Web pages, email attachments, dependencies from public registries, a language server pulled in by an editor plugin. Each of those is a route to the automation key and the vault password that no managed node can see.

The resolution that works in practice is not “use a bastion controller and suffer” but to split by consequence:

• Read-only and check-mode work from the workstation, with a key that can only do that. The command= restriction in lesson 4 makes that enforceable at the far end.
• Production changes from a dedicated controller, with credentials that exist nowhere else.

That keeps the daily loop fast, keeps the credentials that matter off the machine that browses the web, and makes the boundary an explicit decision rather than a habit.

Knowledge check