The controller is permanently lost · Ansible

The controller is a single host. Everything this course has taught you to do — targeting, batching, check runs, vault, rolling deployments — runs from it, and in most organisations there is exactly one of it.

That makes it the least redundant machine in an estate whose entire purpose is to make other machines redundant. Nobody plans it that way. It happens because the controller starts as somebody’s shell account with a playbooks/ directory in it, and by the time it is managing four hundred servers it has never once been treated as production.

This part asks one question and spends six lessons answering it: the controller is permanently lost — can you rebuild it?

The scenario, stated precisely

Vagueness is what lets a recovery plan pass review without working. So the scenario is fixed, and every later lesson is tested against it:

The controller host is destroyed. Not powered off, not corrupted, not recoverable by a filesystem check. The disk is gone, the hypervisor is gone, or the region is gone.
There is no backup of the host itself. If there is one, that is a bonus, not the plan — a plan that depends on a backup you have not restored in eighteen months is a plan you have not tested.
The fleet is fine. Four hundred managed nodes are up and serving traffic. Nothing about them has changed.
Tomorrow morning there is work to do on that fleet: a security patch, a certificate rotation, or an incident. Something that needs automation, and needs it in hours rather than weeks.

Note what is deliberately not in the scenario. This is not a site-wide disaster. No managed node needs restoring. The question is narrow and uncomfortable: your ability to act on the estate is gone, while the estate itself is untouched.

The inventory of what was on that host

Before you can plan a rebuild, you have to enumerate what you are rebuilding. The list below is what a real controller carries. Read it against your own, and be honest about which column each item falls into.

What was on the controller	Where else does it exist?
Playbooks and roles	Git, if they were committed and pushed
Inventory files	Git, if static; the source system, if dynamic
Dynamic-inventory plugin config	Git — and the credentials it uses are not
`ansible.cfg`	Git, if it was the project one and not `~/.ansible.cfg`
`requirements.yml`	Git, if it exists at all
Installed collections and roles	Reinstallable from `requirements.yml`, if pinned
The `ansible-core` version	Reinstallable, if pinned somewhere
Vault-encrypted variable files	Git — safely, because they are encrypted
The vault password	Nowhere, unless you deliberately made it exist elsewhere
The automation SSH private key	Nowhere, unless escrowed
`~/.ssh/known_hosts`	Rebuildable, and see the warning below
Sudo/become passwords	A secret manager, if you used one
Cron or timer schedules for runs	Git, if managed; the crontab, if not
Fact cache	Regenerable; it is a cache
Run logs and artefacts	Wherever they were shipped, if they were shipped

Two rows are in bold because they are categorically different from the rest. Everything else on that list is either in Git or reconstructible from something in Git. Those two are secrets, and secrets are the one thing you are specifically not allowed to put in Git.

The vault password is the load-bearing dependency

If the vault password existed only on the destroyed controller, the repository you carefully committed is a pile of AES-encrypted noise.

Every group_vars/production/vault.yml, every !vault string, every database credential and API token you were praised for encrypting is now unreadable. Permanently. There is no recovery path, no support ticket, and no vendor to escalate to — that is what the encryption was for.

The rebuild does not fail slowly in this case. It fails at the first task that references an encrypted variable, and the fix is not technical: it is regenerating every secret in the estate by hand, on the hosts that still hold the old ones, without automation, because the automation needs the secrets you cannot read.

Lesson 4 of this part exists entirely because of this paragraph.

Two questions that separate a plan from a hope

Ask these of your own environment. They take about ten minutes and the answers are usually unwelcome.

1. What is on the controller that is not in the repository?

The honest way to find out is not to think about it — it is to look.

Read-only / Safewhat is untracked or unpushed on the controller— Read-only. Run this on the controller you would lose. Anything it prints is a candidate for the lost-forever column.

cd /srv/ansible

# Files git does not know about at all.
git status --short --untracked-files=all

# Commits that exist locally and nowhere else.
git log --branches --not --remotes --oneline

# Collections installed on this host, and where from.
ansible-galaxy collection list

# Which config file is actually in effect.
ansible --version | grep 'config file'

Each of those four commands answers a different failure. Untracked files are the hand-edited ansible.cfg and the one-off fix-it.yml somebody wrote during an incident. Unpushed commits are three weeks of work on a branch. The collection list is the collection somebody installed by hand in March that no requirements.yml records. And the config file line tells you whether the settings the fleet depends on live in the repository or in ~/.ansible.cfg, which nobody will think to recreate.

2. Which of your secrets exists in exactly one place?

The vault password: is there a copy outside the controller, and has anyone verified it decrypts the current repository this year?
The automation SSH private key: is there a copy, and would using it require a passphrase that also exists in one place?
The become/sudo password, if you use one: is it in a secret manager, or in somebody head?
Dynamic-inventory credentials: the API token the cloud inventory plugin uses is a secret too, and it is usually the one nobody counted.
CI credentials: if the pipeline can decrypt the vault, then the pipeline is also a copy - which is either your escrow or an unmanaged second copy, and you should know which.

The uncomfortable pattern is that item five is often the accidental answer to item one. Many teams discover during the drill that their only surviving copy of the vault password is a CI secret variable nobody documented as an escrow — which works, right up until the CI system is the thing being migrated.

Recovery time is a number, and you probably do not know it

The DR vocabulary from Proxmox Part XIV applies unchanged here. RTO is how long you can tolerate being unable to act on the fleet. RPO is how much automation work you can afford to lose.

For a controller, RPO is usually easy: it is the age of your last push, because Git is the state store. If people push daily, RPO is a day.

RTO is the number that surprises people. It is not “how long does pip install ansible-core take” — it is the sum of finding a host, getting network access to the fleet from it, restoring credentials, reinstalling a pinned toolchain, and getting a read-only run to succeed. Teams estimate two hours. Teams that have run the drill in lesson 6 usually report a day, and the difference is always the same three things: an undocumented manual step, a credential nobody could locate, and a collection installed by hand.

Under the hood

Why is the controller so easy to lose track of? Because Ansible’s own design encourages it, and the encouragement is mostly a virtue.

There is no daemon, no server component, no database and no registration step — that is the agentless model from Part II, and it is why you can start managing a fleet in ten minutes. The consequence is that the controller has no identity in the system it manages. A managed node does not know it is managed. Nothing anywhere records “this host is the controller for these four hundred servers”. You cannot query the estate to find out.

Compare a pull-based system such as Puppet, where every node is configured with a server address. Lose that server and the nodes tell you loudly and immediately, because they are trying to reach it every thirty minutes. Lose an Ansible controller and the fleet stays silent for exactly as long as nobody needs to change anything — which is how teams discover the loss during an incident rather than during a Tuesday.

The practical implication for DR: the controller has to be discovered from your side, not from the fleet’s. It belongs in the CMDB, it belongs in the asset register, and its hostname belongs in the runbook — because nothing about the running estate will remind you it existed.

The recoverable set, named

The rest of this part builds the answer, and it has four parts. Naming them now makes the following five lessons a checklist rather than a tour.

Layer	What it covers	Lesson
In Git	Playbooks, roles, inventory sources, `ansible.cfg`, `requirements.yml`, encrypted vars	2
Reproducible from pins	`ansible-core`, Python, collections, roles, Python dependencies	3
In escrow	Vault passwords, automation private keys, break-glass procedure	4
Containerised	The whole toolchain as an image, as one legitimate answer to layer two	5
Proved	A timed rebuild on a fresh host, with a read-only run at the end	6

A plan that covers the first three and has never executed the last one is the industry norm and is not sufficient. The drill is where the undocumented step is found, and there is always an undocumented step.

What happens if there are two controllers, so this cannot happen?

Then you have improved your odds and not removed the failure, and it is worth being precise about which.

What two controllers genuinely fix: the single-host outage. If one is destroyed, work continues from the other while a replacement is built. That is real and worth having.

What they do not fix: everything that was never captured. Two controllers that were both built by hand have two different sets of hand-installed collections, and neither is a specification for the other. When you build the third, you will be copying from a host rather than from a definition, which is exactly the failure Part III called snowflaking.

What they can make worse: drift between them. The controller that gets used is the one that works, so the other one quietly rots — its collections age, its SSH key never gets rotated, its ansible.cfg diverges. Six months later it is a fallback that fails on first use. Part XLVII’s shared-controller lesson covers the discipline that keeps two controllers actually equivalent.

What they do not touch at all: the vault password. If it exists on both controllers and nowhere else, then two controllers is two copies of a secret with the same correlated failure mode — the person who tidies away one VM often tidies away both, because they were provisioned together and look like a pair of forgotten dev boxes.

Two controllers is a good availability decision. It is not a DR plan, because a DR plan has to survive losing all of them.

Knowledge check

Why this matters

Every other part of this course has been about controlling what happens to the fleet. This one is about what happens to you.

The asymmetry is worth sitting with. You have spent forty-nine parts building the ability to change four hundred hosts deliberately, safely and reversibly — and all of that capability is resident on a machine that, in most organisations, has no monitoring, no backup, no owner in the CMDB and no label saying it matters. It is the most consequential unmanaged host in the estate.

The scenario in this lesson is not hypothetical and not dramatic. It is a hypervisor being decommissioned by somebody who checked whether anything was running on it and concluded, correctly, that nothing was — because a controller idles at zero load between runs.

The next five lessons turn the inventory in this lesson into a plan: what belongs in Git, what must be pinned, what must be escrowed, one containerised answer to the pinning problem, and the drill that proves the whole thing works. Read them as a single argument. The plan is only worth what the drill demonstrates.