high riskservice affecting~120 min

Runbook: Recover the controller

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The Git remote is reachable and contains the automation repository at the production branch
· The requirements files - Python and Galaxy - are in the repository and pin exact versions
· The inventory source is reachable: a static inventory in Git, or the credentials for the dynamic source
· The vault password is retrievable from the password manager, not from the lost machine
· The automation SSH key is retrievable from the password manager or secret store, OR a decision has been made to issue a new one
· It is established whether the lost controller held anything that existed ONLY there - and that list is written down before rebuilding
· Whether the loss was a failure or a compromise is established, because a compromised controller changes every credential decision

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Declare the controller lost and confirm nothing is still running on it or attempting to
2Enumerate what existed only on that machine, before rebuilding, while people still remember
3Decide the credential position: reuse the existing automation key, or issue a new one and rotate
4Build a replacement controller from the build-controller runbook: pinned interpreter, pinned core, pinned collections
5Restore the repository from Git at the last known-good commit, not at the branch head
6Restore the inventory: from Git if static, or by reconfiguring the dynamic source with a fresh read-only credential
7Restore credentials: vault password from the password manager, SSH key from the secret store or newly issued
8Run the read-only proof sequence and compare its output against the lost controller records
9Run one check-mode playbook against a single non-production host and read the diff for divergence
10Reconcile: any drift accumulated while there was no controller is now visible and must be triaged, not blindly converged
11Record the gap: how long the estate had no controller and what ran or failed to run in that window

4 · Verification

Confirm the procedure actually fixed the problem.

✓ansible --version reports the pinned core version and names the repository ansible.cfg
✓ansible-galaxy collection list matches requirements.yml exactly, with no extra collections
✓The repository is at the recorded commit hash, verified by git log rather than assumed from the branch
✓ansible-inventory --graph produces the same group structure and host count as the last recorded run
✓ansible-vault view succeeds on a known encrypted file
✓ansible all -m ping succeeds fleet-wide, or every exception is recorded with a reason
✓A check-mode run against one non-production host shows only changes that are explained by drift, not by controller divergence

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶There is nothing to roll back - the controller is already gone and this procedure builds a replacement
↶If the rebuild is wrong, delete it and rebuild; nothing on the managed fleet has been touched until the first real run
↶Do NOT run a fleet-wide converge to "get back to normal" - the first run from a rebuilt controller is the highest-risk run the estate will see, because nothing has verified that this controller matches the one that made the current state
↶If a new SSH key was issued, the old key is still trusted fleet-wide until it is removed - that is the SSH rotation runbook, not this one
↶If the loss was a compromise, treat every credential the controller held as exposed; rebuilding does not un-expose them

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the security owner immediately if the controller was compromised rather than merely lost - the vault password, the SSH key and every secret it decrypted are all exposed
· Escalate if the vault password cannot be retrieved from anywhere except the lost machine; enumerate what is now permanently undecryptable before doing anything else
· Escalate to the platform owner before the first non-check run from the rebuilt controller
· Escalate if the repository has no recorded commit hash for what the lost controller was running - the rebuild is then a best guess and must be stated as one
· Escalate if drift accumulated during the outage window includes anything a converge would revert that should not be reverted

The controller is the least backed-up machine in most estates, on the reasoning that everything on it comes from Git. That reasoning is correct about the automation and wrong about everything else - and the gap between the two is what this runbook is for.

A controller is recoverable from four things a team should hold offsite: Git, pinned requirements, the inventory source, and the vault password. If all four exist, the rebuild is mechanical. If the fourth does not, part of the estate’s encrypted material is gone permanently, and the honest first step is establishing which.

When to use this runbook

The controller host has failed and cannot be repaired.
The controller was destroyed, deleted or reclaimed.
The controller was compromised and must be replaced rather than cleaned.
A disaster recovery exercise that tests exactly this.

Blast radius

Rebuilding changes nothing on the managed fleet. The blast radius is the first run from the rebuilt controller, and that run is the highest-risk one the estate will see - because nothing has yet verified that this controller matches the one that produced the fleet’s current state.

While there is no controller: nothing converges, nothing patches, nothing remediates drift. The estate is stable and unmanaged. That is usually survivable for hours and it has a cost that grows.

Step 1: Declare it lost, and stop the ambiguity

Read-only / Safeconfirm it is gone and nothing is retrying

# Is anything still reaching it?
getent hosts controller01.example.com
timeout 5 bash -c 'cat < /dev/null > /dev/tcp/192.0.2.50/22' \
&& echo 'still up' || echo 'unreachable'

# Anything scheduled that expects it
echo 'Pause the CI pipelines and any scheduler entries that target it'

A half-alive controller is worse than a dead one: it may still hold a lease, still answer a health check, still run a timer. Declare it, in writing, and pause everything that expects it.

Step 2: Enumerate what existed only there

Do this before rebuilding. It is the step that gets skipped because rebuilding feels productive, and it is the step that has a deadline - people forget, and the machine is not coming back to be inspected.

Ask, and write down the answers:

Thing	Was it only on the controller?
The vault password	If yes, see the callout below. This is the big one.
The automation SSH private key	If yes, the fleet trusts a key nobody holds
Uncommitted work in the repository	If yes, it is gone; find out what it was
The `ansible.cfg` in use	Only a problem if it was not in the repository, which it should have been
Fact cache	Recoverable by re-gathering; no action needed
Run logs and change history	If they were local only, the audit trail for recent runs is gone
A dynamic inventory credential	Reissue it; note the old one is unaccounted for
Retry files, ad-hoc scripts, one-off inventories	The undocumented estate; enumerate honestly
Locally installed collections not in requirements.yml	The rebuild will not have them - which is a finding about the old controller

If the vault password existed only on the lost controller, some material is unrecoverable

Ansible Vault is symmetric encryption with no key escrow, no recovery key and no backdoor. If the password is not in the password manager, not in another controller’s file, and not in anyone’s memory, then every file encrypted under it is permanently undecryptable.

That is not a recovery problem to work around. It is a loss to enumerate and act on:

Read-only / Safewhat is now unreadable

git clone https://git.example.com/infra/automation.git /srv/automation/repo
cd /srv/automation/repo
grep -rl --binary-files=without-match '^\$ANSIBLE_VAULT' . --exclude-dir=.git \
| tee unrecoverable-files.txt
grep -rn '!vault' --include='*.yml' . --exclude-dir=.git \
| tee -a unrecoverable-inline.txt

For each file, the content is gone but the secrets themselves are not: the database password still exists in the database, the API token still exists in the provider, the certificate still exists on the hosts. What is lost is your copy of them.

So the recovery is rotation, not decryption. Each secret is reissued through its own owner’s process, re-encrypted under a new vault password, and committed. That is a substantial workstream and it should be started immediately, in parallel with the rebuild, not after it.

The other consequence: any playbook that consumes those variables cannot run until they are replaced. Establish which playbooks those are, because they define what the estate cannot do in the meantime.

Step 3: Decide the credential position

Two questions, and the second depends on how the controller was lost.

Can the automation SSH private key be restored? If it lives in the secret store, restore it and continue. If it only lived on the lost machine, the fleet now trusts a key nobody holds - which is both an access problem and a security problem. Issue a new key, get it onto the fleet, and remove the old one via the SSH credential rotation runbook.

Was this a failure or a compromise?

Step 4: Build the replacement

Follow the build-controller runbook. The essentials, in the order that matters:

Configuration changepinned environment

sudo useradd --create-home --shell /bin/bash ansible
sudo -iu ansible

python3 -m venv /home/ansible/.venvs/ansible
/home/ansible/.venvs/ansible/bin/pip install 'ansible-core==2.21.3'
export PATH="$HOME/.venvs/ansible/bin:$PATH"
ansible --version

The version must be the one the lost controller ran. If that is not recorded anywhere, say so explicitly in the recovery notes - a rebuild on a different ansible-core is a rebuild that may behave differently, and “we think it was 2.21” is a material uncertainty, not a detail.

Step 5: Restore the repository at the right commit

Configuration changecheckout the recorded commit

git clone https://git.example.com/infra/automation.git /srv/automation/repo
cd /srv/automation/repo

# The commit the lost controller was running, from the change record
git checkout 4f2a9c1
git log -1 --format='%H %ci %s'

Step 6: Restore collections from requirements

Configuration changepinned collections

cat requirements.yml
ansible-galaxy collection install -r requirements.yml -p ./collections
ansible-galaxy collection list

Compare that list against whatever record exists of the lost controller’s collections. A collection the old controller had that is not in requirements.yml is a divergence you have just removed - which is correct, and which may change behaviour. Note it.

Step 7: Restore the inventory

Static inventory comes from Git with the repository, and the verification is the host count.

Dynamic inventory needs its credential reissued:

Configuration changedynamic inventory

# Issue a NEW read-only credential; do not attempt to recover the old one
export AWS_PROFILE=ansible-inventory-readonly
ansible-inventory -i inventories/production/aws_ec2.yml --graph

The credential should be read-only. An inventory plugin needs to list hosts, not to modify anything, and a recovery is a good moment to notice if the old one was broader than that.

Read-only / Safeverify against the last known good

ansible-inventory -i inventories/production --graph | tee inventory-after-recovery.txt
ansible-inventory -i inventories/production --list \
| python3 -c 'import json,sys; print(len(json.load(sys.stdin)["_meta"]["hostvars"]))'

Compare the host count against the last recorded run. A difference is either real estate change during the outage, or an inventory that has not fully restored. Resolve which before proceeding.

Step 8: Restore credentials

Configuration changecredentials, from the password manager

install -m 0600 -o ansible -g ansible /dev/null /home/ansible/.vault-pass
# write the vault password into it from the password manager

install -m 0600 -o ansible -g ansible /dev/null /home/ansible/.ssh/id_ed25519
# write the key material, then confirm which key you installed
ssh-keygen -lf /home/ansible/.ssh/id_ed25519

stat -c '%a %U:%G %n' /home/ansible/.vault-pass /home/ansible/.ssh/id_ed25519

Read-only / Safeprove decryption works

ansible-vault view --vault-password-file /home/ansible/.vault-pass \
inventories/production/group_vars/all/vault.yml | head -3

If that fails, stop and return to Step 2. A controller that cannot decrypt is not recovered, and discovering it during the first real run is discovering it in the worst place.

Step 9: The read-only proof sequence

Read-only / Safeprove before you change

ansible --version
ansible-config dump --only-changed
ansible-inventory --graph
ansible-playbook site.yml --list-hosts
ansible all -m ping -o | tee recovery-ping.txt
grep -c SUCCESS recovery-ping.txt

Compare each output against the records from the lost controller. Differences are the point of the exercise: each one is either estate change during the outage, or a divergence between the rebuild and the original. Both need naming; only the first is acceptable.

Step 10: One check-mode run, read for divergence

Read-only / Safecheck one non-production host

ansible-playbook site.yml --limit staging-web01.example.com --check --diff \
| tee recovery-check.txt

Read the diff with a specific question in mind: is this change caused by drift on the host, or by a difference between this controller and the last one?

A package version the role would install and the host does not have - drift, probably fine to converge.
A configuration file the role would rewrite differently than the host currently has - suspicious. That file was written by the old controller. If this controller would write it differently, this controller is not the same controller.

The second case means the rebuild diverged: a different collection version, a different commit, a different variable resolution. Find it before running anything for real.

Step 11: Do not converge the fleet

Triage the drift instead:

Read-only / Safewhat drifted while there was no controller

ansible-playbook site.yml --check --diff | tee drift-report.txt
grep -cE '^changed:' drift-report.txt

# Which hosts, and what would change on each
grep -B2 -A10 '^changed:' drift-report.txt | head -60

For each drift item: is it something to revert, or something that was done deliberately during the outage and should now be declared? A converge answers that question with “revert everything”, which is the wrong answer for any emergency fix that is holding a service up.

Step 12: Record the gap

When the controller was lost, and when the replacement passed verification.
What ran, or failed to run, during the window - patching, drift remediation, scheduled converges.
What existed only on the lost machine, and its disposition.
Whether the rebuild used the recorded commit, or a best guess.
The drift triage outcome.

The most valuable entry is the “existed only there” list, because it is the input to making the next loss cheaper.

Common patterns

Symptom	Likely cause	Resolution
Vault files cannot be decrypted	The password existed only on the lost host	Not recoverable. Rotate every secret through its owner
Rebuilt controller proposes changes on a converged host	Different collection version, or the branch head instead of the recorded commit	Compare `collection list` and `git log`; check out the recorded commit
Fleet unreachable from the new controller	The SSH key was not restored, or a new one was issued and not distributed	SSH rotation runbook
Inventory host count differs	Real estate change, or a dynamic source not fully configured	Reconcile against the source of truth before proceeding
A playbook fails that used to work	A collection the old controller had locally, absent from `requirements.yml`	Add it to requirements, pinned; note the old controller was undocumented
Emergency hand fixes reverted after recovery	A fleet-wide converge as the first action	Triage drift before converging; declare deliberate changes
Nobody knows which commit was running	It was never recorded	Rebuild is a best guess - state it, and record the hash from now on

Escalation

Escalate when:

The controller was compromised. Every credential is exposed and that is a security workstream, not a rebuild.
The vault password cannot be retrieved. Enumerate the unrecoverable material first.
Before the first non-check run from the rebuilt controller.
No commit hash was recorded for what the lost controller ran.
Drift during the outage includes something a converge would revert that should not be.

Why this matters

Controllers are treated as disposable because the automation lives in Git, and that is right up to the point where it is catastrophically wrong. Git holds the playbooks. It does not hold the vault password, and without that the encrypted half of your automation is not recoverable by any means - there is no escrow, no recovery key, and no vendor to call.

That single fact should determine where the vault password lives, and the test is simple: if the controller vanished right now, could you decrypt? If the honest answer involves the machine that vanished, fix that today rather than during the recovery.

The second thing worth carrying: the dangerous moment in this procedure is not the rebuild, it is the first run afterwards. A rebuilt controller is a new controller until proven otherwise, and the estate has spent the outage accumulating drift and emergency fixes. Converging everything to “get back to normal” reverts both at once, across the whole fleet, at the moment when everyone most wants to believe the incident is over.