Lab: Rebuild the controller from nothing, against the clock

Objective

By the end of this lab you will have built a working Ansible controller on a bare VM using only a git URL and a sealed envelope, run a read-only play against the fleet from it, and produced a timed log of every step where you had to reach for something that was not in the repository. That list is the deliverable — it is your controller’s real recovery-time driver.

Architecture

Two controllers and a fleet. The original is the one you are pretending to have lost; the new one is a bare VM.

  original controller  ──── git push ────▶  git remote
   (do not touch)                               │
                                                │ clone
  bare VM  ─────────────────────────────────────┘
     +  escrow envelope: vault password, SSH key, decisions record
     ▼
  read-only run against node1..node4

Requirements

• A bare VM with a supported OS, network access and nothing else installed. Not your workstation, and not a machine that already has Ansible on it — the presence of a working ansible binary defeats the whole exercise. B-nested only.
• A git remote the bare VM can reach, holding the automation repository.
• The fleet from earlier labs, or any four hosts you can reach read-only.
• A stopwatch. The number matters; a rebuild that “works eventually” is not a recovery procedure.
• No out-of-band access requirement: nothing here changes a managed node. The final run is deliberately read-only.

Scenario

The controller is gone. The VM was on a host that failed, there was no backup of the VM itself, and the person who built it left in March.

What you have: the git URL of the automation repository, and a sealed envelope from the safe. The envelope is supposed to contain everything the repository cannot.

You are going to find out whether it does.

Tasks

Task 1: Prepare the escrow envelope

Before the drill, assemble what a real escrow would hold. Do this on the original controller, and then do not look at it again until the drill needs it.

# On the ORIGINAL controller, before the drill
ESCROW="$HOME/escrow-$(date +%Y%m%d)"
install -d -m 0700 "$ESCROW"

# 1. The repository location and the branch that is production
cat > "$ESCROW/repository.txt" <<'EOF'
Automation repository:  git@git.example.com:platform/ansible.git
Production branch:      main
Read-only mirror:       https://git.example.com/platform/ansible.git
EOF

# 2. The vault password(s), by identity
install -m 0600 /dev/null "$ESCROW/vault-prod.txt"
cat > "$ESCROW/vault-prod.txt" <<'EOF'
REPLACE_ME_WITH_THE_REAL_PROD_VAULT_PASSWORD
EOF

# 3. The automation SSH private key
cp -a "$HOME/.ssh/ansible_automation" "$ESCROW/" 2>/dev/null || \
  echo 'no automation key found - this is itself a finding'

# 4. The decisions record: everything that is a choice, not a file
cat > "$ESCROW/decisions.md" <<'EOF'
# Controller decisions

ansible-core version:   (pinned in requirements.txt in the repo)
Python version:         3.12
Install method:         virtualenv at /opt/ansible, not the distro package
Collections:            pinned in collections/requirements.yml in the repo
Galaxy server:          internal mirror at https://galaxy.example.com/api/
Automation account:     'ansible' on every managed node, sudo NOPASSWD for
                        the commands in roles/base/files/sudoers.d/ansible
SSH known_hosts:        signed by the SSH CA; @cert-authority line in the
                        repo at files/ssh/known_hosts_ca
Log destination:        /var/log/ansible/ansible.log, mode 0600, rotated daily
Who to tell:            #platform, change-board@example.com
EOF

chmod -R go-rwx "$ESCROW"
ls -la "$ESCROW"

Task 2: Confirm the repository is genuinely self-sufficient

Before the drill, check what the repository actually pins. This is the audit that predicts your rebuild time.

# On the original controller
cd /path/to/automation-repo

# Is the Ansible version pinned anywhere?
cat requirements.txt 2>/dev/null || echo 'NO requirements.txt - core version not pinned'

# Are collections pinned?
cat collections/requirements.yml 2>/dev/null || echo 'NO collections requirements'

# Is there a documented build procedure?
ls -1 docs/ README.md 2>/dev/null | head

$ cat requirements.txt collections/requirements.yml

ansible-core==2.21.3
ansible-lint==26.6.0
molecule==26.6.0

---
collections:
- name: ansible.posix
  version: 2.0.0
- name: community.general
  version: 11.4.0
- name: containers.podman
  version: 1.17.0

Task 3: Start the clock

# On the BARE VM
date -Is | tee ~/rebuild-log.txt
echo "=== rebuild started ===" >> ~/rebuild-log.txt

# Confirm the machine is genuinely bare
command -v ansible || echo 'no ansible: good' | tee -a ~/rebuild-log.txt
python3 --version | tee -a ~/rebuild-log.txt

From here on, log every command and every moment you had to reach for something outside git:

log() { printf '%s  %s\n' "$(date -Is)" "$*" | tee -a ~/rebuild-log.txt; }

log "starting: install python and git"

Task 4: Build the controller from the pinned versions

log "installing base packages"
sudo apt-get update
sudo apt-get install -y python3 python3-venv python3-pip git

log "cloning the repository"
# Substitute your own values before running:
REPO_URL=https://git.example.com/platform/ansible.git
BRANCH=main

git clone --branch "$BRANCH" "$REPO_URL" "$HOME/automation"
cd "$HOME/automation"

log "creating the virtualenv at the documented path"
sudo install -d -o "$USER" -g "$USER" /opt/ansible
python3 -m venv /opt/ansible
/opt/ansible/bin/pip install --upgrade pip

log "installing the pinned python packages"
/opt/ansible/bin/pip install -r requirements.txt

log "installing the pinned collections"
/opt/ansible/bin/ansible-galaxy collection install \
  -r collections/requirements.yml -p "$HOME/automation/collections"

$ /opt/ansible/bin/ansible --version

ansible [core 2.21.3]
config file = /home/operator/automation/ansible.cfg
ansible collection location = /home/operator/automation/collections
executable location = /opt/ansible/bin/ansible
python version = 3.12.7

Compare that against what you recorded for the original. Every line that differs is a finding.

Task 5: Install what git could not hold

This is the part the drill exists to measure. Open the envelope.

log "OPENING ESCROW - vault password"
install -d -m 0700 "$HOME/.ansible-secrets"
install -m 0600 /path/to/escrow/vault-prod.txt "$HOME/.ansible-secrets/vault-prod"

log "OPENING ESCROW - automation ssh key"
install -d -m 0700 "$HOME/.ssh"
install -m 0600 /path/to/escrow/ansible_automation "$HOME/.ssh/ansible_automation"

log "configuring ssh host key trust"
# The CA public key IS in the repo; the client config is not
cat >> "$HOME/.ssh/known_hosts" < files/ssh/known_hosts_ca

log "verifying the key is accepted by one host"
# Substitute your own values before running:
PROBE_HOST=node1.example.com

ssh -i "$HOME/.ssh/ansible_automation" -o BatchMode=yes \
    -o ConnectTimeout=10 "ansible@$PROBE_HOST" 'echo reachable'

Task 6: Prove the controller works, read-only

The success criterion is a successful run, not a successful install.

cd "$HOME/automation"
export ANSIBLE_CONFIG="$HOME/automation/ansible.cfg"

log "verifying inventory resolves"
/opt/ansible/bin/ansible-inventory -i inventories/prod/hosts.yml --graph | head -20

log "verifying connectivity to the fleet"
/opt/ansible/bin/ansible -i inventories/prod/hosts.yml all \
  --private-key "$HOME/.ssh/ansible_automation" -m ping

$ ansible -i inventories/prod/hosts.yml all -m ping

node1 | SUCCESS => {"changed": false, "ping": "pong"}
node2 | SUCCESS => {"changed": false, "ping": "pong"}
node3 | SUCCESS => {"changed": false, "ping": "pong"}
node4 | SUCCESS => {"changed": false, "ping": "pong"}

Then the stronger test — decrypt the vault and run the real playbook in check mode:

log "verifying vault decryption"
/opt/ansible/bin/ansible-playbook -i inventories/prod/hosts.yml site.yml \
  --vault-id prod@"$HOME/.ansible-secrets/vault-prod" \
  --private-key "$HOME/.ssh/ansible_automation" \
  --check --diff | tail -20

log "=== rebuild complete ==="
date -Is | tee -a ~/rebuild-log.txt

Task 7: Compute the number and write the findings

cd "$HOME"
head -1 rebuild-log.txt
tail -1 rebuild-log.txt

grep -c 'OPENING ESCROW' rebuild-log.txt
grep 'OPENING ESCROW' rebuild-log.txt

Write controller-dr-findings.md:

# Controller rebuild drill — findings

Elapsed:            __ minutes
Escrow opens:       __
Blocked at any point on something not in git or escrow: yes / no

## Escrow contents actually needed
1.
2.
3.

## Things I wanted from the old controller and could not have
(These are the real findings. Each one is either a repository gap or an
escrow gap.)
-
-

## Version comparison
| Item             | Original | Rebuilt | Match |
|------------------|----------|---------|-------|
| ansible-core     |          |         |       |
| python           |          |         |       |
| ansible.posix    |          |         |       |
| community.general|          |         |       |

## Actions
- [ ]
- [ ]

Task 8: Turn the findings into a checklist

The drill’s output is a checklist somebody can follow at 03:00 without having done this before.

# Controller DR checklist

## Before you start
- [ ] Confirm the original controller is genuinely unavailable
- [ ] Notify #platform and the change board
- [ ] Retrieve the escrow envelope; record who opened it and when

## Build          target: 25 minutes
- [ ] Provision a VM: 2 vCPU, 4 GB, <OS from decisions.md>
- [ ] Install python3, python3-venv, git
- [ ] Clone <repo> branch <branch>
- [ ] Create the virtualenv at /opt/ansible
- [ ] pip install -r requirements.txt
- [ ] ansible-galaxy collection install -r collections/requirements.yml
- [ ] Verify: ansible --version matches decisions.md

## Restore from escrow    target: 10 minutes
- [ ] Vault password to ~/.ansible-secrets/, mode 0600
- [ ] Automation SSH key to ~/.ssh/, mode 0600
- [ ] SSH CA trust line appended to ~/.ssh/known_hosts
- [ ] Create the log directory with the mode from decisions.md

## Verify         target: 10 minutes
- [ ] ansible-inventory --graph resolves
- [ ] ansible all -m ping succeeds on every host
- [ ] ansible-playbook site.yml --check --diff completes
- [ ] Version table matches the original

## After
- [ ] Rotate the vault password and the automation key; the escrow was opened
- [ ] Update the escrow with the new material
- [ ] Record the elapsed time against the target

Validation

• ansible --version on the bare VM before Task 3 reports command not found.
• After the rebuild, /opt/ansible/bin/ansible --version reports the same ansible core version as requirements.txt pins.
• ansible-galaxy collection list on the new controller matches the versions in collections/requirements.yml, exactly.
• ansible -i inventories/prod/hosts.yml all -m ping succeeds on every host, using only the key from escrow.
• ansible-playbook site.yml --check --diff completes with no vault or connection errors.
• rebuild-log.txt has a start and end timestamp and at least three OPENING ESCROW lines.
• controller-dr-findings.md lists at least one thing you wanted from the old controller and could not have. If it lists none, the drill was too easy — add a component to the environment and run it again.
• The DR checklist has a time target per section and the actual elapsed time recorded against it.

Expected Outcome

new controller:
  /opt/ansible/            pinned virtualenv
  ~/automation/            the repository, at the production branch
  ~/automation/collections pinned collections
  ~/.ansible-secrets/      vault password, mode 0600
  ~/.ssh/                  automation key, mode 0600
  ~/rebuild-log.txt        timestamped, with escrow opens marked
  ~/controller-dr-findings.md
  ~/controller-dr-checklist.md

A controller that reproduces the original’s pinned versions and completes a read-only run against the fleet, plus a measured recovery time and a checklist somebody else can follow.

Troubleshooting

pip install -r requirements.txt fails with externally-managed-environment. PEP 668. That is exactly why the procedure builds a virtualenv; the error means you are installing into the system Python. Use /opt/ansible/bin/pip, not pip3.

ansible-galaxy cannot reach the Galaxy server. Either the internal mirror in decisions.md needs credentials you do not have — a finding — or the new VM has no route to it. Try the public server to distinguish the two.

A collection installs but a module in it is not found. The collections path is not where Ansible is looking. ansible-config dump --only-changed | grep -i collections shows the configured path; the repository’s ansible.cfg should set collections_path, and if it does not, that is a repository gap.

ansible all -m ping fails with Host key verification failed. The new controller has an empty known_hosts. The CA trust line from the repository is the correct fix. Do not set host_key_checking = False to get past it — a fresh controller with host-key checking disabled will happily connect to anything answering on those addresses, which during a recovery is precisely when you least want that.

Decryption failed on the first playbook run. The escrow password is wrong or stale. Check the vault file’s header for the identity label and confirm the escrow holds a password for that label — a repository that moved from one vault id to two, with only one in escrow, is a common and quiet gap.

The rebuild works but the versions do not match. Something is unpinned. Find which line of requirements.txt or requirements.yml lacks a version and fix it in the repository, not on the controller.

Cleanup

The drill built a new controller and opened an escrow envelope. Both need handling, and the escrow is the one that matters.

Step 1. Rotate everything the envelope contained. This is not optional housekeeping — it is the correct response to a disclosure:

# Substitute your own values before running:
REPO="$HOME/automation"

# Rotate the vault password
cd "$REPO"
/opt/ansible/bin/ansible-vault rekey \
  --vault-id prod@"$HOME/.ansible-secrets/vault-prod" \
  --new-vault-id prod@"$HOME/.ansible-secrets/vault-prod-new" \
  $(grep -rl '\$ANSIBLE_VAULT' inventories/ group_vars/ 2>/dev/null)

Rotate the automation SSH key by the procedure your fleet uses — generate, distribute, verify, then remove the old public key — and update the escrow with both new secrets.

Step 2. Securely remove the secret material from the drill controller:

cd "$HOME"
shred -u "$HOME/.ansible-secrets/vault-prod" 2>/dev/null \
  || rm -f "$HOME/.ansible-secrets/vault-prod"
shred -u "$HOME/.ssh/ansible_automation" 2>/dev/null \
  || rm -f "$HOME/.ssh/ansible_automation"

grep -rl 'REPLACE_ME' "$HOME" 2>/dev/null || echo 'no placeholder secrets remaining'

Step 3. Decide what happens to the drill controller itself.

# If destroying: verify what you are about to delete, then delete from the
# hypervisor. Substitute your own values before running:
DRILL_VM=controller-drill

echo "About to destroy: $DRILL_VM"
echo "Confirm this is the DRILL controller and not production:"
hostname
ls -la /opt/ansible/bin/ansible

Step 4. Keep the findings — they are the entire point of the drill:

mkdir -p "$HOME/ansible-lab-deliverables/controller-dr"
cp -a "$HOME/rebuild-log.txt" \
      "$HOME/controller-dr-findings.md" \
      "$HOME/controller-dr-checklist.md" \
      "$HOME/ansible-lab-deliverables/controller-dr/"

Copy them off the VM before you destroy it. A drill whose findings were on the machine you deleted is a drill you will have to run again.

What You Learned

• The install is not the hard part. Cloning a repo and pip-installing a pinned version takes twenty minutes. The time goes on the material that is not in git.
• An unpinned collection makes the controller unreproducible. A requirements.yml entry with no version: gives you a different controller on a different day, silently.
• Trust material is the forgotten escrow item. The SSH CA line or known_hosts is neither in git nor in the envelope on most teams, and its absence fails the first run in a way that looks like networking.
• --check --diff is the right final test and it is not sufficient, because command, shell and uri tasks are skipped. Decide which standard your organisation requires.
• An opened escrow is a disclosure and triggers a rotation. A drill is a real open, which is also how the rotation procedure stays exercised.
• A drill controller is a second production controller. Destroy it or promote it; a powered-off VM holding the automation key is a credential nobody is watching.
• The findings list is the deliverable. Every “I wanted to check the old controller for X” is either a repository gap or an escrow gap, and both are fixable in an afternoon — before you need them.