The timed rebuild drill · Ansible

Five lessons of plan. This one is the only lesson in the part that produces evidence.

Everything before it can be completed on paper and reviewed favourably. The repository has the right contents, the pins are exact, the escrow exists, the runbook is written. None of that is a claim about whether a controller can be rebuilt; it is a claim about whether the artefacts that a rebuild would use are present.

The drill is the difference. It is a scheduled exercise with a clock, a defined start condition and a binary success criterion, and it has one property that makes it uniquely valuable: it cannot be passed by knowing things, because the operator is chosen not to know them.

The drill, defined

The clock is not there to make anyone feel bad. It is there because RTO is a number your organisation has committed to somewhere, and this is the only measurement of it that exists.

Who runs it

This decides whether the drill measures anything.

The person who built the original controller cannot run the drill, for the same reason an author cannot proofread their own writing: they will supply the missing step without noticing it was missing. They will type the extra export from muscle memory, know that the inventory needs a flag, and remember that the internal Galaxy server needs a proxy — and each of those is exactly the finding the drill exists to produce.

The right operator is a competent engineer who has not built this controller. New team members are ideal, and running the drill as part of onboarding is a pattern worth stealing: it produces a tested runbook and a productive new colleague from the same afternoon.

The sequence

Start the clock. Provision a fresh host and record what OS and how it was obtained - "provisioned by the platform team, 40 minutes" is a legitimate and important part of the RTO.
Clone the repository. If this needs a credential, that credential is a DR dependency and belongs in the escrow inventory - most teams discover it here.
Read the rebuild runbook from the clone. The operator follows it literally. Where it is ambiguous, they record the ambiguity rather than interpreting it.
Install the pinned toolchain: the virtualenv from lesson 3, or the execution environment from lesson 5, whichever the runbook names.
Restore the secrets from escrow: vault password per vault ID, automation private key, become password if used.
Verify the toolchain: ansible --version, the collection diff against requirements.yml, and ansible-galaxy collection verify.
Verify the inventory resolves: ansible-inventory --graph, and confirm the host count matches expectations.
Prove decryption: ansible-vault view on every encrypted file, redirected to /dev/null.
Run the read-only play against a small representative group, then against the whole fleet if the group succeeds.
Stop the clock. Record the elapsed time, every question asked, and every step that was not in the runbook.

The verification play

The success criterion needs to be a real play against real hosts, not ansible --version. It also must not change anything.

Read-only / Safedrill-verify.yml - proves connectivity, privilege, facts and vault— READ-ONLY throughout: ping connects and returns, setup gathers facts, assert evaluates, debug prints. become: false is deliberate - see the note below on why the privileged check is separate.

- name: Rebuild drill - prove the controller works, change nothing
hosts: all
gather_facts: true
become: false
tasks:
  - name: Connectivity and a working remote interpreter
    ansible.builtin.ping:

  - name: Facts are usable, so templates and conditionals will resolve
    ansible.builtin.assert:
      that:
        - ansible_facts['distribution'] is defined
        - ansible_facts['default_ipv4'] is defined
      fail_msg: 'Facts incomplete on {{ inventory_hostname }}'

  - name: The vault password decrypts what the plays actually use
    run_once: true
    ansible.builtin.assert:
      that: vault_smoke_test == 'decrypted'
      fail_msg: 'Vault variable did not resolve - the escrowed password is wrong for this vault ID.'

  - name: Report the toolchain that produced this run
    run_once: true
    ansible.builtin.debug:
      msg: >-
        core {{ ansible_version.full }},
        python {{ ansible_playbook_python }},
        {{ ansible_play_hosts_all | length }} hosts reached

vault_smoke_test is a variable you place in an encrypted group_vars file for exactly this purpose, holding the literal string decrypted. It is not a secret, and its only job is to make “the escrowed password opens the file the plays actually read” a task that either passes or fails rather than an assumption.

Read-only / Saferun the drill verification— READ-ONLY: every task in the play only reads. Run against the canary group first - if the controller is misconfigured you want to find out against three hosts, not four hundred.

# Confirm the target set before running anything.
ansible-playbook -i inventory/ drill-verify.yml --limit canary --list-hosts

# The proof.
ansible-playbook -i inventory/ drill-verify.yml --limit canary
ansible-playbook -i inventory/ drill-verify.yml

The privileged path needs its own proof, and it is not read-only

The play above runs with become: false, so it proves the automation account can log in and gather facts. It does not prove that account can escalate — and a rebuilt controller with a working SSH key but a broken become password can do nothing that matters.

Proving escalation requires actually escalating, which is no longer read-only in the strict sense even if the task itself only reads. The proportionate form is a single become: true task that reads something harmless on one canary host, run as a deliberate, separately recorded step of the drill.

Do not fold it into the fleet-wide play. Do not skip it either, because “the drill passed and then the first real change failed on sudo” is a recorded outcome, not a hypothetical.

The three findings, and what to do with each

A first drill essentially always produces these. Their reliability is the argument for running one.

Finding	What it looks like	The repository change
An undocumented manual step	The operator is stuck; the builder says “oh, you need to export `ANSIBLE_CONFIG` first”	Add the step to the runbook, or better, remove the need for it
A collection installed by hand	The rebuild succeeds and the first real play fails with “couldn’t resolve module/action”	Add it to `requirements.yml`, pinned
A credential that only existed on the dead host	The operator cannot clone, cannot decrypt, or cannot connect	Add it to the escrow inventory and re-escrow

Every finding closes as a commit. A drill whose output is a document describing what went wrong has produced a report; a drill whose output is a merged pull request has produced a fix. Track them the way you would track incident actions, because that is what they are — you have simply run the incident on purpose.

Under the hood

Why does a drill find things that a careful review does not?

Because reviewing a runbook and executing one exercise different faculties, and the gap between them has a name in human factors: knowledge-in-the-head versus knowledge-in-the-world.

When you read your own runbook, every ambiguity resolves silently against what you already know. “Install the collections” reads as complete because you know it means -r requirements.yml -p ./collections with the path matching ansible.cfg. The sentence is not wrong; it is underspecified, and the underspecification is invisible from inside your own head. A reviewer who also built controllers reads it the same way, which is why peer review does not catch these either.

The operator who has never built this controller has none of that context, so the ambiguity becomes a stop. That stop is the measurement. The drill is not testing whether the runbook is correct — it usually is — but whether it is sufficient, and sufficiency is only observable by execution against someone who lacks the missing knowledge.

The same mechanism explains why the second drill finds far less than the first, and why that is not evidence you can stop. The first drill extracts most of the knowledge-in-the-head into the runbook. What accumulates afterwards is new: the collection added last month, the inventory source that moved, the credential that rotated. The drill’s job after the first one is not discovery but detection of drift, which is why it belongs on a schedule rather than in a project plan.

What happens if the drill fails and there is no time to finish it?

Record the failure and the elapsed time at the point you stopped. That is a result, and it is more useful than the successful drill you did not have time for.

The instinct is to treat an unfinished drill as invalid and reschedule. Resist it, because the elapsed time at the point of failure is your current actual RTO lower bound, and it is a number your organisation almost certainly does not have. “We stopped after four hours, having got as far as the collection install” is a finding with budget implications. “Drill rescheduled” is not.

Write down, specifically:

The wall-clock time at the stop, and what phase it was in.
The blocking step, quoted from the runbook, and what the operator did not know.
What would have unblocked it — a person, a credential, a file, a network path. This is the DR dependency you did not know you had.

Then decide, deliberately, whether to fix and re-run or to accept the current RTO. Both are legitimate; only the undocumented version is not.

One caution about the tempting shortcut: finishing the drill by having the original builder take over does produce a completed rebuild, and it measures nothing about your documentation. If you do it — and it is reasonable, because a half-built controller is untidy — record the two phases separately, with the clock stopping when the operator stopped.

Cadence and scope

Full drill, fresh host, unfamiliar operator: annually at minimum, and after any structural change to the controller build.
Escrow-only exercise - open it, prove every vault ID decrypts, prove the SSH key authenticates against a canary: quarterly. This is an hour and catches the rotated-key failure from lesson 4.
Pin verification - the collection diff and ansible-galaxy collection verify against requirements.yml: on every CI run, because it is cheap and it is exactly the check that catches a hand-installed collection the day it appears.
Runbook read-through by someone who has not read it: whenever a new engineer joins. Free, and it finds ambiguities before the annual drill does.

The fire-drill discipline itself — how to schedule exercises so they happen, how to keep them from becoming ceremonial, how to write up results people act on — is developed at length in the Proxmox course’s DR part. This lesson deliberately does not restate it. What is specific to Ansible is the content of the drill: the toolchain, the escrow, the inventory, the vault, and the read-only run that proves all four at once.

Knowledge check

Why this matters

This part opened with a scenario and has spent five lessons building an answer to it. The honest position is that none of that answer is worth anything until it has been executed, because every team that has run a first drill has found something, and the something is never the thing they expected.

The value is concentrated in an uncomfortable place. A drill produces a list of ways your plan does not work, delivered by a colleague who could not follow your documentation, measured against a clock. It is designed to be mildly embarrassing, and teams avoid it for that reason rather than for any reason to do with cost — a drill is one engineer for one day, which is less than the meeting about whether to run one.

Set against that: the alternative venue for discovering the same list is an incident, at 03:00, with a fleet nobody can change and a business asking for an estimate you have no basis for. Every finding is the same finding. Only the conditions differ.

The number the drill produces is the one to hold on to. Not “we have a DR plan” but “we rebuilt the controller in five hours forty, and here are the three commits that came out of it”. That is a statement about the estate that somebody can act on.