Skip to main content
RunBook Academy

← All checklists in Ansible

Annualdisaster-recovery

Checklist: Controller disaster recovery

20 items ·13 critical ·6 warn ·1 info

Run this annually as a readiness review, and run the post-rebuild items as the acceptance test of an actual rebuild. The two halves answer different questions: could we rebuild, and did the thing we rebuilt work.

Set VAULT_PASS_FILE to the escrowed password file - not the one already on the controller - before running the vault item. Using the controller own copy tests nothing, because the whole premise is that the controller is gone.

What a failure means

A finding here is never urgent and is always expensive to ignore. Nothing on this list causes an outage; every item on it decides how long an outage lasts.

The rebuild-performed and rebuild-timed items are the two that carry the others. A plan that has never been executed has an unknown duration and an unknown number of missing steps, and both of those are discovered under the worst conditions the estate ever produces.

What is genuinely irreplaceable

The repository is not the hard part. Git is replicated to every clone, and a controller can be reinstalled from a base image in minutes.

What is irreplaceable is the small set of things that exist only on the controller: the vault passwords, the private key the fleet trusts, and any credential file that was never meant to be in the repository. Everything else on this checklist is logistics. Those three are the reason a controller can be unrecoverable while every byte of the automation still exists.

Access this needs

The pin and collection checks are local and read-only, with one exception: ansible-galaxy collection download writes tarballs into the directory you name, so point it at scratch space. The post-rebuild connectivity and vault checks connect to hosts or read the escrow; the check-mode run connects and reports without changing.

Where the evidence goes

The measured rebuild duration, the date, and who performed it go into the recovery plan itself, replacing whatever number was there before. The findings go to named owners with dates. If the rebuild took four hours and the stated objective is two, the honest outcome is either a shorter rebuild or a longer objective - not an unchanged document.

Sign-off

  • Rebuild performed by: _____ Date: ___________
  • Duration: _________________ RTO: ____________
  • Escrow verified by: _______ Date: ___________

Critical13 items

  1. git status --porcelain --untracked-files=all
  5. pip install --dry-run -r requirements.txt
  6. ansible-galaxy collection verify -r requirements.yml --offline
  7. ansible-galaxy collection download -r requirements.yml -p ./collection-artifacts
  11. ansible -i inventories/production all -m ansible.builtin.ping --one-line | grep -v 'SUCCESS'
  12. ansible-playbook -i inventories/production playbooks/deploy.yml --check --diff
  13. grep -rlZ --include='*.yml' '^\$ANSIBLE_VAULT' inventories/production/ 2>/dev/null \
  | xargs -0 -r -n1 sh -c '
      ansible-vault view --vault-password-file "$VAULT_PASS_FILE" "$1" >/dev/null 2>&1 \
        || echo "FINDING: $1 will not decrypt with the escrowed password"' sh

Warning6 items

  2. ls execution-environment.yml 2>/dev/null && cat execution-environment.yml
  3. ansible-config dump | grep '^DEFAULT_LOG_PATH'
  4. ansible-config dump | grep -E '^CACHE_PLUGIN|^FACT_CACHING'
  6. ansible --version | head -1; ansible-galaxy collection list; pip freeze

Info1 item