Ansible and the boundary of reconstruction

Code rebuilds infrastructure and backup restores state is a claim about a whole estate, and it is easier to agree with than to act on. It becomes actionable one machine down, where it turns into a list of paths, accounts, tables and keys. Ansible is a good tool to draw that line with, because every resource it manages is named by a task somebody wrote: what it reconstructs can be read off the repository, and the remainder is that host’s backup requirement.

What the orders-api role puts back on a bare host

orders-api runs on a single Debian host: a Python application behind nginx, a PostgreSQL instance beside it, a system account, a data directory holding customer-supplied attachments, and an environment file rendered from group variables. One role builds it.

- name: Packages
  ansible.builtin.apt:
    name: [nginx, postgresql]
    state: present

- name: Service account
  ansible.builtin.user:
    name: orders
    uid: 4021
    system: true

- name: Upload directory
  ansible.builtin.file:
    path: /var/lib/orders/uploads
    state: directory
    owner: orders
    group: orders
    mode: '0750'

- name: Environment file
  ansible.builtin.template:
    src: orders.env.j2
    dest: /etc/orders/orders.env
    owner: orders
    mode: '0640'

- name: Database
  community.postgresql.postgresql_db:
    name: orders
    owner: orders
    state: present

All of it comes back on a machine that has never run the service: packages from a repository that must still exist and still carry the version asked for, the account with the numeric UID 4021 because the role states it, the directory with its declared owner and mode, the environment file byte-for-byte as the template renders it against this host’s variables, and — in the tasks omitted above — the unit file, its enablement and its running state as three separate assertions.

The database task is where the boundary becomes visible without any argument. state: present is satisfied by a database called orders existing and owned by the right role, and that is the entire assertion. Rows are not an argument to the module, so a converged host reports success with an empty schema and the same success with four years of orders in it. The task is not weak; it is precise, and its precision is the whole lesson.

The state with no declaring task

Everything the machine acquired after convergence sits outside that list, and the categories are ordinary.

The rows. Customers, orders, line items, the sequence values behind them. No task names them, and re-running the role produces the database object again, not its contents.

The uploads. /var/lib/orders/uploads is declared as a directory with an owner and a mode. Its contents are not an argument to ansible.builtin.file, so the reconstruction returns the directory and returns it empty, as faithfully the second time as the first.

The history. The journal, the nginx access logs, an application audit table. A rebuilt host’s history begins at convergence — after whatever you are trying to explain.

The generated material. The SSH host key the package regenerates on install, a TLS private key a task creates only when the file is absent, the session-signing secret minted once by an installer. Each reproduces a value rather than the value, and the consequences land elsewhere: clients holding the old host key refuse to connect, sessions signed with the old secret are rejected, and anything encrypted with it is unreadable on a host that is otherwise perfectly configured.

The identity and the registrations. /etc/machine-id, the monitoring target’s registration, a licence bound to a hostname: the rebuilt host is a different machine to everything that recognises machines.

The numeric UID couples the two halves together and so belongs in a category of its own. A restored archive carries numeric owners, not names. If the role pins uid: 4021 the restored files land owned by the account meant to own them; if it does not, the rebuilt host allocated some other number, the tree is owned by an integer nobody recognises, and the service starts and cannot read its own data. Pinning UIDs and GIDs is what makes a restore land correctly on a host the repository built.

Idempotence is a recovery property, not a style preference

Idempotence usually gets taught as good practice: a playbook reports changed only when it had to act, which makes runs quiet and diffs meaningful. In a recovery it becomes a functional requirement, because recovery is where you run the same playbook against a host in an unknown, partial state — a fresh VM, then that VM after a restore that failed halfway, then again after somebody fixed something by hand at two in the morning. Three ordinary habits break the property.

Tasks that shell out without a guard re-perform their action every run. ansible.builtin.command reports changed unconditionally unless given creates, removes or an explicit changed_when, so an unguarded schema-initialisation command runs again on a host that already has a schema. A guard pointing at the wrong witness is worse because it is invisible: if creates names a marker file that lives outside the backup, the restored host does not have the marker and the command fires over restored data.

Tasks that assume existing state fail on the machine that has least of it. ansible.builtin.lineinfile editing a file a package’s post-install script was supposed to create will, on a host where the service never started, either fail or leave a file holding one line.

Handlers are the subtlest, and the one that costs a recovery an hour. A handler runs only when a task notifies it, and a task notifies only when it reports changed. Converge a host, restore the data underneath it, then re-run the playbook: nothing changes, so nothing notifies, so the service is never restarted and the process keeps serving whatever it read at start-up. The recap is clean and the service is wrong.

$ ansible-playbook site.yml --limit orders01

PLAY RECAP *********************************************************************
orders01                   : ok=14   changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

Read the fields: counts of tasks that matched, tasks that acted, and the ways a run can go wrong. No field describes content, which is why the same line is printed for a host whose database holds every order and for one whose database is empty.

Ordering: the run that overwrites what the restore returned

The boundary explains what a playbook cannot bring back. Ordering explains how a playbook destroys what something else brought back, and it surprises careful teams because every task involved is correct. The dangerous sequence is the natural one: build the host, restore the data, then run the playbook once more to be sure everything is configured. Consider what that last run does.

The template task rewrites /etc/orders/orders.env with the repository’s rendering. If the live file had held a value nobody ever committed — a token pasted in during an integration, a tuned worker count — the restored copy carried it and the run has just discarded it. The file task with recurse set walks the restored tree applying owner, group and mode to everything it finds, which is occasionally the fix and occasionally the damage: a numeric mode applied recursively is applied to directories as well as files, which is what symbolic modes exist to avoid. And a retention role that deletes files older than a stated age will delete restored files, because a faithful restore preserves their original timestamps and they are genuinely old.

None of that is a bug: every one of those tasks is idempotent and doing what it was told. Idempotence guarantees convergence towards the declared value, so when the declared value and the restored value disagree, convergence is the mechanism by which restored data is destroyed.

The correct sequence separates the two halves and never lets the second run over the first. Converge the bare host with the build tasks only, leaving the unit enabled but stopped. Restore the data into the paths those tasks created. Then run a narrow post-restore play that fixes ownership deliberately, starts the service and verifies it. Tag the plays so the tool enforces that distinction rather than the operator remembering it.

HOST=orders01.example.net
PLAY=/srv/infra/site.yml
STAGE=/var/tmp/orders-restore
ansible-playbook "$PLAY" --limit "$HOST" --tags build --skip-tags data --check --diff
ansible-playbook "$PLAY" --limit "$HOST" --tags build --skip-tags data
restic restore latest --target "$STAGE" --include /var/lib/orders
ansible-playbook "$PLAY" --limit "$HOST" --tags post-restore -e "stage_dir=$STAGE"

The --check --diff line is the one people skip and the one worth insisting on. Run against a host that already holds restored data, it prints the diff of every managed file the run would rewrite — the list of what the playbook is about to take away.

Inventory, group_vars and the vault password are recovery material

The playbook is the visible half of the reconstruction tool; its inputs are the half that goes missing.

Inventory is state in its own right. Which hosts exist and which groups they belong to is not derivable from any role — a role describes what a member of a group looks like, and the inventory is the only record of the membership. Dynamic inventory is worse rather than better here, because it is computed at run time from a live platform API, and during a site failure that platform is frequently what is unavailable.

Group and host variables are the values that turn a generic role into this particular machine: the pinned UID, the database name, the certificate subject, the peer addresses. Converging without them produces a host rather than this host, and the difference often surfaces only when something else refuses to talk to it.

The variables held under ansible-vault are the sharpest case, for a reason familiar from Part IX. The ciphertext lives in Git, which is exactly why people feel covered: versioned, replicated to every clone, surviving the loss of any single machine. The password that opens it does not live in Git, and if its only copy is a file on a controller inside the estate, the encrypted variables end up where this course measured a backup repository.

$ restic snapshots

Fatal: wrong password or no key found
>>> exit code: 12

Nothing was corrupted and nothing was missing. The capture’s own summary is the sentence to carry across to automation: the data survived the disaster and the ability to read it did not. Substitute an encrypted group_vars/all/vault.yml and the shape is identical — the file is in three git remotes and nobody can render it into a host. The fix Part IX established transfers directly: a second, independent way in, held by people who do not administer production. The same capture added a recovery passphrase with restic key add, then destroyed production again.

$ restic --password-file /work/recovery-pass restore latest --target /work/rec

restoring snapshot b96ba7cf of [/work/prod2] at 2026-08-28 14:04:52.481565631 +0000 UTC by root@17dffded9807 to /work/rec
Summary: Restored 3 files/dirs (38 B) in 0:00
>>> exit code: 0

recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0

Byte-identical, using a credential the production host never held. Ansible offers the same structure through multiple vault identities: a second vault id whose password the recovery team holds, applied to the files the recovery path needs and escrowed separately from the controller and from the repository it decrypts.

Production discipline

1. Read the boundary off the repository before an incident forces you to. Every resource a task names is reconstructed; everything else is a backup requirement. postgresql_db with state: present is satisfied by an empty database, and that fact locates the line for the whole service.
2. Pin numeric UIDs and GIDs for every service account a role creates. A restored archive carries numbers, not names, so a role that lets the host allocate orders its own UID produces a machine whose service cannot read the data the restore returned.
3. Converge first, restore second, start third, and enforce it with tags. Build tasks run against a bare host with the unit stopped; the restore lands in the directories those tasks created; a narrow post-restore play starts the service. Never re-run the full playbook against a host already holding restored data.
4. Point --check --diff at a restored host before any real run. The diff is the list of managed files the run would rewrite, including anything a retention task considers old because the archive preserved its timestamps.
5. Escrow the vault password the way Part IX escrowed the repository key. With the passphrase on the lost host, an intact repository answered Fatal: wrong password or no key found with exit code 12; with a second passphrase held elsewhere, the same data came back with exit code 0 and md5 9eb4e2ad8e08e1dcaaf87ababab964b0 unchanged.

Cross-course references

• Ansible for Production Sysadmins — Part XII (Idempotency and Change Reporting) and Part XVI (Handlers) supply the mechanism this lesson depends on, explaining why a changed=0 recap cannot notice a restore and why a handler-driven restart silently does not happen; Part IV (Inventory Fundamentals) and Part L (Automation Disaster Recovery) cover the inventory and controller state treated here as recovery material.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part II (The Secret Lifecycle) and Part XV (KMS, HSM and Key Protection) develop the custody rules the vault password needs here, and Part XVIII (Incidents and Recovery) covers the regeneration problem this lesson meets when a role produces a key rather than the key.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part CX (Ansible Delivery Pipeline) is where the build-versus-post-restore tag separation described above has to be enforced, and Part XXXV (Secrets in Git) explains why ciphertext safely committed is not the same as ciphertext openable during a recovery.

Quiz