What configuration management can and cannot reconstruct

Choosing to rebuild rather than restore is a bet on coverage, and in most estates the thing being bet on is a configuration management repository. It is not an unreasonable bet. That repository is usually the most complete written description of the estate anybody has, and it is the one artefact that can put a machine back without anybody remembering how the machine was built. It is also the bet that turns a bad afternoon into an unrecoverable position, because a description complete enough to build a host is not the same thing as a description complete enough to bring one back. The boundary between those two statements is exact, and it can be drawn on an ordinary Tuesday instead of discovered at four in the morning.

What a converged run against a clean host does produce

The argument here is not that configuration management is weak. Give it its due first, precisely, because the credit is what makes the debit legible.

It reconstructs package sets. Every package a role installs arrives on the new host, from repositories that must still exist and still carry the version being asked for. If the role pins versions, the versions come back; if it does not, whatever the repository holds today comes back instead, which is a different system wearing the same package names.

It reconstructs the contents of the files it manages. Files written literally from the repository and files rendered from templates both come back byte-for-byte, with the ownership and mode the task declares. This is the category people have in mind when they say the configuration is in Git, and it is genuinely the strongest thing on this page.

It reconstructs services and their enablement. A unit file placed, the service enabled so it survives a reboot, the service started so it is running now. Three separate assertions, all of them declarable, all of them reproduced.

It reconstructs the users and groups it declares. The account exists, it has the shell and the home directory and the supplementary groups the role names. Its numeric UID comes back only if the role states the number; otherwise the fresh host allocates the next free one, and the restored data directory is owned by an integer that no longer maps to the same account.

It reconstructs settings it templates. A rendered postgresql.conf, a firewall rule set, a scrape target list, a sysctl value — anything whose content is a function of variables the repository holds comes back by evaluating that function again.

The common shape of all five is worth stating once, because everything else in this lesson follows from it. Each is reconstructed exactly to the extent that something in the repository declares it. Configuration management is not a recording of the host; it is a program whose output happens to resemble the host, and re-running the program reproduces the output, not the host.

The state that has no declaring statement

Everything a machine acquired after convergence is outside that program, and the categories are ordinary rather than exotic.

Data written at runtime. Database contents, queue depths, the rows an application wrote while serving traffic, caches that turned out to be authoritative because nothing else holds the value. No role declares these, and no re-run produces them.

Uploaded files. Attachments, avatars, generated reports, exports somebody will ask for during the audit. They arrive through the application, they land in a directory the configuration created and left empty, and the configuration will create it and leave it empty again.

Generated keys and certificates. A role that runs a key generator when the key is absent reproduces a key, not the key. The rebuilt host presents a new SSH host key to every client holding the old one in known_hosts, a new self-signed certificate to every client that pinned the old one, and a new enrolment identity to whatever issued the last one. Material obtained through a web console, an API token minted once and never re-minted, and a certificate signed by an authority that requires human approval are all in this category and none of them regenerate on demand.

Machine identity. /etc/machine-id, a cluster member ID, a licence bound to a hostname or a hardware identifier, a node’s registration in a directory or a monitoring target list. The rebuilt host is a different machine to everything that recognises machines, and each of those relationships has to be repaired individually.

Accumulated log and metric history. The evidence you would most want during the incident is the first thing a rebuild discards. A converged host starts its history at the moment of convergence, which is after the event you are trying to explain.

Manual changes nobody committed. The fix applied at two in the morning that made the outage stop, and never went back into the repository. Convergence reproduces the repository faithfully, so it reproduces the host as it was believed to be rather than as it was, and the difference is invisible until the service behaves differently on the new machine.

Software that keeps its configuration in its own database. Dashboards in an embedded SQLite file, jobs in an application home directory, realms and clients in a relational schema, rules in a device’s own configuration store. The configuration management model can install the package and place one file next to it; the entire operational configuration lives inside a blob the model has no way to address. This is the category that surprises people most, because the work clearly is configuration, and it is nonetheless data.

Measured: the declared objects, and the bytes that were not among them

A container makes the boundary small enough to see in one screen. An application container was given a named volume for its data, and two files were written into it: one into the volume at /var/lib/app/orders.csv, and one into the container’s own writable layer at /etc/app-marker. Then the state was captured the way container state is commonly captured, by committing the container to an image — the container equivalent of believing that the declared artefact holds everything.

$ docker commit rbdr-app rbdr-committed:v1

--- and the file that was in the volume? ---
total 8
drwxr-xr-x    2 root     root          4096 Aug 28 13:48 .
drwxr-xr-x    1 root     root          4096 Aug 28 13:48 ..
>>> exit code: 0

total 8, and two entries that are the directory and its parent. The image carried the declaration — the package, the layer, the mount point — and the mount point came back as an empty directory, because its contents were never part of the image. The volume’s real location was reported as /var/lib/docker/volumes/rbdr-data/_data, outside the image entirely, and capturing it took a separate action against a separate object:

$ docker run --rm -v rbdr-data:/src:ro -v $PWD:/out alpine tar czf /out/rbdr-data.tgz -C /src .

  -rw-r--r-- 1 root root 160 Aug 28 13:49 /tmp/rbdr-out/rbdr-data.tgz

One hundred and sixty bytes, and they are the only bytes in the exercise that nothing could regenerate. The container and the volume were then both destroyed and the archive was unpacked into a volume that had never held anything. The data returned byte-identical, md5 9eb4e2ad8e08e1dcaaf87ababab964b0 on the original and on the copy. The file in the container layer did not:

$ docker run --rm -v rbdr-data-restored:/dst -v /tmp/rbdr-out:/in alpine tar xzf /in/rbdr-data.tgz -C /dst

--- but what about the file that lived in the container layer? ---
cat: can't open '/etc/app-marker': No such file or directory
>>> exit code: 1

Read the two halves together and the general rule falls out. The declared half was reproducible from an artefact and needed no backup. The accumulated half was recoverable only because somebody named it and copied it. The half that was neither declared nor named was simply gone, and nothing in the exercise reported its absence until something asked for it.

The declarative orchestration case has the same shape at a larger scale. Applied against k3s v1.36.3+k3s1, a manifest of desired state produced exactly four lines — namespace/rbdr-shop created, persistentvolumeclaim/rbdr-orders created, configmap/rbdr-config created and pod/rbdr-orders-writer created — and exit code 0. Four objects, all of them declared, all of them reproducible from the file. The orders the pod then writes into that claim appear in none of them, and re-applying the manifest on an empty cluster gives you the claim again, still empty.

The test that turns the boundary into a backup requirement

Because the boundary is precise, it is enumerable, and the enumeration is a half-day of work rather than an argument in a meeting. Build a host from nothing but the configuration repository, on infrastructure that has never run the service. Converge it until the run reports no changes. Then compare it against the machine it is supposed to replace, and write down what is still missing.

That list is the backup requirement. Not a list of things that would be nice to protect — the specific, named set of state that the estate’s strongest reconstruction tool provably cannot produce.

The comparison is worth doing mechanically rather than by inspection, because inspection finds the things you already suspected:

BUILT=/mnt/converged
LIVE_LIST=/backup/orders01/paths.lst
MISSING=/tmp/not-reconstructed.lst
: > "$MISSING"
while read -r path; do
  [ -e "$BUILT$path" ] || printf '%s\n' "$path" >> "$MISSING"
done < "$LIVE_LIST"
wc -l < "$MISSING"

Paths are the easy half. Run the same discipline against the categories that are not files: whether the rebuilt host presents the same SSH host key, holds the same certificates and private keys, resolves to the same numeric UIDs, is registered in the same monitoring targets and directory, carries the same licence binding, and answers a request that reads real data. A missing answer to any of those is an entry on the same list.

The reconstruction tool is itself part of the estate

There is a second-order dependency in all of this that catches careful people, and it is worth stating flatly: the configuration management system is infrastructure, and it has state of its own.

Its inventory is state. The list of hosts, their groups and their roles is not derivable from anything else, and a dynamic inventory is worse rather than better, because it is derived from a live platform that may be exactly what is unavailable. Its variables are state — group variables, host variables, the values that turn a generic role into this particular host. Its secrets are state, and they are the part with the sharpest failure mode: encrypted variables are useless without the key that opens them, and that key belongs somewhere with a different fate from the repository it decrypts.

None of that is protected by the fact that the repository is in version control, because version control protects the files and not the hosting. If the repository, the automation controller, the runner fleet and the secrets store all live inside the same platform as the production estate, then the loss of that platform removes the production systems and the tool intended to rebuild them in a single event. The recovery plan then contains a step that depends on the outcome of the recovery plan.

STAMP=$(date -u +%Y%m%dT%H%M%SZ)
OUT=/backup/automation/controller-$STAMP.tar.gz
tar -czf "$OUT" \
  /srv/infra/inventory \
  /srv/infra/group_vars \
  /srv/infra/host_vars \
  /etc/ansible/ansible.cfg
sha256sum "$OUT" > "$OUT.sha256"

Declarative infrastructure tooling has the same problem in a more concentrated form. A state file is the mapping between the declarations and the real objects they created; without it the tool does not know that the resources exist and will propose creating a second set. That file is not derivable from the configuration, it lives in a backend that is frequently inside the environment being rebuilt, and it is the single most valuable small file in an infrastructure-as-code estate.

The rule that falls out is short. Recover the ability to reconstruct before you attempt to reconstruct, and prove that ordering by rehearsing it: rebuild the controller, restore its inventory, variables and key material somewhere it has never run, and only then converge a host. An estate that has never done that does not know whether its rebuild path starts inside the outage.

Production discipline

1. Converge a clean host and write down what is still missing. The difference between that machine and the one it replaces is the backup requirement, expressed as a list of named items rather than as a category.
2. Treat generated identity as data, not as configuration. A role that creates a key when none exists reproduces a different key; SSH host keys, TLS private keys, enrolment tokens and machine identifiers have to be captured and stored with a fate independent of the host.
3. Back up the interior of anything that stores its configuration in a database. The declaration is satisfied by the package and the directory, exactly as docker commit was satisfied by a mount point that listed total 8 and nothing but . and ...
4. Protect the automation controller, its inventory, its variables and its key material as production systems. Version control preserves the files; it does not preserve the hosting, the runners or the secret that decrypts the variables.
5. Rehearse the rebuild path before the estate depends on it. Restore the reconstruction tool onto infrastructure that has never run it, then converge a host from it, and record which step first needed something nobody had captured.

Cross-course references

• Ansible for Production Sysadmins — Part XXXVI (Drift and Convergence) explains why a run reports no changes for state it never declared, which is the mechanism behind the blindness described here, while Part L (Automation Disaster Recovery) and Part XXI (Secrets Management) cover the second-order dependency this lesson insists on: the controller, its inventory and its vault key are themselves state that has to survive the event.
• Terraform for Production Sysadmins — Part IX (State: The Core Production Concept) and Part XII (State Recovery and Backup) describe the file that maps declarations to real resources, which is the clearest example of the point made above that the reconstruction tool has state of its own and that losing it breaks reconstruction even when every declaration survives.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part LXXIV (Reconciliation) and Part LXXV (Drift) formalise the same comparison loop for a cluster, and Part LXXXII (GitOps Secrets) covers the material that cannot live in the repository being reconciled, which together explain why a reconciled cluster is not a recovered one.

Quiz