Code rebuilds infrastructure; backup restores state · Backup & DR

Validating a recovered database ended Part XIV with a comparison, and that comparison assumed the infrastructure underneath it was already back. Part XV asks how it gets back. Infrastructure is where this course’s opening question arrives with an answer attached: the estate is described in code, the code is in a repository, and the repository is protected. Most of that answer is true, which is exactly what makes the false part — the business’s data — easy to miss.

What a declarative description contains, and what it cannot

A declarative configuration is a statement about what should exist, written by a person, before the thing exists. That sentence contains the whole boundary. Terraform configuration says that a database instance of a given size and engine version should exist in a given network; it contains no rows. An Ansible role says that a package should be installed and a file should have a given content; it contains neither the log the service wrote nor the key the service generated on first start. A Kubernetes manifest says that a claim for 64Mi of storage should exist and that a Pod should mount it; it contains none of the bytes the Pod then writes.

Everything in an estate therefore falls into one of two categories, and the categories do not follow system boundaries. They follow items.

Reconstructable from code covers compute instances and the images they boot, networks, subnets, routes and security groups, load balancers and their listeners, DNS zones whose records are committed, identity roles and the policies attached to them, installed packages and pinned versions, service unit files, firewall rulesets, monitoring and alerting rules, and the whole set of API objects in a cluster. These have one property in common: someone wrote them down, and running the code again produces them again.

Irreplaceable state covers database contents, the objects inside a bucket, the filesystems inside persistent volumes, certificate authority private keys and the record of what they have issued, infrastructure state files, backup catalogues and repository indexes, secrets generated at first boot rather than supplied, queue contents and scheduler position, artefacts that were built once, and the audit logs a regulator expects to still be there. These have the opposite property: nobody wrote them down, a running system produced them, and executing the code a second time produces a fresh empty version of the container they lived in.

The split cuts through individual systems rather than around them. The database resource is code; the rows are state. The bucket, its policy and its lifecycle rules are code; the objects are not. The cluster is code; the volumes are not. Teams that classify by system — “the database is backed up” — end up protecting whichever half the sentence happened to mean, and the missing half is discovered during the rebuild.

Measured: every object returned, the volume was empty

The claim is testable, so it is worth testing rather than asserting. A single-node k3s cluster was given a complete set of manifests: a Namespace, a PersistentVolumeClaim, a ConfigMap and a Pod. The Pod wrote two order lines into the volume. The namespace was then deleted, which released the claim and reclaimed the volume, and the identical manifests were applied again — the move every team reaches for first, and the move people mean when they say everything is in Git.

Configuration changere-applying the complete desired state after the deletion— EXECUTION VALIDATED on k3s v1.36.3+k3s1. Step 5 of the desired-state capture. The capture records this step's output rather than the command line it ran, so the filename shown stands for the manifest file applied; the capture's two-space indentation of output has been removed and the '---' marker lines are the capture's own.

$ kubectl apply -f rbdr-shop.yaml

namespace/rbdr-shop created
persistentvolumeclaim/rbdr-orders created
configmap/rbdr-config created
pod/rbdr-orders-writer created
>>> exit code: 0
pod Ready after 6s

--- every object is back ---
persistentvolumeclaim/rbdr-orders   Bound   pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93   64Mi   RWO   local-path   <unset>   6s
configmap/kube-root-ca.crt   1     6s
configmap/rbdr-config        1     6s
pod/rbdr-orders-writer   1/1   Running   0     6s

--- and the business data? ---
total 8
drwxrwxrwx    2 root     root          4096 Aug 28 14:34 .
drwxr-xr-x    1 root     root          4096 Aug 28 14:34 ..
cat: can't open '/data/orders.csv': No such file or directory
command terminated with exit code 1

Read the two halves against each other. Every declared object returned, the apply exited 0, the Pod was 1/1 Running and Ready after 6s, and the claim reached Bound. Any dashboard watching object health would have gone green. The directory listing shows total 8 and two entries, both of them the directory itself and its parent, and reading the orders file returned cat: can't open '/data/orders.csv': No such file or directory with exit code 1.

One detail carries the general case. The claim bound to pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93, and the capture records the original data as having lived under pvc-6edd5db0-25f4-4b33-b26b-e2ad78aac9cf. The rebuild did not reattach the old volume. It provisioned a new one, because the manifest asked for 64Mi of ReadWriteOnce storage and a manifest cannot ask for bytes. Kubernetes is only the cheapest place to observe this quickly. Substitute a Terraform configuration for the manifests and a managed database for the claim and the shape is identical: the resource returns, empty, with a new identifier, and everything that depended on the old identifier has to be found and repointed.

The inventory question: is there an upstream that still holds a copy?

Classification stops being a philosophical exercise the moment it is written as a question with a per-item answer. For every item in the estate, ask: if this were destroyed right now, does a copy exist outside my estate that I can still fetch, and does my code identify it precisely enough to fetch the same one?

Both halves matter. A distribution’s package archive, a container registry, a provider or module registry, a vendor’s release page and a base image are all upstreams that somebody else operates. When the answer is yes, the item is genuinely reconstructable — and the reconstruction now depends on a third party’s availability and retention policy, which belongs in the recovery plan as a named dependency rather than as a certainty. Versions get yanked, tags get moved, registries have outages, and an unpinned version means the rebuild fetches something that is merely newer rather than the same.

When the answer is no, the item is state and a backup is the only mechanism that returns it. Nobody upstream holds your certificate authority’s private key, your state files, your backup catalogue, your rows or your uploads. You are the upstream. There is no third party to be disappointed by, and equally none to fall back on.

The instructive answers are the ones in between, where an upstream holds the inputs but not the outputs. An image built from a Dockerfile is reconstructable only while every base layer, package version and build-time download it referenced is still fetchable, which is a stronger condition than the Dockerfile being in Git. A compiled artefact promoted through environments is reproducible in principle and, in practice, only if the build is deterministic and every input is pinned. Items in this band should be recorded as state until somebody demonstrates the rebuild, because assuming they are code is the assumption that fails silently.

Why reconciliation produced a new empty volume rather than the old full one

Reconciliation compares a desired description against observed reality and acts to remove the difference. Everything about the measured result follows from what that comparison is able to see.

The claim’s spec describes a request: a storage class, an access mode and a quantity. The recreated claim was a new object with nothing bound to it, so the local-path class named in its spec had a volume provisioned to satisfy the request. The node directory holding the bytes is named for the volume rather than for the claim, which is why the capture’s path moved to pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93 when a new volume appeared under the same claim name. The reconciler’s success condition — an object exists whose observed attributes match the spec — was fully met by an empty 64Mi volume, because no field in the spec describes contents.

The same holds outside Kubernetes. Terraform’s documentation describes state as the record binding configuration to real remote objects, and a plan is a comparison between configuration, that record, and the attributes a provider reports. Those attributes are the resource’s metadata: size, class, version, tags, network placement. Telling a populated database apart from a freshly created one would mean comparing rows, and rows are not among the attributes that comparison has in front of it — which is why a plan can report no difference at all on an instance that came back empty.

This is the same principle Part I established for verifying a repository: a check that avoids reading the bytes has not verified the bytes. Here it appears in its constructive form. A system that never reads your data cannot recreate your data, and no amount of successful reconciliation is evidence about content.

The code drifts, and the drift surfaces during the rebuild

A repository describes intent. The estate is what actually exists. The two diverge continuously, and a rebuild reconstructs the intent — including the places where the intent is now wrong.

Drift arrives from ordinary work. Someone fixed an incident through a console at 03:00 and the change was never committed. A controller or operator inside the platform created resources nobody wrote down. A provider version changed a default and the estate quietly took the new one. A resource pre-dates the repository and was never imported. Each of these is invisible while the estate is running, because the running estate is the thing that works, and each becomes visible at the worst moment: the rebuild produces something that differs from what was lost, in ways nobody has a list of.

Both major configuration tools offer a read-only way to ask the question. Terraform’s plan compares the configuration against the recorded state and the provider’s view of the real objects. Ansible documents check mode as a way to run a playbook without making changes on remote systems, reporting what would change, with diff mode showing the differences in file content. Neither is a backup and neither proves the rebuild works, but running them regularly converts drift from something discovered during recovery into something noticed on a Tuesday.

"It is all in Git, so we can rebuild it" — as an answer to "are we backed up"

The sentence is not false, and that is what makes it an anti-pattern rather than a mistake. It is a precise statement about one of the two categories, offered as though it covered both. In the measured capture the repository was complete, the apply exited 0, every object returned, and the orders did not.

Two variants travel with it. The first is that the state file lives in a managed backend, so its protection is somebody else’s problem — a recovery plan that names no owner for the state file has no owner. The second is keeping a certificate authority’s private key on the host the code recreates, so that the key exists only inside the thing being rebuilt. Ask the person making the claim which category each item is in. If they have not separated them, one category is protected and the other has not been considered.

An unproven rebuild is a claim

The repository has probably never been executed against nothing. Every apply it has ever seen was a small diff against an estate that already existed, and the from-empty path — no account resources, no cluster, no state file, no pre-existing identities — is a path that has never once run. Ordering dependencies that were satisfied historically are unproven in a fresh run. Hard-coded identifiers refer to objects that will be recreated with new ones. Credentials that exist because a person created them once do not appear at all.

Rebuild duration is in the same position. It is not a property of a tool, and no product supplies it. It follows from a stated architecture and stated assumptions: how many resources there are, how serialised the dependency graph is, how long each provider takes to create its objects, how much of the work can run in parallel, and — usually dominating everything else — how long the data restore takes on top of the finished infrastructure. A figure produced any other way is a guess wearing a unit.

The evidence that settles both questions is one exercise: rebuild into a clean target that shares nothing with production, restore the data into what was rebuilt, and compare the result against the original. The capture ends by doing exactly that with the smallest possible instance of the problem — a tar of the volume directory taken before the second deletion, unpacked into the directory backing the newly bound claim.

Service impact possiblerestoring the volume contents into the volume the rebuilt claim bound to— EXECUTION VALIDATED on k3s v1.36.3+k3s1. Step 7 of the desired-state capture, where NEWDIR is the node directory backing the newly bound claim. The capture's two-space indentation of output has been removed.

$ tar xf /tmp/rbdr-pv-backup.tar -C $NEWDIR

ORDER-1001,4500.00
ORDER-1002,1250.00
recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - the application data is back, byte-identical

The recovered digest 9eb4e2ad8e08e1dcaaf87ababab964b0 matches the original, and the capture’s own conclusion is the sentence this part is built on: full recovery needed both the desired state to rebuild the objects and a separate data backup to refill the volume, and neither alone was enough. That is the finished shape of an infrastructure recovery plan — two mechanisms, two inventories, one exercise that runs them together.

Production discipline

Classify every item as code or state before designing any protection for it. Do it per item, not per system: the database instance and the rows in it belong to different categories and need different mechanisms, and a plan written at system granularity protects whichever half the sentence meant.
Apply the upstream test in writing and name the upstream. “Reconstructable” is only true while a specific third party still serves a specific pinned version. Record the registry, the pin and the retention you are relying on, so the dependency is reviewable rather than assumed.
Protect the repository as an artefact in its own right. The code is not reconstructable from the estate it built; it is the one input with no upstream. An independent mirror, held somewhere that survives the loss of the hosting account, is the minimum.
Rebuild from empty on a schedule, into a target that shares nothing with production. Every apply against a live estate is a diff, and diffs never exercise the from-empty path. Measure the duration while you are there, because that measurement is the only honest input to a recovery time.
Restore data into the rebuilt estate and compare it, rather than starting it. The capture’s recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0 matching the original is the standard: a service that starts proves the objects returned, and only a comparison proves the contents did.

Cross-course references

Terraform for Production Sysadmins — Part XII (State Recovery and Backup) treats the state file as the recovery-critical artefact this lesson classifies as irreplaceable state, and it is the material to read next for how that particular no-upstream item is protected and recovered.
Ansible for Production Sysadmins — Part XXXVI (Drift and Convergence) develops the drift problem this lesson only names, which matters here because a rebuild reproduces the committed intent rather than the estate, and the gap between them is discovered during recovery unless it is measured before it.
Kubernetes for Production Sysadmins — Part IV (Desired State and Reconciliation) explains the control loop that produced the measured outcome above, and reading it makes clear why an empty 64Mi volume fully satisfied the spec that the rebuild was reconciling towards.