Rebuild versus restore: the central choice

The previous lesson produced an inventory: the partition table and its duplicate, the EFI system partition, the firmware variables, the assembly metadata, the LUKS header. Each of those is something a backup has to capture deliberately because nothing walks it. That inventory answers the question of what to copy. It leaves the larger question untouched, and the larger question shapes every recovery plan in the estate: for each item on any such list, should it be copied at all, or should it be made again from the definition that produced it the first time?

Every recovery is a mixture of two operations

There are exactly two ways to make something exist again after it is gone.

The first is to rebuild: run a definition against infrastructure that holds nothing, and let the definition determine the result. A kickstart file and a package list produce an operating system. A Dockerfile and a pinned base produce an image. A module and a provider produce a network. A directory of manifests produces a namespace full of objects. The output is a function of the definition and its inputs, and it does not depend on anything that survived from the system that was lost.

The second is to restore: read a copy that was taken earlier and put the bytes back. The output is a function of the copy, and it depends on nothing except the copy being intact and reachable.

Almost no real recovery is purely one or the other, and the interesting design work is not choosing between them but deciding, item by item, which of them applies. That decision has more leverage than any tool selection in the estate, because it determines four things at once: what the backup system is obliged to hold, how large it has to be, how long a recovery takes, and how much of the original fault travels forward into the recovered system.

Most teams never make the decision explicitly. They inherit it, usually from whichever tool was adopted first, and they discover its shape during an incident when something that was assumed to be on one side of the line turns out to be on the other.

The test that sorts one side from the other

The sorting question is short. Can this be regenerated, identically enough to serve, by running a definition whose inputs still exist? Yes puts the item on the rebuild leg. No puts it on the restore leg.

Applied to a normal estate, the rebuild leg collects the operating system and its package set, the kernel and its modules, container images built from pinned bases by a pipeline, the network configuration expressed as templates or resource definitions, the cluster objects described by manifests, and the service configuration a convergence run writes. All of it is derived. None of it is unique. If it is deleted and the pipeline runs, an equivalent thing appears.

The restore leg collects databases, user-uploaded content, ledgers and financial records, queued messages that have been accepted but not processed, monitoring and audit history, and the secret material that nothing else can reissue. All of it accumulated. None of it is described anywhere. If it is deleted and the pipeline runs, an empty container of the right shape appears in its place.

Two words in the sorting question carry more weight than they look. The first is inputs. A definition alone rebuilds nothing; it needs a git host to read it from, a runner to execute it, a registry holding the base image, a package mirror, a provider API that answers, and credentials that still authenticate. An item is only genuinely on the rebuild leg if that whole chain survives the failure being planned for, which is a different claim from the definition being in version control.

The second is identically enough to serve. Where some other system holds a reference to a value the rebuild would generate afresh, the value is state even though a definition produced it. Numeric UIDs already recorded in file ownership, an SSH host key already pinned by clients, a licence bound to a machine identifier, a certificate whose fingerprint is in someone else’s allow-list: each is generated by a process, and each is unrecoverable by re-running that process. In the capture below, the same claim was satisfied by three different volumes across three lifecycles — pvc-6edd5db0-25f4-4b33-b26b-e2ad78aac9cf, then pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93, then pvc-ef5cf541-eebf-4eb3-9c91-cb4e59d6d787. Any procedure that had recorded the first identifier as the place to put data back would have written into a directory that no longer belonged to anything.

Measured: every object returned and the orders did not

The sharpest available demonstration is a cluster, because a cluster is the one place where the definition is genuinely complete and the gap is therefore impossible to blame on sloppiness. A namespace, a claim, a ConfigMap and a pod were applied to a single-node k3s cluster from a manifest of the kind that lives in a GitOps repository.

$ kubectl apply -f rbdr-shop.yaml

  namespace/rbdr-shop created
persistentvolumeclaim/rbdr-orders created
configmap/rbdr-config created
pod/rbdr-orders-writer created
>>> exit code: 0
pod Ready after 9s

The pod then did what applications do: it wrote two orders into the claim, and the resulting file carried md5 9eb4e2ad8e08e1dcaaf87ababab964b0. At that point three separate things existed. The manifest in the repository described the four objects. The API server held those objects as records. And a file on the node held bytes that appear in neither. Only the first of the three was in version control.

The namespace was then deleted, which took the claim with it and, under the provisioner in use, reclaimed the volume directory along with it. Recovery attempt one was the move every team reaches for first: re-apply the repository.

$ kubectl -n rbdr-shop get pvc,configmap,pod

--- every object is back ---
persistentvolumeclaim/rbdr-orders   Bound   pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93   64Mi   RWO   local-path   <unset>   6s
configmap/kube-root-ca.crt   1     6s
configmap/rbdr-config        1     6s
pod/rbdr-orders-writer   1/1   Running   0     6s

--- and the business data? ---
total 8
drwxrwxrwx    2 root     root          4096 Aug 28 14:34 .
drwxr-xr-x    1 root     root          4096 Aug 28 14:34 ..
cat: can't open '/data/orders.csv': No such file or directory
command terminated with exit code 1

Every object came back. The claim is Bound, the pod is 1/1 Running, the ConfigMap is present, and the whole reconstruction finished with the pod Ready after 6s. The rebuild leg worked perfectly and returned exactly what it was asked for. The orders are gone, and they are gone for a reason that has nothing to do with the quality of the repository: they were never in it.

What each leg costs

Rebuilding has three real advantages and they are underrated. The result is clean: nothing that was on the old system is on the new one unless a definition put it there, which means an intrusion that established persistence in a cron entry, a modified binary or an injected library does not survive the rebuild. The result has no drift: it is the described system rather than the described system plus four years of undocumented edits. And the definition was tested by CI, which is a stronger statement than anything a backup archive can offer, because the definition is exercised on every merge while an archive is exercised only when somebody deliberately restores it.

The costs are equally real. The first is time: a pipeline takes as long as it takes, and that duration belongs in the recovery estimate rather than being waved through as automation. The second is the dependency chain described above. Rebuilding converts a data dependency into a supply-chain dependency, and it does so at the moment when the supply chain is least likely to be healthy — the git host, the runners, the registry and the package mirrors are infrastructure too, and a failure large enough to require rebuilding is a failure large enough to have taken some of them out.

Restoring has the mirror-image profile. Its advantage is fidelity: you get what you had, including the parts nobody declared, the parts nobody remembered and the parts nobody knew about. For large state it is also the faster path by a wide margin, because moving bytes at line rate beats re-deriving them from scratch. Its cost is that fidelity is not selective. A restore faithfully reproduces whatever was wrong: the corrupted row, the misconfiguration that caused the outage, the vulnerable package version, and the attacker’s foothold if the copy was taken after the foothold existed.

That is the shape of the trade. Rebuilding gives a known-good system and cannot return the unique parts. Restoring returns the unique parts and cannot promise the system is good. Neither is a general answer, which is why the estate needs the line drawn per item rather than per tool.

Where the seam belongs in the procedure

A mixed recovery runs in an order, and the order is not arbitrary. Empty infrastructure is rebuilt first. The workload is held before it starts — zero replicas, a masked unit, a stopped service — because a workload that starts against empty state will initialise it, and an initialised-empty database is harder to argue with afterwards than a missing one. The state is then restored into the containers the rebuild created. The restored state is verified against a recorded fingerprint before anything reads it. Only then does the workload start, and the final check is that the service returns real data rather than that it reports itself healthy.

The capture followed exactly that order for its second recovery attempt. The volume contents had been archived separately into a 2560-byte tar file, the namespace was destroyed and rebuilt from the same YAML, and the archive was unpacked into the new volume directory that the rebuild had produced.

$ tar xf /tmp/rbdr-pv-backup.tar -C $NEWDIR

  ORDER-1001,4500.00
ORDER-1002,1250.00
recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - the application data is back, byte-identical

Both legs were necessary and neither was sufficient. The manifest could not produce the orders and the archive could not produce a running pod.

The seam belongs at the last point where the definition still fully describes what exists. Above that point everything is regenerated; below it everything is copied. The useful discipline is to name the seam for each service and write it down, because the act of naming it exposes the items nobody had assigned. A configuration file an operator edited by hand is the classic case: it is neither declared nor backed up, and it sits in the gap until an incident finds it. The resolution is always one of two moves — declare it, which lifts it onto the rebuild leg, or capture it, which drops it onto the restore leg. Leaving it unassigned is the only wrong answer.

What to take from this

• Re-applying the manifest recreated four objects and reported the pod Ready after 6s, while the same exec returned cat: can't open '/data/orders.csv': No such file or directory and exit code
  1. The rebuild leg succeeded and the data did not return.
• The claim came back Bound to pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93, a different volume from the pvc-6edd5db0-25f4-4b33-b26b-e2ad78aac9cf it had before. Identifiers generated by a rebuild are not stable across it.
• The cluster-state snapshot was 1208352 bytes and covered the API objects only. It is a copy of what the cluster believes, not of what its volumes hold.
• The restore leg returned md5 9eb4e2ad8e08e1dcaaf87ababab964b0, identical to the original, from a 2560-byte archive that had to be captured by a separate action against a separate object.
• Measured on Docker 29.7.2, the same seam appeared in miniature: a committed image reproduced its declared layer and presented the mounted data directory as total 8, holding nothing.
• Full recovery needed both legs. The manifest could not produce the orders and the archive could not produce a running pod, which is the general case rather than a property of this example.

Cross-course references

• Kubernetes for Production Sysadmins — Part IV (Desired State and Reconciliation) develops the controller loop that this lesson uses as its sharpest illustration, and Part XCVI (Workload Backup) covers the separate mechanism that has to exist for the volume contents, because the loop in Part IV provably cannot supply them.
• Terraform for Production Sysadmins — Part XII (State Recovery and Backup) is the same seam applied to the rebuild tool itself: the configuration is regenerable and the state file that maps it to real resources is not, so an estate that rebuilds everything still has one file on the restore leg.
• Ansible for Production Sysadmins — Part XXXVI (Drift and Convergence) explains why a converged host matches its declarations rather than the machine it replaces, which is exactly the boundary that decides whether an item on a Linux host belongs to the rebuild leg or the restore leg here.

Quiz