Restoring onto clean infrastructure to expose hidden dependencies

The maturity progression ends on a rung nobody can fake: a recovery performed onto infrastructure that has never run the service. That rung is not there for ceremony. Every rung below it shares one weakness, which is that the target of the restore is also a witness for the restore, and a target with history testifies in favour of the backup regardless of what the backup actually contains. Changing the target changes what the exercise is able to discover, and of all the variables in a restore test it is the only one that does.

A restore only measures what the target cannot supply

Restore onto the machine that has been serving traffic all week and count what the backup is forced to produce. Files, into a directory tree that already exists, owned by an account that already exists, read by a process whose unit file is already enabled, presenting a certificate whose issuer is already trusted, reachable through a name that already resolves, on a filesystem already mounted at the path the configuration names.

Everything in that sentence except the word “files” is a constant during the exercise, and a constant cannot be measured. That is worse than uninformative, because the bias runs one way: an exercise holding nine dependencies fixed and varying one reports success far more readily than the recovery it stands in for, and nothing in the result says how much the host underwrote.

The bias lands hardest on the number the exercise is usually run to produce. A same-host restore is timed, and the elapsed time becomes the RTO. But the stages the surviving host performed for free — provisioning, installing the package set at the versions that were running, creating accounts, placing unit files, trusting an issuer, publishing a name — took zero seconds because they had already happened. The recorded duration is one stage of a recovery, recorded as though it were all of it.

Eight classes of dependency, and how each one fails

The list below is not exotic. Every item is ordinary, invisible on a host with history, and has stopped a real recovery.

Packages and their versions. The clean host installs whatever the repository offers today, which is a different system from the one that was running. Config formats change between versions, a schema migrates forward on first start, a default flips. Reconstructing that version list from memory is where recovery most often stalls, which is why the list has to be a backup artefact.

Numeric user and group ids. A package installed on a fresh host takes whatever number is free when the account is created, which is not the number the archive recorded. Files come back owned by an integer that maps to nothing, and the failure surfaces as an application permission error rather than a restore error.

Unit files and their ordering. The unit file is often outside the data backup, in a configuration repository or a package. Its ordering directives are the subtler half: After= states an order and not a requirement, and systemd.unit(5) is explicit that ordering is orthogonal to the requirement dependencies configured by Requires=. A unit whose ordering names something the clean host lacks starts anyway, earlier than intended.

Certificates and trust stores. Two directions, failing separately. The restored service presents a certificate, so inbound clients that verify are satisfied. Its outbound calls fail, because the clean host has a stock CA bundle and does not trust the internal issuer that signs everything else.

Name resolution. The service name still resolves to an address that no longer hosts anything, and cached negative answers extend the outage past the moment the service is ready. On the original host this class does not exist, because the record already points where it needs to.

Mounted paths. A path that was a mount point on the original host is an ordinary directory on the clean one. Writes succeed — into the root filesystem — and the failure is a capacity alert days later, not an error now.

Licence and identity material. Node-locked licences keyed to a hostname, MAC address or machine identifier that changed during the rebuild; SSH host keys, without which automation refuses to proceed; service credentials, API tokens and enrolment secrets that make the rebuilt host the same principal to the broker, the object store and the database. Some of it is not regenerable without a human approving a fresh issuance.

Kernel modules and filesystem features. The restored volume was created with features the clean host’s kernel or userspace does not implement, or its filesystem needs a module the minimal image lacks. This is one of the few classes that fails loudly, which makes it the most cited and the least dangerous.

Measured: every object back, Ready after 6s, and an empty directory

The cleanest instance of the argument is a Kubernetes namespace, because Kubernetes separates the two halves further than anything else: the desired state is complete, declarative and in Git, and the application data is entirely outside it. A capture on k3s v1.36.3+k3s1 created a namespace, a claim, a ConfigMap and a Pod, let the application write two orders into the volume, deleted the namespace, then did what teams mean when they say everything is in Git.

$ kubectl apply -f rbdr-shop.yaml

  namespace/rbdr-shop created
persistentvolumeclaim/rbdr-orders created
configmap/rbdr-config created
pod/rbdr-orders-writer created
>>> exit code: 0
pod Ready after 6s

--- every object is back ---
persistentvolumeclaim/rbdr-orders   Bound   pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93   64Mi   RWO   local-path   <unset>   6s
configmap/kube-root-ca.crt   1     6s
configmap/rbdr-config        1     6s
pod/rbdr-orders-writer   1/1   Running   0     6s

--- and the business data? ---
total 8
drwxrwxrwx    2 root     root          4096 Aug 28 14:34 .
drwxr-xr-x    1 root     root          4096 Aug 28 14:34 ..
cat: can't open '/data/orders.csv': No such file or directory
command terminated with exit code 1

Read the two halves against each other. Every object was recreated and the apply exited 0. The claim reached Bound against a volume identity — pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93 — the cluster minted on the spot, and the Pod reported 1/1 Running, Ready after 6s. A reconciliation controller would have called the namespace synced, and a runbook step reading “confirm all Pods Running and all claims Bound” would have been ticked.

The directory the business cares about listed total 8 and two entries, . and .., and reading the file returned No such file or directory with exit code 1. No platform signal distinguished this state from a correct one, because from the platform’s point of view it was correct: every object it was asked to create exists and is healthy. The clean target did not cause the gap. It made the gap the only thing left in the room.

What the archive itself declined to carry

One class sits underneath the others, because it defeats a restore that got every dependency right: the metadata the archive was never told to record. The same evidence set captured a source tree carrying an ACL on app.conf, a user xattr of sha256:deadbeef on index.dat, the capability cap_net_raw=ep on netcheck, a setuid bit, a 200M sparse image allocating nothing, and a hard-linked pair sharing one inode — then archived it with default flags.

$ tar -xf naive.tar -C r1

  archive size: 201M
[restored from default tar]
  ACL on app.conf        : 0 entr(y|ies)
  xattr on index.dat     : ABSENT

  capability on netcheck : ABSENT
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 200M
  payload.a link count   : 2
  payload a/b same inode : yes

Every file has the right name, size and contents; a checksum comparison of content would have passed. What did not survive is the metadata that decides whether the restored system works. The ACL granting a second account read access came back as 0 entr(y|ies). The capability that let an unprivileged binary open a raw socket is ABSENT, so the binary is present, executable and correctly moded and still cannot do its job. The corrected capture in the same run, with --acls --xattrs --xattrs-include='*' --sparse, returned all of it.

The sparse file carries a second lesson, belonging to the RTO rather than to correctness. The source allocated nothing; the default archive was 201M and restored 200M of real blocks, where the corrected archive was 100K. On the original host, where the file already existed in sparse form, that difference was invisible. On a clean target it is storage to provision and bytes to move.

The discovery log is the deliverable

The restored service is not the output of this exercise. The list of things the clean host turned out to need is. Run it with somebody whose only job is to write a row every time the work stops and a human supplies something: what was missing, where it came from, how long the stop lasted, and who knew. Rows are cheap during the exercise and unrecoverable afterwards, because by the next morning everyone remembers that it went fine.

Each row is then used twice. First as an RTO stage: the same-host estimate was one number, the log is a list, and the honest figure for a stated architecture — this service, this repository, this network path — is the sum of the rows, larger than the same-host number by exactly the work that host had been doing for free. Second as an inventory row: item, source of truth, owner, date last proven. A row with no source of truth is the finding; a row with a source but no owner is the second.

Most of the log can be pre-empted by capturing the machine’s description of itself alongside its data, so the next exercise starts from a diff:

MANIFEST=/backup/host-manifest
install -d -m 0750 "$MANIFEST"
dpkg-query -W -f='${binary:Package} ${Version}\n' > "$MANIFEST/packages"
getent passwd > "$MANIFEST/passwd"
getent group > "$MANIFEST/group"
systemctl list-unit-files --state=enabled > "$MANIFEST/units"
cp /proc/self/mounts "$MANIFEST/mounts"
lsmod > "$MANIFEST/modules"

On the clean target the same facts are gathered again and compared, and every difference that required intervention is recorded with the minutes it cost:

MANIFEST=/backup/host-manifest
LOG=/var/log/restore-exercise.log

note() {
  printf '%s\tstage=%s\tminutes=%s\t%s\n' "$(date -Is)" "$1" "$2" "$3" >> "$LOG"
}

getent passwd > /tmp/passwd.restored
if ! diff -q "$MANIFEST/passwd" /tmp/passwd.restored > /dev/null; then
  note uid-reconciliation 12 'local account numbering differs from the manifest'
fi

if ! grep -q ' /srv/orders ' /proc/self/mounts; then
  note missing-mount 8 '/srv/orders is a plain directory on the clean host'
fi

The final step is a read, not a status check. Ask the restored service for a record that existed before the loss, through the address real clients use, and compare it against an independently recorded value. That one request exercises the name, the certificate, the trust store, the account permissions on the data directory and the restored bytes — five of the eight classes at once. In the Kubernetes capture the equivalent check was a cat, and it returned exit code 1 against a cluster reporting itself entirely healthy.

Production discipline

1. Choose the target for what it cannot supply. A host with history lends the restore its packages, account numbering, unit files, trust store, resolver and mounts, and every borrowed item is a measurement the exercise silently declines to take.
2. Appoint a scribe before the exercise starts. The deliverable is the list of things the clean host turned out to need, with the minutes each cost and the person who knew; none of it survives to the following morning.
3. Promote every discovered item to an RTO stage and an inventory row. The same-host figure is a floor, and the honest number for a stated architecture is the sum of the log — item, source of truth, owner, last proven.
4. Tell the archive to carry metadata, and verify it did. Measured on GNU tar 1.35, the default archive returned 0 entr(y|ies) of ACL and an ABSENT capability, and re-materialised a sparse image as 201M where the corrected capture was 100K.
5. Close with a read of pre-loss data, never a readiness condition. On k3s v1.36.3+k3s1 the claim was Bound and the Pod Ready after 6s over a directory holding only . and ...

Cross-course references

• Linux for Production Sysadmins — Part IV (Users, Groups and Identity) explains how a fresh installation allocates numeric UIDs independently of any other host, and Part VII (systemd and Service Management) covers the ordering and requirement directives whose asymmetry lets a unit start against absent dependencies; those are the first and third dependency classes enumerated above, and neither is exercised by a restore onto a host that already has both.
• Kubernetes for Production Sysadmins — Part IV (Desired State and Reconciliation) describes the loop that recreated every object in six seconds, and Part L (PersistentVolumes and Claims) covers the binding that produced a fresh volume identity behind an unchanged claim name; together they explain why the measured cluster could be completely healthy and completely wrong, which is the archetype this lesson generalises from.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part VI (Chains and Trust Stores) covers the fourth dependency class: a rebuilt host satisfies inbound peers with a restored certificate and still fails its own outbound calls to an internal issuer it was never told about, which cannot appear on a target where that issuer was already installed.

Quiz