Bare-metal recovery as an architecture

A block-level image reproduces a disk exactly, and that precision makes it easy to mistake the image for the rebuild. It is not the rebuild. Writing an image back is one stage in a sequence, and it sits somewhere near the middle: several things have to be true before the image can be written at all, and several more have to happen after it before anything answers a request. Rebuilding a server from nothing is a chain of dependent stages, and the stage that stops the recovery is almost never the one that gets rehearsed.

Nine stages, and the one people rehearse is the eighth

Reconstruction from nothing has a fixed shape. The names differ between estates, the order does not, because each stage consumes something the previous one produced.

Hardware, or a virtual machine of adequate size. Something has to exist to receive the rebuild: a chassis with enough disks, or hypervisor capacity with enough CPU, memory and storage to run the workload rather than merely to boot it.

A boot environment that can reach the network and the backup. A machine that will not boot to anything you control is not a recovery target. This stage is a small operating system — installer, live image, PXE payload, rescue partition — carrying the drivers for that hardware’s storage controller and network interface, and reachable through a console you can open without the estate.

A storage layout matching what the restore expects. Partition table, volume group, logical volumes, filesystems, labels and UUIDs, mount points. Restores fail here quietly: data lands on a layout that boots and does not match the paths anything else was written against.

The operating system. Installed at the release the system was running, not at whatever is current, because the configuration and the application about to be applied were written against the former.

Networking: addressing, routing and name resolution. Addresses on the right interfaces and VLANs, a default route, and resolvers that answer. Everything after this stage crosses a network, so an error here presents as a failure of whichever later stage happens to run first.

Configuration. The /etc of the system: unit files, application configuration, firewall policy, scheduled work, identity material, security labels. Configuration decides where data lives and who may read it, which is why it precedes the data.

The application. Package or image at the version that was running, with its dependencies at their versions, plus licence material if the application refuses to run in full without it.

The data. The restore command. This is the stage people rehearse, the stage the tooling documents, and the stage most likely to succeed.

Validation. A request that does work — reads restored data and returns it — rather than a port check or a green unit state.

Two of those are worth separating deliberately, because capture tooling routinely conflates them. The application stage and the data stage cover different trees, and a capture aimed at one silently omits the other. Measured on Docker 29.7.2, a container’s state was captured with docker commit and a new container started from the committed image: the writable layer came back, and the directory that had held the application’s data listed nothing but . and ... The volume was captured by a second, separate mechanism — a helper container writing a tar archive — and it restored byte-identical at md5 9eb4e2ad8e08e1dcaaf87ababab964b0 on both sides. On that restored service, the layer file the commit had held returned cat: can't open '/etc/app-marker': No such file or directory, because the volume archive was the only capture taken. Two stages, two mechanisms, and whichever one the estate skips is the half it discovers missing during the rebuild.

Each stage has a precondition, and the precondition is what is missing

Listing the stages is the easy half. The half that decides whether a recovery completes is the artefact each stage consumes, because that artefact was produced at a calmer time on a system that no longer exists.

Stage one needs a capacity record: what the machine actually was, in disks, cores and memory, current at the last change rather than at the last audit. A rebuild onto a smaller target discovers the gap after the restore has been running for an hour.

Stage two needs the recovery media itself, and a path to it. An image on a file share inside the estate is not recovery media during a site loss; nor is a PXE server whose boot files live on the storage being rebuilt. It also needs out-of-band console access with credentials that are not in the password manager you are recovering.

Stage three needs the recorded layout. This is a backup artefact and it is routinely not backed up, because it is not a file anyone edits. Capture it on the schedule that captures everything else:

HOST=$(uname -n)
OUT=/var/backups/layout
install -d -m 0700 "$OUT"
lsblk -o NAME,SIZE,TYPE,FSTYPE,UUID,MOUNTPOINT > "$OUT/$HOST-lsblk.txt"
for DISK in /dev/sda /dev/sdb; do
  test -b "$DISK" || continue
  sfdisk --dump "$DISK" > "$OUT/$HOST-$(basename "$DISK").sfdisk"
done
cp -a /etc/fstab "$OUT/$HOST-fstab"
cp -a /etc/lvm/backup "$OUT/$HOST-lvm-metadata"

The dump sfdisk writes is the input sfdisk reads, which makes the partition table a reproducible artefact rather than an act of recall. The same reasoning applies to the LVM metadata that LVM itself archives under /etc/lvm/backup on a default install, and to the fstab that names the UUIDs; all of it belongs in the repository, not on the machine it describes.

Stage four needs a package repository or mirror that answers, and a record of the release. An estate whose mirror is an internal host has made the operating system stage depend on the estate.

Stage five needs the address plan — which address, which VLAN, which gateway, which resolvers — as data rather than as something that used to arrive from DHCP served by a machine that is also gone. In a recovery environment it is applied by hand, which means it has to be written somewhere legible:

IFACE=enp1s0
ADDR=10.20.4.11/24
GW=10.20.4.1
RESOLVER=10.20.1.53
ip link set "$IFACE" up
ip addr add "$ADDR" dev "$IFACE"
ip route add default via "$GW"
printf 'nameserver %s\n' "$RESOLVER" > /etc/resolv.conf

Stage six needs the configuration source and a credential for it. Stage seven needs the artefact registry and a version manifest. Stage eight needs two things that are usually counted as one: a credential that reaches the repository, and the key that decrypts what is in it. Stage nine needs a written statement of what correct output looks like, produced before the incident, and a checker that does not run on the estate being validated.

Measured: an intact 11 MiB repository that opened for nobody

The precondition failure is worth seeing rather than asserting, because it does not resemble the failures backup tooling is built to report. A restic repository was created with the passphrase held in a file inside the directory being backed up — the arrangement that works flawlessly in every test run while the estate is healthy. The backup completed. Then the site was destroyed, taking the production directory and the passphrase with it, and leaving the repository untouched.

$ restic snapshots

--- the repository is completely intact ---
11M	/work/repo
files in repository: 6

--- attempting a restore with no passphrase ---
$ restic snapshots
Fatal: wrong password or no key found
>>> exit code: 12

Nothing in that output is a storage failure. Eleven mebibytes, six files, every byte present and readable — this is neither a corruption failure nor a media failure. The data survived the disaster and the ability to read it did not. The repository is intact and permanently closed, and the stage that failed is stage eight, with stages one through seven completed, on hardware that exists, on a network that works.

The natural assumption is that the key was left behind. It was not.

$ ls /work/repo/keys/

  d5f39ef5517fab2e4d1e6dc1d9d5b4b1f1ca4eeacecde523af8b5a0c3c3d120e

The recursive dependency: recovering with tools that live inside the failure

The encryption case is one instance of a shape that recurs at every stage, and it is the shape that turns a plausible recovery plan into one that cannot execute. The tooling required to perform the recovery is frequently hosted on the infrastructure being recovered.

Written out, the list is uncomfortable. The runbook is in a wiki that authenticates against the identity provider in the affected estate. The configuration management controller that would rebuild the hosts is itself one of the hosts. The package mirror is an internal VM. The container registry holding every application image runs on the cluster being restored. The DNS name of the backup repository is served by resolvers inside the estate, so recovery cannot find its own backups even though the backups are fine. The credential for the object store is in a secret manager that is on the restore list. The monitoring that would tell you a stage succeeded is downstream of the stages it is meant to confirm.

Every one of those arrives through ordinary good engineering. Centralising the runbooks is right. Moving a hard-coded credential into a secret manager is right. Running the mirror internally is faster and cheaper. Each decision is sound in isolation and each one adds an edge from the estate into its own recovery path, and the edges are invisible while everything is running because every endpoint answers.

An independent recovery path is a design, not a spare copy

Breaking these dependencies means supplying the precondition from somewhere no member of the estate touches. For key material, restic supports this directly: a repository can hold more than one key, and a second passphrase — held by a different team, in a different system — opens the same repository without the first.

$ restic key add --new-password-file /work/recovery-pass

  saved new key with ID 66c34166d8443d16e8c5899fe3f792e749cb24fa3a90ac90bbe8348979b57d90
>>> exit code: 0

$ restic key list
 ID        User  Host          Created
--------------------------------------------------
 66c34166  root  17dffded9807  2026-08-28 14:04:55
*4bc6f61a  root  17dffded9807  2026-08-28 14:04:52
--------------------------------------------------

Two keys, one repository. Neither passphrase can derive the other, and both decrypt the same master key. Run the same disaster again — production destroyed, production passphrase gone with it — and the outcome inverts:

$ restic --password-file /work/recovery-pass restore latest --target /work/rec

  restoring snapshot b96ba7cf of [/work/prod2] at 2026-08-28 14:04:52.481565631 +0000 UTC by root@17dffded9807 to /work/rec
Summary: Restored 3 files/dirs (38 B) in 0:00
>>> exit code: 0

recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical, using a passphrase production never held

The second mechanism in the same capture is key export. borg key export --paper produces a printable block described in its own output as BORG PAPER KEY v1, which is the encrypted key material in a form that survives the loss of every system that held the repository. It is still useless without the passphrase, so the two must be escrowed separately and to different custodians — which is the general rule for every artefact in this lesson, not a quirk of that command.

The same treatment applies stage by stage. Recovery media on physical media, or in a second cloud account, or at a colocation the estate cannot reach. A copy of the runbook printed, or in a document store in a different trust domain, with the address plan and the layout dumps in it. The package mirror mirrored somewhere independent, or a record of which upstream release to install from the public internet. Console credentials in a sealed envelope with a named custodian. None of that is exotic; all of it is boring, and boring is what makes it get skipped.

The check that this has been done is not a restore test. It is a restore test with an assumption attached: assume the primary site is gone. Under that assumption, walk the stages and name, for each precondition, the surviving system it is read from. The preconditions with no answer are the recovery plan’s real content. Run the walk from the recovery environment rather than from a desk, and make it produce a file rather than a feeling:

RECOVERY_CHECKS=/run/recovery/preflight.txt
install -d -m 0700 /run/recovery
: > "$RECOVERY_CHECKS"
for URL in \
  "https://mirror.dr.example.net/debian/dists/stable/Release" \
  "https://repo.dr.example.net/estate/config" \
  "https://runbooks.dr.example.net/bare-metal.pdf"
do
  if curl -fsS --max-time 10 -o /dev/null "$URL"; then
    printf 'reachable    %s\n' "$URL" >> "$RECOVERY_CHECKS"
  else
    printf 'UNREACHABLE  %s\n' "$URL" >> "$RECOVERY_CHECKS"
  fi
done
test -s /run/recovery/repo.pass \
  || printf 'no repository passphrase present in the recovery environment\n' >&2

Production discipline

1. Write the rebuild as an ordered list of stages with named preconditions. Hardware, boot environment, storage layout, operating system, networking, configuration, application, data, validation — and beside each stage, the artefact it consumes and the surviving system that artefact is read from.
2. Back up the layout, not only the contents. The sfdisk --dump output, the LVM metadata, the fstab UUIDs and the address plan are recovery inputs; they are small, nobody edits them, and they are consequently the artefacts most often absent when stage three runs.
3. Escrow the key separately from the repository. Measured on restic 0.19.1, an intact 11 MiB repository holding six files returned Fatal: wrong password or no key found at exit code 12, and restic key add produced a second key ID 66c34166 that opened the same repository after the first was destroyed.
4. Enumerate every recovery tool that runs inside the estate. The runbook, the configuration controller, the mirror, the registry, the resolver, the secret manager and the monitoring are candidates by default; each one that is confirmed needs an independent copy or an independent path.
5. Rehearse with the assumption that the primary site is gone. A restore that begins on a provisioned, addressed, configured host proves stage eight and inherits the other eight, which is why it has never failed.

Cross-course references

• Linux for Production Sysadmins — Part IX (Boot Process) covers what has to be in place for a machine to reach a usable environment, and Part LXX (Out-of-Band Management) covers the console path that stage two of this sequence depends on; together they are the mechanics behind the claim here that a boot environment is a recovery artefact with preconditions of its own, not a step you improvise on the night.
• Ansible for Production Sysadmins — Part L (Automation Disaster Recovery) is the recursive dependency described above applied to the tool that would perform stage six: a controller that rebuilds the estate while being one of the hosts in it, and what an independently recoverable controller looks like.
• Proxmox VE for Production Operators — Part XIV (Disaster Recovery) and Part IX (Virtual Machines) supply stage one when the recovery target is a virtual machine rather than a chassis, which changes the lead time on hardware but leaves every later stage and every precondition in this lesson unchanged.

Quiz