Boot, partitioning and the parts nobody backs up · Backup & DR

The bare-metal rebuild in the previous lesson was a chain of dependent stages, each waiting on a precondition, and the storage-layout stage hides a precondition a file-level backup does not supply: that the replacement can be made to look, to the firmware and to the kernel, like the machine that was lost. A file tree does not carry that. tar, rsync and every repository built on top of them capture files, and most of what turns a set of files into a bootable system is not a file. Some of it lives on the disk but outside every filesystem, some lives on a filesystem nobody thought to include, and one part of it is not on the disk at all. This lesson is the inventory, and the deliberate capture of each item.

Three places boot state hides

The first place is on the disk, outside every filesystem. A GPT-labelled disk begins with a protective MBR at LBA 0, a primary GPT header at LBA 1 and the partition entry array immediately after it; a duplicate header and array sit at the far end of the device. On a BIOS-booted machine the gap between the partition table and the first partition holds the bootloader’s second stage. An encrypted container carries its LUKS header in the first sectors of the partition. LVM writes metadata onto every physical volume, and mdadm writes a superblock onto every member device. None of these has a path, so a backup tool that walks mount points cannot see any of them.

The second place is on a filesystem that is easy to leave out of scope. The EFI system partition is an ordinary FAT filesystem holding loader binaries, usually mounted at something like /boot/efi. If the backup selection was written as the list of directories that matter to the business, the ESP is missing, and with it everything the firmware needs to start the machine.

The third place is not on any disk. UEFI firmware keeps its boot menu in non-volatile memory on the mainboard: entries that name a device and a loader path, plus the order in which they are tried. efibootmgr reaches them through the kernel’s access to EFI non-volatile variables under /sys/firmware/efi/efivars, which is documented in its manual page, but reading them through Linux does not put them on the disk. Replace the board, clear the NVRAM, or move a restored disk into different hardware, and those entries are gone while every byte on the disk is intact.

Between the categories sits the initramfs, the one that catches careful people out. It is a file under /boot, so a file-level backup does capture it — but it is a built artefact encoding the layout it was built against, and restoring it verbatim onto a machine assembled differently restores a description of a machine that no longer exists.

Two copies of the partition table, and one command that captures both

GPT’s duplicate header is genuine redundancy, and it is worth being precise about what it protects against: damage to the primary copy, such as a bad sector, a stray write to the front of the device, or a firmware utility that decides the disk is unformatted. It does not protect against anything this course cares about, because both copies live on the same disk. When the disk fails, the layout description fails with it.

The capture is one command and produces a text file.

set -euo pipefail

DISK=/dev/sda
DEST=/var/backups/layout

mkdir -p "$DEST"
sfdisk --dump "$DISK" > "$DEST/partition-table.sfdisk"

The dump format is documented as suitable for later sfdisk input, restored with sfdisk /dev/sda < sda.dump, and the two properties that make it useful in a runbook are that it is text — diffable, reviewable, small enough to commit beside the host’s configuration — and that it is complete. The manual page is explicit that sfdisk “completely restores partition types and partition UUIDs”. That is exactly what you want during a rebuild, because an fstab or a bootloader configuration written in terms of PARTUUID= keeps resolving to the same partitions afterwards.

It is also the sharp edge. The same page warns that restoring one dump onto several disks in the same system produces duplicate UUIDs, and a system with two partitions claiming one identity resolves that ambiguity however the kernel happens to enumerate the devices that boot.

The boot entries that are not on any disk

A UEFI machine does not scan its disks for something bootable. It consults its own variables, each naming a device path and a loader file within an EFI system partition, and tries them in the recorded order. Those variables are firmware state. A disk image, however faithful, contains none of them.

This produces one of the more confusing rebuild outcomes: a machine with a perfectly restored disk, a valid ESP, a working loader binary and no way to reach any of it. Two things get you out. Most firmware falls back to the removable-media path — \EFI\BOOT\BOOTX64.EFI on x86-64 — when no configured entry resolves, so a copy of the loader placed there boots without any firmware variable. And the entry can be recreated from a rescue environment: efibootmgr documents -c to create an entry, -d for the disk, -p for the partition holding the loader, -L for the label and -l for the loader path, so efibootmgr -c -d /dev/sda -p 1 -L debian -l '\EFI\debian\grub.efi' is a complete entry on one line.

That is a two-minute job with the old entry written down and guesswork without it, so the output of efibootmgr -v belongs in the layout document below.

Why a LUKS header is not metadata about the volume — it is the volume

The instinct that a header is descriptive — a label, something a tool could reconstruct from the data it describes — is what makes this failure mode so expensive. A LUKS header is not descriptive. It is constitutive.

When a container is formatted, cryptsetup generates a volume key at random. That key, and only that key, encrypts the data area. It is never derived from your passphrase and it is never stored in the clear. Each keyslot holds a copy of the volume key wrapped under a key derived from one passphrase, and the parameters of that derivation — the salt above all — are recorded in the header next to the slot.

Follow that through. The passphrase alone derives nothing, because the salt it must be combined with exists only in the header, and the wrapped volume key exists only in the keyslot area. The data area holds ciphertext and nothing else: no copy of the key, no structure, no recoverable fragment. Lose the header and a correct passphrase, typed correctly, on the correct hardware, unlocks nothing — the container is indistinguishable from a partition of random data. There is no partial recovery and no salvage of the files further along the device.

The region at stake is small and sits at the front of the partition, which is precisely where accidents land: an installer told to use the whole disk, a dd aimed one letter wrong, a vendor tool that “initialises” a device.

Two consequences follow that people usually meet in the wrong order. First, cryptsetup luksHeaderBackup stores, in the manual page’s words, “a binary backup of the LUKS header and keyslot area” — a few kilobytes to a few megabytes whose loss is indistinguishable from losing the whole disk, and whose possession is worth as much as the disk. Second, and stated as a warning on the same page: “This backup file and a passphrase valid at the time of backup allows decryption of the LUKS data area, even if the passphrase was later changed or removed from the LUKS device.” A header backup is a credential with its own lifecycle. Revoking a passphrase on the live volume does not revoke it in the backups.

The capture is short, and the two lines around it matter as much as the command: cryptsetup luksDump records the header and keyslot layout in readable form, and a checksum lets a restore tell whether the file it is about to write is intact.

set -euo pipefail

DEV=/dev/sda3
DEST=/mnt/escrow/luks-sda3

cryptsetup luksDump "$DEV" > "$DEST.txt"
cryptsetup luksHeaderBackup "$DEV" --header-backup-file "$DEST.img"
chmod 600 "$DEST.img"
sha256sum "$DEST.img" > "$DEST.img.sha256"

Restoring is the mirror image, cryptsetup luksHeaderRestore with the same --header-backup-file, and it is the first step of any recovery on an encrypted host, because nothing else about the volume can be attempted until it succeeds.

What tells the kernel that these devices are one volume

LVM and Linux software RAID are both self-describing by design. LVM writes metadata onto every physical volume in a group; mdadm writes a superblock onto every member device, recording the array identity and each device’s role. That is why a set of disks moved to a new machine assembles without a configuration file: the description travels with the devices.

It is an excellent property, and it holds right up to the moment the devices are gone. The LVM capture used earlier in this course ends by destroying the single physical volume behind the group and asking what survived.

Data-loss riskthe volume group before and after its only physical volume was destroyed— EXECUTION VALIDATED on LVM 2.03.31(2) (2025-02-27), library 1.02.205, driver 4.50.0, Debian GNU/Linux 13 (trixie), loop-backed VG rbdrvg. Step 5 of the capture: the PV listing, the simulated permanent loss of the backing store, and the same reporting commands afterwards.

$ pvs; vgs; lvs

pvs before:
PV         VG     PSize   
/dev/loop5 rbdrvg 1020.00m

Simulating permanent loss of the underlying device:
re-attached the same (now destroyed) backing store as /dev/loop5

--- what survived? ---
vgs:
lvs:

Nothing under vgs: and nothing under lvs:. The group had one physical volume, the metadata describing it lived there, and both went at once. Read that as a statement about where descriptions are stored rather than about LVM: the same holds for a RAID array whose member superblocks are all on the failed shelf, and for a partition table whose two copies are both on the failed disk.

Rebuilding requires knowing what was there — the physical volumes and their sizes, the volume groups, the logical volumes with their names and extents, the RAID level and device order. With pvs, vgs, lvs and the array details recorded as text, that is arithmetic; without them it is archaeology under time pressure. Check, too, whether your distribution keeps LVM’s own metadata archives somewhere under /etc, and if it does, confirm /etc is inside the backup set rather than assuming it.

`UUID=` is a promise the restore has to keep

Modern systems refer to filesystems by identity rather than device name, which is what lets them survive a disk being renumbered. /etc/fstab mounts by UUID=, /etc/crypttab names containers the same way, the kernel command line usually carries root=UUID=, and the initramfs was built to look for exactly those identities.

Every one of those references is a promise about a value a restore can silently break. Creating a filesystem generates a fresh UUID, so a rebuild that partitions the disk, runs mkfs and copies the files back produces a machine whose fstab names filesystems that do not exist on it. The failure arrives at the worst point in the sequence: the restore reports success, and the machine then stops in the initramfs unable to find its root device, or reaches an emergency shell after waiting for a device that will never appear.

There are two honest ways through, and both require having recorded the identities beforehand. Either recreate each filesystem with its original UUID — most Linux filesystems allow the UUID to be set rather than only generated — or accept the new ones and update fstab, crypttab and the bootloader configuration to match, then rebuild the initramfs so the boot-time view agrees with the on-disk one. blkid output is the record that makes either possible.

The initramfs is also a place secrets hide. On encrypted systems it may embed a keyfile so the root volume unlocks without a prompt, alongside the module set the machine needs to see its own storage — which makes an archive of /boot sensitive in a way its contents do not advertise.

The layout document and the rescue image that has to read it

Everything above becomes usable when it is one artefact, produced on a schedule and stored with the backups rather than on the host.

set -euo pipefail

DISK=/dev/sda
DEST="/var/backups/layout-$(hostname -s)-$(date +%F)"
mkdir -p "$DEST"

sfdisk --dump "$DISK"                         > "$DEST/partition-table.sfdisk"
lsblk -o NAME,SIZE,TYPE,FSTYPE,UUID,MOUNTPOINT > "$DEST/lsblk.txt"
blkid                                          > "$DEST/blkid.txt"
cat /proc/mdstat                               > "$DEST/mdstat.txt"
pvs                                            > "$DEST/pvs.txt"
vgs                                            > "$DEST/vgs.txt"
lvs                                            > "$DEST/lvs.txt"
cp /etc/fstab /etc/crypttab "$DEST/" 2>/dev/null || true

if [ -d /sys/firmware/efi ]; then
  efibootmgr -v > "$DEST/efi-boot-entries.txt"
fi

The if is not defensive padding. The presence of /sys/firmware/efi is how the host tells you whether it booted through UEFI, and a rebuild that gets that wrong produces a machine partitioned for one firmware mode and booted in the other.

A document is a hypothesis until something has read it back. The test is a drill on disposable hardware: boot the rescue image you actually keep, restore the partition table, restore the LUKS header, open the container, activate the volume groups, restore the files, fix up the identities, reinstall the bootloader, recreate the firmware entry — then reboot with the rescue media removed.

set -euo pipefail

DISK=/dev/vda
LAYOUT=/mnt/recovery/layout

sfdisk "$DISK" < "$LAYOUT/partition-table.sfdisk"
cryptsetup luksHeaderRestore "${DISK}3" \
  --header-backup-file "$LAYOUT/luks-sda3.img"
cryptsetup open "${DISK}3" cryptroot
vgchange -ay

That drill answers a question no document can: whether the rescue environment contains cryptsetup, the LVM and RAID tooling, and the kernel modules for the storage controller in the replacement machine. A rescue image that cannot open the container is discovered either on a Tuesday afternoon or during the outage, and choosing between those two is the point of testing.

Two habits that make a rebuild impossible

The first is “just rsync it” — treating a mirror of the file tree as a recoverable machine. It is a real backup of the data and not a reconstruction of the host: the partition table, the ESP contents, the firmware entries, the LVM and RAID metadata and the LUKS header are all outside its scope. The mirror restores your files onto a machine somebody else made bootable.

The second is storage placement. Never store the LUKS header backup on the machine it protects, and never inside the encrypted volume it unlocks — the first shares the fate of the disk, and the second is a circular dependency that looks like a backup right until it is needed. The header backup is a credential: it belongs in the same escrow arrangement as the passphrase, with its own access record.

Production discipline

Dump the partition table on every layout change and keep the dump in version control. sfdisk --dump produces text that restores with sfdisk /dev/sda < sda.dump and, per its manual page, “completely restores partition types and partition UUIDs” — which is why the same dump must never be written to two disks in one system.
Back up the LUKS header the day the container is created, and store it where the container is not. The manual page warns that the backup file plus a passphrase valid at backup time decrypts the data area “even if the passphrase was later changed or removed”, so treat every header backup as a live credential and record where each copy lives.
Record the assembled layout as text rather than as knowledge. In the LVM capture, vgs and lvs both returned nothing once the single physical volume was destroyed; pvs, vgs, lvs, blkid, /proc/mdstat and efibootmgr -v collected on a schedule are the difference between arithmetic and archaeology.
Treat firmware boot entries as hardware state, not disk state. They live in NVRAM on the mainboard, efibootmgr reaches them through /sys/firmware/efi/efivars, and a board swap loses them while the disk stays intact — so keep the printed entries and put a loader at the removable-media fallback path.
Prove the rescue image can assemble the layout before you need it. Boot it, restore the table, restore the header, open the container, activate the groups and reboot with the media removed; a rescue environment missing cryptsetup or a storage driver is a discovery to make on a Tuesday.

Cross-course references

Linux for Production Sysadmins — Part IX (Boot Process) establishes the firmware-to-initramfs-to-init sequence and the roles of the EFI system partition, the bootloader stages and the initramfs; this lesson takes that sequence apart from the recovery side, asking which of its inputs a backup actually captured.
Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XV (KMS, HSM and Key Protection) covers the escrow, custody and rotation machinery that a LUKS header backup needs, because the header is a credential rather than a configuration file: possession of it plus a passphrase valid at backup time decrypts the volume.
Proxmox VE for Production Operators — Part IX (Virtual Machines) covers how a guest’s firmware state is provisioned by the hypervisor rather than living inside the guest’s filesystems, which is the virtualised form of the NVRAM problem here: a guest restored from a file-level backup needs that state supplied from the platform side.