Backing up what is not a file - partition tables, LVM metadata, LUKS headers and package state · Linux

A file-level backup captures files. That is the whole of its job and it does it well. What it cannot capture is the structure those files were living in, because none of that structure is a file: where the partitions started, which physical volumes made up the volume group, which keyslot held the master key, which packages were installed and from which repositories, and the fact that /var/lib/mysql was a separate filesystem rather than a directory.

None of that is missed during the backup. All of it is missed during the rebuild, at the point where you have a blank disk, a restore archive, and no idea what shape to put the disk in before the archive can go back.

The inventory

Structure	Lost when	Captured by
Partition table	Disk replaced, table overwritten	`sfdisk --dump`, `sgdisk --backup`
LVM metadata	PV header damaged, VG not importable	`/etc/lvm/backup/`, `vgcfgbackup`
Software RAID config	Array not assembled on a new host	`mdadm --detail --scan`
LUKS header	Header overwritten - data unrecoverable	`cryptsetup luksHeaderBackup`
Filesystem geometry	Restore target built with wrong sector or stripe size	`xfs_info`, `tune2fs -l`
Mount topology	Restore lands one filesystem where three belong	`/etc/fstab`, `findmnt`
Package selection	Rebuilt host missing half its software	`apt-mark showmanual`, `dnf repoquery --userinstalled`
Repository and key config	Packages cannot be reinstalled at all	`/etc/apt/`, `/etc/yum.repos.d/`
Enabled unit state	Services restored but not starting at boot	`systemctl list-unit-files --state=enabled`

Every capture below writes to a file, never to a device. All of them are safe on a live production host.

Partition tables

Two formats, and you want both. sfdisk --dump produces text you can read, review in a diff, and reason about six months later. sgdisk --backup produces a binary image of the protective MBR, both GPT headers and one copy of the partition table.

Read-only / Safehuman-readable partition backup— --dump reads the table and writes nothing. The output records start sector, size, partition type GUID and partition UUID for each entry, plus the sector size the table was written against. That last line is why the dump belongs in the disaster-recovery bundle: a replacement disk with a different logical sector size cannot take this layout as-is, and the dump is where you find that out before you have restored 4 TB into it.

$ sudo sfdisk --dump /dev/sda

label: gpt
label-id: 4A1C8E2F-7B3D-4E56-9A01-2C3D4E5F6A7B
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 268435422
sector-size: 512

/dev/sda1 : start=        2048, size=     2097152, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=1B2C3D4E-..., name="EFI System Partition"
/dev/sda2 : start=     2099200, size=   266334208, type=E6D6D379-F507-44C2-A23C-238F2A3DF928, uuid=5F6A7B8C-..., name="lvm"

Illustrative output

Read-only / Safebinary GPT backup— --backup writes the protective MBR, both GPT headers and the partition table to a file. Note that man 8 sgdisk says the backup reflects the CURRENT IN-MEMORY table, so if the on-disk structures are already damaged the backup may record a repaired version rather than the damage. Take these backups while the disk is healthy, not during the incident. --verify reports problems and corrects nothing.

$ sudo sgdisk --backup=/root/dr/sda-gpt.bin /dev/sda && sudo sgdisk --verify /dev/sda

The operation has completed successfully.
No problems found. 32 free sectors (16.0 KiB) available in 1
segments, the largest of which is 32 (16.0 KiB) in size.

Illustrative output

Replaying a partition table destroys whatever is on the disk

The restore direction of both commands writes to the device and orphans every filesystem that does not line up with the new table:

sudo sfdisk /dev/sdb < /root/dr/sda.sfdisk      # DESTRUCTIVE
sudo sgdisk --load-backup=/root/dr/sda-gpt.bin /dev/sdb   # DESTRUCTIVE

Before either of them, confirm three things about the target, out loud, by serial number rather than by device name:

lsblk -o NAME,SIZE,SERIAL,WWN,MOUNTPOINT /dev/sdb
sudo blkid /dev/sdb*
sudo wipefs --no-act /dev/sdb

lsblk proves which physical disk /dev/sdb is right now. blkid shows any filesystem signatures already present - anything at all here means you are about to overwrite a disk that has data on it. wipefs --no-act lists signatures without removing them. If any of the three surprises you, stop.

LVM metadata

LVM keeps this one for you. Every change to a volume group writes a new text copy of its metadata into /etc/lvm/backup/, and the previous versions into /etc/lvm/archive/. Backing up /etc therefore captures it - as long as somebody knows that is what those files are for.

Configuration changeforce a fresh metadata backup— vgcfgbackup rewrites /etc/lvm/backup/<vg> from the current metadata. It touches no volume and no data - it writes one text file. The two UUIDs are the reason this file matters: they are how a PV whose on-disk header is damaged can be re-created with its original identity rather than as a new, empty PV.

$ sudo vgcfgbackup && head -20 /etc/lvm/backup/vg0

  Volume group "vg0" successfully backed up.
# Generated by LVM2 version 2.03.16(2)

contents = "Text Format Volume Group"
version = 1
description = "Created *after* executing 'lvextend -L +50G vg0/data'"

vg0 {
id = "kR3aZ1-9Xc2-Qw8T-mN4v-Lp6B-hY0d-Ke7Rs2"
seqno = 14
physical_volumes {
pv0 {
id = "Tz5Wq8-Nb1L-Rk3X-cV7m-Jd2P-oU9s-Ay6Fe4"
device = "/dev/sda2"
}
}
}

Illustrative output

The recovery this enables is the one worth rehearsing. A PV header damaged by a stray pvcreate, a dd to the wrong device, or a controller that wrote garbage leaves a volume group that will not activate because one of its members is unrecognisable. With the metadata file, the PV can be re-created with its original UUID and the group restored:

# 1. find the original PV UUID in the saved metadata
grep -A3 'physical_volumes' /etc/lvm/backup/vg0

# 2. recreate the PV header with that identity, on the right device
sudo pvcreate --uuid Tz5Wq8-Nb1L-Rk3X-cV7m-Jd2P-oU9s-Ay6Fe4 \
  --restorefile /etc/lvm/backup/vg0 /dev/sda2

# 3. put the volume group metadata back
sudo vgcfgrestore --list vg0
sudo vgcfgrestore -f /etc/lvm/backup/vg0 vg0
sudo vgchange -ay vg0

Steps 2 and 3 write LVM metadata to the device. They are recovery operations, not maintenance, and they belong in a runbook with the same care as a restore. vgcfgrestore --list first, always - it enumerates the available backup and archive files so you restore the version you meant rather than the most recent one, which may be the one that recorded the damage.

LUKS headers

The LUKS header holds the wrapped master key. The passphrase does not decrypt the data; it decrypts a keyslot, which yields the master key, which decrypts the data. Overwrite the header and the data is cryptographically gone - the correct passphrase is worth nothing without the keyslot it unlocks.

Configuration changeheader backup— Reads the LUKS header and keyslot area and writes it to a file. The device is not modified. Take one at LUKS creation time and again after every keyslot change, because a header backup taken before a keyslot was added does not contain that keyslot. Verify the file with 'sudo cryptsetup luksDump --header /root/dr/sda3-luks-header.img' which prints the keyslots it actually contains.

$ sudo cryptsetup luksHeaderBackup /dev/sda3 --header-backup-file /root/dr/sda3-luks-header.img

A header backup is a secret, and it outlives passphrase changes

man 8 cryptsetup-luksHeaderBackup states the consequence directly: the backup file together with a passphrase that was valid at the time of the backup decrypts the data area, even if that passphrase was later changed or removed from the device.

Two things follow, and both are usually missed:

Revoking a passphrase does not revoke it. An administrator leaves, their keyslot is removed with cryptsetup luksKillSlot, and the header backup taken last year still contains that keyslot. Anyone with the backup file and the old passphrase reads the disk. Re-take every header backup after any keyslot change and securely destroy the old ones.
You can no longer wipe the disk by wiping the header. The same man page notes that a header backup removes the ability to securely erase the device by overwriting the header and keyslots. Decommissioning now means erasing every header backup as well, or overwriting the encrypted data area.

Store header backups where you store keys, not where you store backups: encrypted, access-logged, with an inventory of which device and which date each one covers. And never store the header backup inside the volume it unlocks, or inside a backup repository whose only key is on that volume - both are the same circular dependency, and it is only discovered at restore time.

Package state, repositories and enabled units

A restored /etc and /var on a freshly installed OS is not the same host. The binaries came from packages, the packages came from repositories, and the services were enabled by symlinks that a file restore may or may not have carried.

Read-only / Safewhat was installed deliberately— apt-mark showmanual lists packages an operator asked for, excluding the hundreds pulled in as dependencies. That distinction is what makes the list usable: reinstalling 147 manual packages reproduces the host, while replaying a full dpkg --get-selections of 1400 entries reproduces the exact dependency state of a distribution version that may no longer exist. On RHEL-family hosts the equivalent is 'dnf repoquery --userinstalled --qf %{name}'.

$ apt-mark showmanual | tee /root/dr/packages-manual.txt | wc -l

Illustrative output

Package selection: apt-mark showmanual, or dnf repoquery --userinstalled. The deliberate set, not the dependency closure.
Repository configuration and signing keys: /etc/apt/sources.list, /etc/apt/sources.list.d/, /etc/apt/keyrings/ - or /etc/yum.repos.d/ and /etc/pki/rpm-gpg/. Without these the package list cannot be replayed at all.
Enabled units: systemctl list-unit-files --state=enabled --no-legend. A restored unit file that was never enabled does not start at boot, and this is the check that catches it.
Masked units too: systemctl list-unit-files --state=masked. A service deliberately masked on the old host will start on the rebuilt one unless the mask is reproduced.
Kernel command line and boot config: cat /proc/cmdline, plus /etc/default/grub. Anything set there is invisible to a filesystem restore of /.
sysctl settings: sysctl -a is the running state, /etc/sysctl.d/ is the intent. Capture the directory; capture the running state as evidence.

One script, run before every backup

The point of collecting all of this is that it must be in the backup, not in someone’s home directory. The pattern that works is a single script writing to one directory that the backup job already covers.

#!/bin/bash
set -euo pipefail
OUT=/root/dr
mkdir -p "$OUT"

# structure
lsblk -o NAME,SIZE,TYPE,FSTYPE,SERIAL,WWN,MOUNTPOINT > "$OUT/lsblk.txt"
findmnt --real -o TARGET,SOURCE,FSTYPE,OPTIONS      > "$OUT/mounts.txt"
for disk in /dev/sd? /dev/nvme?n?; do
  [ -b "$disk" ] || continue
  name=$(basename "$disk")
  sfdisk --dump "$disk" > "$OUT/parttable-$name.sfdisk" 2>/dev/null || true
done

# LVM and RAID
vgcfgbackup >/dev/null 2>&1 || true
cp -a /etc/lvm/backup "$OUT/lvm-backup" 2>/dev/null || true
mdadm --detail --scan > "$OUT/mdadm.conf" 2>/dev/null || true

# software state
apt-mark showmanual > "$OUT/packages-manual.txt" 2>/dev/null || \
  dnf repoquery --userinstalled --qf '%{name}' > "$OUT/packages-manual.txt"
systemctl list-unit-files --state=enabled --no-legend > "$OUT/units-enabled.txt"
systemctl list-unit-files --state=masked  --no-legend > "$OUT/units-masked.txt"
cat /proc/cmdline > "$OUT/cmdline.txt"

chmod -R go-rwx "$OUT"

Two deliberate omissions. The LUKS header is not in this script, because it must not travel with the ordinary backup - capture it separately, into key storage. And nothing here writes to a device, so the script is safe to run from cron on a production host.

The || true on the partition dumps is there because the loop walks device patterns that may not all exist; without it, set -e aborts the whole collection on the first absent device and you get a partial bundle that looks complete.

The inventory

Partition tables

LVM metadata

LUKS headers

Package state, repositories and enabled units

Knowledge check