What actually has to be backed up on a Linux host

A proven restore answers whether a copy comes back; it does not answer whether the copy contained everything the host needed. That second failure is the quieter one, because restoring an incomplete backup succeeds. Every file in it returns, the exit code is zero, and the gap only surfaces later, when a service refuses to start, an automation account cannot log in, or the machine stops at an emergency shell. This lesson is about producing the list before an incident produces it for you.

Sorting a filesystem: what could a rebuild produce again?

There is exactly one question worth asking about any path on a host, and it is not whether the data looks important. It is whether a rebuild could produce that byte again from something you still hold. If it could, the path is a recipe and what you owe the recovery is the recipe. If it could not, the path is state, and the only way to have it after the host is gone is to have copied it.

The distinction is clean in principle and treacherous in practice, because the answer is almost always given as reproducible in principle when the honest answer is reproducible if several other things are also true. A package tree is reproducible if you know the package set, the versions and the repository, and if that repository still serves them. A configuration directory is reproducible if configuration management genuinely owns every file in it, which is a claim about coverage that can be measured and is almost never one hundred per cent. A container image is reproducible if the registry still holds the tag and the tag still points at the same digest.

So the sort produces three piles, not two: state that must be copied, recipes that must be recorded well enough to execute, and the large middle pile of paths that somebody believes are recipes without having tested the claim. The middle pile is where hosts are lost. The rest of this lesson walks the filesystem to populate the first pile honestly, and then tests it.

Walking the tree from /etc to /boot

/etc is configuration, and configuration is exactly the category people believe they can regenerate. Sometimes they can. The sshd_config that a role templates, the nginx site that a playbook writes, the sysctl drop-in that a baseline enforces — all genuinely reproducible. What is not reproducible is everything that arrived by hand: the /etc/hosts entry added during a DNS incident, the sudoers.d fragment written for a contractor, the unit override that fixed a start-up ordering bug at three in the morning and was never committed anywhere. /etc is also where several items of pure host identity live, which the next section treats individually. It is small, it compresses well, and there is no engineering argument for excluding it. Back up /etc whole and treat configuration management as the reconstruction path, not as the backup.

/var/lib is where the answer to “is this host stateful?” usually is, and it is the directory most often lost to a rule that excludes /var because “that is logs and cache”. Under it sit database data directories, container volumes, the package manager’s own database of what is installed, certificate and ACME account stores, DHCP leases, directory-service caches, and the persistent state of every daemon whose authors needed somewhere to put it. Nothing in that list announces itself. A machine that everyone describes as a stateless application node commonly turns out to have several of them.

/home holds user data and, more consequentially for recovery, the per-user operational state that has accumulated in dotfiles: authorized_keys that grant access, client configuration for cloud and cluster tooling, credential files that were never meant to be there but are. Application data directories follow the same logic wherever they were placed — they are state by definition, because their contents were produced by running the application.

/opt and /srv are the two trees the distribution deliberately does not own. A vendor application unpacked into /opt has been patched, licensed and configured in place, and the installer tarball you kept is not that tree. /srv is served data, which is state in the plainest sense. Neither directory gets regenerated by anything unless you built the thing that regenerates it.

/usr and /boot are the reproducible end of the walk. /usr comes from packages; /boot comes from kernel packages plus a bootloader configuration generated from a template. Reproducible, though, is not the same as ignorable. It means the recipe has to exist and has to be current: the package selection, the repositories and their pinning, the kernel command line, and any initramfs content that is host-specific — a driver, a crypto keyfile, a network configuration baked in at build time. A recipe you cannot execute inside the recovery window is not a recipe; it is a hope.

A correct file list, restored wrong

Suppose the sort above went perfectly and the list of paths is complete. That is still not sufficient, because a file is not only its contents, and the tools most often reached for do not carry the rest by default.

A source tree was built with the metadata a production host actually carries: a POSIX ACL on one file, an extended attribute on another, a file capability on a helper binary, a setuid bit on an admin tool, a 200 MiB sparse image with nothing allocated, and two paths sharing one inode. It was then copied four ways.

$ tar -cf naive.tar src

  [source]
  ACL on app.conf        : 1 entr(y|ies)
  xattr on index.dat     : sha256:deadbeef
  capability on netcheck : cap_net_raw=ep
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 0
  payload.a link count   : 2
  payload a/b same inode : yes

$ tar -xf naive.tar -C r1
archive size: 201M
[restored from default tar]
  ACL on app.conf        : 0 entr(y|ies)
  xattr on index.dat     : ABSENT

  capability on netcheck : ABSENT
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 200M
  payload.a link count   : 2
  payload a/b same inode : yes

The ACL is gone, the extended attribute is gone, the capability is gone, and a file that occupied no blocks now occupies 200M of them — which is also why the archive itself measured 201M rather than the 100K produced when tar was told --acls --xattrs --xattrs-include='*' --sparse. The same tree copied with the flag everyone reaches for loses one more thing:

$ rsync -a src/ r3/

  [restored from rsync -a]
  ACL on app.conf        : 0 entr(y|ies)
  xattr on index.dat     : ABSENT

  capability on netcheck : ABSENT
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 200M
  payload.a link count   : 1
  payload a/b same inode : NO - now two separate files

Told to carry it, both tools carry it:

$ rsync -aAXH --sparse src/ r4/

  [restored from rsync -aAXH --sparse]
  ACL on app.conf        : 1 entr(y|ies)
  xattr on index.dat     : sha256:deadbeef
  capability on netcheck : cap_net_raw=ep
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 0
  payload.a link count   : 2
  payload a/b same inode : yes

The capture is explicit about what this does and does not prove: “Every restore above produced files with the right names, the right sizes and the right contents. A checksum comparison of file CONTENT would have passed for all four.” Content verification is the verification most restore tests actually perform, and it is blind to every difference above.

The flag-by-flag treatment — which attribute each letter requests, and why the extraction command needs them as well as the creation command — belongs to the archive-semantics lesson later in this part. What belongs here is narrower and harder to unlearn: a backup specification is two documents, not one. The list of paths says what to copy, the command says how faithfully, and a perfect list copied by the wrong command still returns a host that does not work.

The items that are missed every time

Seven items go missing more reliably than anything else on a host, and every one of them is small enough that its absence is invisible until the machine is asked to run.

Crontabs outside /etc. Per-user crontabs live under /var/spool, not under /etc, so a backup rooted at /etc/cron.d and /etc/crontab misses them completely. The scheduled work simply stops, silently, on a host that otherwise looks healthy.

Unit overrides and enablement under /etc/systemd/system. Drop-ins and overrides are files, but enablement is a set of symlinks in .wants directories. Restore the unit files without the symlinks and you get a host where the services exist, start by hand, and start not at all at boot.

Local user and group databases. Where accounts are local rather than directory-backed, the passwd, shadow, group and subordinate-ID files are host state, and so are the numeric IDs inside them. Restoring files that carry numeric ownership onto a host that allocated its UIDs in a different order reassigns ownership quietly and plausibly.

SSH host keys. These are generated once. Lose them and every client that pinned the fingerprint refuses to connect, which in practice means the automation, the monitoring and the backup agent all fail simultaneously with a warning that reads like an attack — during the recovery.

machine-id. Generated at install, referenced by whatever derived a stable per-host identifier from it. A rebuilt host with a fresh one is, to those systems, a different machine.

Filesystem UUIDs. A UUID lives in the filesystem superblock and is created when the filesystem is made. Restore files onto freshly created filesystems and the UUIDs are new, while the restored fstab and bootloader configuration still name the old ones. The result is a boot that stops in an emergency shell with every file intact. Capture the identifiers next to the data:

OUT=/var/backups/host-state
mkdir -p "$OUT"
blkid > "$OUT/blkid.txt"
findmnt --real --output SOURCE,TARGET,FSTYPE,UUID,OPTIONS > "$OUT/mounts.txt"

Package selections. Without the installed set, its versions and the repositories they came from, “reproducible from packages” is a claim you cannot execute. Record it on the same schedule as the data, and record enablement and scheduled work with it:

OUT=/var/backups/host-state
dpkg-query -W -f='${binary:Package} ${Version}\n' | sort > "$OUT/packages.txt"
systemctl list-unit-files --state=enabled --no-legend > "$OUT/enabled-units.txt"
find /etc/systemd/system -type f -o -type l | sort > "$OUT/local-units.txt"

State the filesystem walk does not see

A path-based walk finds state that presents itself as files in the tree it walks. Containers break that assumption twice, and the second break is the one that costs data.

The first break is benign once known: a named volume is a directory on the host, and the capture records its mount point as /var/lib/docker/volumes/rbdr-data/_data. That is under /var/lib, so a walk that includes /var/lib picks it up — while a live database inside it needs the consistency treatment the previous part covered, not merely a file copy.

The second break is that the obvious container backup captures the wrong half.

$ docker commit rbdr-app rbdr-committed:v1

  image created

--- start a NEW container from that committed image, with NO volume ---
written-into-the-container-layer
>>> exit code: 0

--- and the file that was in the volume? ---
total 8
drwxr-xr-x    2 root     root          4096 Aug 28 13:48 .
drwxr-xr-x    1 root     root          4096 Aug 28 13:48 ..
>>> exit code: 0

The commit captured the writable layer and nothing of the volume, because the volume was never part of the image. The mirror-image failure appears when the volume is the only thing backed up:

$ docker run --rm -v rbdr-data-restored:/dst -v /tmp/rbdr-out:/in alpine tar xzf /in/rbdr-data.tgz -C /dst

--- the restored service reads its data ---
ORDER-1001,4500.00
ORDER-1002,1250.00
restored md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the volume data was recovered byte-identical

--- but what about the file that lived in the container layer? ---
cat: can't open '/etc/app-marker': No such file or directory
>>> exit code: 1

The orders came back byte-identical against 9eb4e2ad8e08e1dcaaf87ababab964b0, and the file written into the container layer did not come back at all. Both halves of that result are the lesson: volume state is recoverable when it is copied, and layer state, as the capture puts it, “is state nobody is protecting”.

Discovery by restore

Everything above is a list written from knowledge, and lists written from knowledge are always shorter than reality. The only method that produces the real list is to restore.

Take the backup as designed. Restore it onto a host built the way a rebuild would build one — fresh install, configuration management applied, nothing carried over by hand — and then bring the service up and record every single thing that has to be fixed to make it work. Each fix is an item the backup was missing. Do it again with the corrected list until a run produces no fixes.

The comparison can be mechanical as well as behavioural. A manifest of paths with mode, ownership and size, taken on the source and again on the restored host, converts “it seems fine” into a diff:

MANIFEST=/var/backups/host-state/manifest.txt
find / -xdev -printf '%p %m %U %G %s\n' | sort > "$MANIFEST"
comm -23 /var/backups/host-state/manifest-source.txt "$MANIFEST" \
  > /var/backups/host-state/missing-after-restore.txt

What the diff cannot tell you is whether the service works, which is why the behavioural half is not optional. Run both. The list you end with is a measurement; the list you started with was an opinion.

Production discipline

1. Sort every path into state, recipe, or unverified belief. The third pile is the one that loses hosts, and its size is a real number you can report. Anything you cannot demonstrate is reproducible is state until proven otherwise.
2. Back up /etc and /var/lib whole, and argue about exclusions afterwards. Both are small relative to application data, and /var/lib is where the volume mount point /var/lib/docker/volumes/rbdr-data/_data and every other forgotten service state actually lives.
3. Name the metadata on the command line. rsync -a measurably restored the ACL as 0 entr(y|ies), the capability as ABSENT, the sparse image as 200M allocated and the hard link with link count : 1. -aAXH --sparse restored all four correctly, and only the second is a host backup.
4. Capture host identity as explicitly as data. SSH host keys, machine-id, the local account databases, blkid output and the package selection are a few kilobytes whose absence turns a complete file restore into an emergency shell or a fleet-wide connection failure.
5. Derive the list from a restore onto a clean host, then keep re-deriving it. Each fix required to make the restored service work is a defect in the backup specification, and a run that requires no fixes is the only evidence the list is complete.

Cross-course references

• Linux for Production Sysadmins — Part XV (/etc/fstab and Mount Management) covers how mounts are identified by UUID and label rather than by device name. That is the mechanism behind the failure listed above: a restore onto freshly created filesystems produces new UUIDs, so the fstab you faithfully restored now names devices that do not exist, and the host stops before any of the recovered data is reachable.
• Ansible for Production Sysadmins — Part XXXVI (Drift and Convergence) measures the gap between what configuration management declares and what a host actually contains. That measurement is the evidence for or against this lesson’s central claim about /etc: the share of files under management is exactly how much of /etc a rebuild can genuinely reproduce, and the remainder is state whether or not anyone calls it that.
• Docker & Containers for Production Sysadmins — Part VIII (Storage) explains the separation between an image, a container’s writable layer and a named volume. This lesson consumes that separation from the recovery side: it is the reason docker commit returned an empty mount point and the reason /etc/app-marker was unrecoverable after a volume-only backup.

Quiz