Restoring a complete service onto a clean host

Pulling a single file back out of an archive put one byte range where it belonged and left the rest of the machine untouched, which is precisely why it proved so little about the service. The host was already running. The package was already installed at a version somebody chose months ago, the unit file was already present and enabled, the certificate was already trusted, the service account already resolved to the same numeric UID it had at midnight. A single-file restore borrows all of that and never has to declare any of it. The moment the host itself is gone, every borrowed thing becomes something you must supply, in order, from a source you can name.

What the original host was quietly supplying

A restore is only ever a test of the things it is forced to produce. Run it onto the machine that has been serving traffic all week and the list of things it is forced to produce is short: some files, into a directory tree that already exists, owned by an account that already exists, read by a process whose unit file already exists, reachable through a DNS name that already resolves and a firewall rule that is already open.

Everything in that sentence after “some files” is being supplied by the host, not by the backup. None of it is exercised, so none of it is evidence. The restore finishes, the service answers, and the exercise records a success that was underwritten almost entirely by state nobody captured. The gap does not show up as an error, because there is no error to show. It shows up later, on a day when the host is not there to supply anything.

This is why the interesting restore target is a machine with no history. A host that has never run the service cannot lend it a package, a user account, a trusted CA, a scheduled job or an open port. Every one of those has to arrive from somewhere you can point at — a repository, an archive, a configuration repository, a certificate store — and the ones that cannot be pointed at are found in the only way they are ever found, which is by their absence.

The sequence for one service, in the order it has to happen

Reconstruction of a single service on empty infrastructure has a fixed shape, and the ordering is not cosmetic — several stages fail silently when run out of sequence.

Provision the host. An operating system at a known release, with the storage layout the service assumes: the same mount points, the same filesystem types where the service depends on them, enough capacity for the restored data in its restored form rather than in its compressed form.

Install the package set at the versions that were running. Not “the latest”, which is a different system. This requires that the version list is itself part of the backup — a package manifest captured at backup time, or a configuration repository that pins versions. Reconstructing it from memory during an incident is where reconstruction usually stalls.

Restore configuration before data. Configuration decides where data lives, which user owns it and which ports and sockets the service binds. Restoring data first frequently means restoring it into a path the service will not read.

Restore the data itself, with an archive tool told to carry the metadata, because the defaults do not:

SVC=orders
STAGE=/srv/restore/$SVC
install -d -m 0750 "$STAGE"
tar --acls --xattrs --xattrs-include='*' -xf "/backup/$SVC-data.tar" -C "$STAGE"

Restore identity material. SSH host keys, so administrators and automation are not met with a changed-host-key refusal; TLS certificates and their private keys, so clients that pin or verify continue to connect; service account credentials, API tokens and any machine identity the service presents to systems it depends on. This material is not application data and is routinely excluded from the backup that covers application data.

Fix ownership and security context. Numeric ownership survives an archive only if the archive recorded it and the extraction ran with the privilege to apply it, and on a system with SELinux or AppArmor the restored files need their labels re-applied before the service is permitted to read them.

Reconcile UIDs and GIDs if the accounts are local. A package installed on a fresh host takes whatever number that host has free when the account is created, which is very often not the UID recorded inside the archive.

Start dependencies in order, then the service, then the things that depend on the service.

Validate with a request that does real work — one that reads restored data and returns it, not one that confirms a port is listening.

Measured: the volume came back, the container layer did not

The compact version of this whole argument fits in one container. An application container was given a named volume for its data and had two files written into it: one into the volume, one into the container’s own writable layer. Then the state was captured the way people commonly capture container state, with docker commit.

$ docker commit rbdr-app rbdr-committed:v1

  image created

--- start a NEW container from that committed image, with NO volume ---
written-into-the-container-layer
>>> exit code: 0

--- and the file that was in the volume? ---
total 8
drwxr-xr-x    2 root     root          4096 Aug 28 13:48 .
drwxr-xr-x    1 root     root          4096 Aug 28 13:48 ..
>>> exit code: 0

The committed image carried the writable layer and an empty directory where the data should be. The volume was captured separately, by mounting it into a helper container and writing a tar archive — the portable method, because it does not depend on reaching into the daemon’s storage directory. The container and the volume were then both destroyed, which is the part that makes this an actual test rather than a rehearsal, and the archive was unpacked into a new volume that had never held anything.

$ docker run --rm -v rbdr-data-restored:/dst -v /tmp/rbdr-out:/in alpine tar xzf /in/rbdr-data.tgz -C /dst

--- the restored service reads its data ---
ORDER-1001,4500.00
ORDER-1002,1250.00
restored md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the volume data was recovered byte-identical

--- but what about the file that lived in the container layer? ---
cat: can't open '/etc/app-marker': No such file or directory
>>> exit code: 1

Both halves of that output matter, and the second half is the one worth carrying. The data was recovered perfectly: the same md5, 9eb4e2ad8e08e1dcaaf87ababab964b0, on the original and on the restored copy. And the file written into the container layer was simply not there, because it had never been inside the only thing that was backed up. On the original host it would have been present the whole time and nobody would have learned anything.

Identity material, ownership, and the metadata that decides whether it runs

Identity material is the category that most often survives as an assumption. A service is not merely a binary and a data directory; it is a thing other systems recognise. SSH host keys make the rebuilt machine the same machine to every client that has it in known_hosts and to every automation run that refuses to proceed on a mismatch. TLS certificates and their private keys make it the same endpoint to clients that verify. Service account credentials, API tokens and enrolment secrets make it the same principal to the message broker, the object store and the database it needs. None of this is regenerable from the data backup, and some of it is not regenerable at all without a human approving a new issuance.

Ownership is the other half. A restore reproduces file contents easily and file metadata only when it is told to, and metadata is what decides whether the service can read what it just restored. The measured cost of the default is plain:

$ rsync -a src/ r3/

  [restored from rsync -a]
  ACL on app.conf        : 0 entr(y|ies)
  xattr on index.dat     : ABSENT

  capability on netcheck : ABSENT
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 200M
  payload.a link count   : 1
  payload a/b same inode : NO - now two separate files

Every file in that copy has the right name, the right size and the right contents. The ACL that granted a second account read access is gone, the capability cap_net_raw=ep that let an unprivileged binary open a raw socket is gone, a sparse image that occupied nothing now occupies 200M of real blocks, and two paths that shared an inode are two independent files that will drift apart. On the original host, the destination directory usually already carried the right ownership and labels, so none of this was visible. On a clean host it is the difference between a service that starts and a service that reports a permission error nobody can explain at three in the morning.

Numeric reconciliation belongs in the same pass. If the service account is local rather than directory-backed, compare the UID the archive recorded against the UID the fresh installation allocated before anything is started:

SVC_USER=orders
WANT_UID=$(awk -F: -v u="$SVC_USER" '$1 == u {print $3}' /srv/restore/etc/passwd)
HAVE_UID=$(id -u "$SVC_USER" 2>/dev/null || echo none)
printf 'archive uid=%s host uid=%s\n' "$WANT_UID" "$HAVE_UID"

When they differ, the choice is to align the account to the archive or to re-own the tree, and the choice must be made deliberately and once. Discovering it after the service has written new files with the new numbering leaves a directory with two ownerships in it.

The dependencies that surface only on a clean host

The list below is not exotic. Every item is ordinary, every item is invisible on the original host, and every item has stopped a real recovery.

DNS entries. The service name resolves to an address that no longer hosts anything. Clients keep resolving it, and cached negative answers extend the outage past the moment the service is actually ready.

Firewall rules. The rebuilt host has a default policy, not the accumulated rule set of a machine that has been in production for three years. The service listens correctly and nothing reaches it.

Certificate trust. The service presents a restored certificate, but the new host has a stock CA bundle and does not trust the internal issuer, so its own outbound calls fail even though its inbound ones succeed.

Licence files. Node-locked licences are keyed to a hostname, a MAC address or a machine identifier that changed during the rebuild, and the application starts in a degraded mode rather than refusing outright.

Cron jobs and timers. Retention pruning, index rebuilds, report generation and the backup job itself lived in a crontab or a set of systemd timers that nobody listed as part of the service.

Log shipping. The restored host writes logs locally and forwards nothing, which is discovered during the next incident rather than this one.

Monitoring registration. The new host is not scraped and not in any target list, so the recovered service is unobserved, and its dashboards stay flat in a way that reads as healthy.

Which is why the closing step of the sequence is a request rather than a status check. A service that is listening has demonstrated that a binary parsed its configuration and bound a socket. It has not demonstrated that the restored data is behind it, that the account it runs as can read that data, or that its own dependencies answer. Ask it for something specific that only the restored data can produce, and assert on the content rather than on the exit status of the client:

ENDPOINT=https://orders.restore.internal.example/api/orders/1001
BODY=/tmp/orders-probe.json
CODE=$(curl -sS --cacert /etc/ssl/internal-ca.pem -o "$BODY" -w '%{http_code}' "$ENDPOINT")
test "$CODE" = "200" || { printf 'restore not validated: HTTP %s\n' "$CODE" >&2; exit 1; }
grep -q 'ORDER-1001' "$BODY"

That probe exercises the certificate the host presents, the trust store it was given, the route to the service, the account permissions on the data directory and the restored rows themselves. Every one of those was supplied by the original host in the exercise this lesson is arguing against, and every one of them is now being produced rather than borrowed.

Production discipline

1. Restore onto infrastructure that has never run the service. The only dependencies you will discover are the ones the target cannot supply for you, and a host with history supplies almost all of them.
2. Capture the package set and its versions as part of the backup. A manifest written at backup time, or pinned versions in a configuration repository, is the difference between reinstalling the system and installing a similar one.
3. Back up identity material as deliberately as data. SSH host keys, TLS certificates and their private keys, and service account credentials are what make the rebuilt host the same principal to everything that talks to it.
4. Apply ownership, ACLs, capabilities and security labels before start-up. Measured on rsync 3.4.1, the default -a returned 0 entr(y|ies) of ACL and an ABSENT capability while every file name, size and checksum matched.
5. Validate with a request that reads restored data and returns it. A listening port and a green unit state are properties of a process, not evidence that the data behind it came back.

Cross-course references

• Linux for Production Sysadmins — Part IV (Users, Groups and Identity) covers how local accounts get their numeric UIDs and why a fresh installation allocates different ones, and Part XXVIII (SELinux and AppArmor) covers the relabelling step; both are the mechanisms behind the ownership and security context stage of the sequence above, which a restore onto the original host never has to perform.
• Ansible for Production Sysadmins — Part XLI (Service and Application Deployment) is where the package set, unit files and configuration of a service are expressed as something reproducible, which is what turns the “install the package set at the versions that were running” stage from an act of memory into an act of execution.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XI (SSH Keys, Host Trust and SSH CAs) and Part VI (Chains and Trust Stores) explain the two identity failures described here: a rebuilt host that is no longer recognised by its clients, and a rebuilt host that does not trust the internal issuer its own outbound calls depend on.

Quiz