Restore a complete Linux service

1 · Prerequisites

Confirm every item is in place before any state change.

• A named service and a named owner who can accept the recovery point that will be chosen, because that choice discards work and is not yours alone to make
• An incident timeline with at least two timestamps: when the damage began, and when it was noticed. Without the first one there is no criterion for selecting a recovery point
• Read access to the backup repository, and the passphrase or key from wherever it is escrowed rather than from the host being rebuilt
• A target host with the operating system release, the mount points and the free capacity the service assumes, sized for the restored data rather than its stored size
• The package manifest or configuration repository that records which versions were running, since "the current version" is a different system
• The identity material the service presents: TLS certificate and private key, SSH host keys, service account credentials — sourced from the secret manager or escrow, not from the archive of application data
• A scratch path with room for a staged copy of the restored tree, so the archive can be inspected before anything is placed in its final location
• Somewhere to record timings, because the elapsed time of this procedure is the recovery time this service actually has

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Confirm the target has never run this service. systemctl list-unit-files | grep rbdr- and getent passwd for the service account. A host with history lends the restore a package set, an account number, a trusted CA and an open port, and every borrowed dependency is one the exercise cannot discover.
• · Enumerate the dependency set before touching the repository. Packages and their versions, the service account and its numeric UID and GID, unit files and drop-ins, certificates and their trust chain, secrets, name resolution, and every path the service expects to be a mount rather than a directory.
• · Confirm the repository is readable and lists the snapshots you expect. restic snapshots against the repository, with the credential taken from escrow. A repository that cannot be listed is an escalation, not a slow step.
• · Verify the recovery point covers the incident window. Compare snapshot timestamps against the moment the damage began. If every available snapshot is later than that moment, stop: this procedure cannot produce clean data and the plan has to change.
• · Read the archive's own record of ownership. Extract etc/passwd and etc/group from the recovery point into a scratch path and note the numeric UID and GID the service ran as. The fresh installation will allocate whatever number is free, which is very often not that one.
• · Confirm the restore tool will carry metadata. ACLs, extended attributes, file capabilities, sparse allocation and hard links are all carried only when the tool is told to carry them, and their absence is invisible to a content checksum.
• · Confirm the mount layout exists before the data lands. A path that was a separate filesystem on the original host and is a directory on the target will fill the root filesystem quietly and pass every check until it does.
• · Record the start time and the chosen recovery point ID in the incident log before the first write, so the decision is auditable afterwards rather than reconstructed.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Select the recovery point explicitly against the incident timeline, never "the latest". List the snapshots, identify the last one that predates the moment the damage began, and write its ID down. The newest snapshot is the one most likely to contain the damage you are recovering from.
2. 2Have the service owner accept the data loss implied by that recovery point. Everything written between the snapshot and the incident is discarded. That is a business decision with a name attached, and it belongs in the incident log before the restore, not in the review afterwards.
3. 3Decide: original host or clean host. Restoring onto the surviving original is faster and proves almost nothing, because the host supplies the dependencies. Restoring onto a host with no history is slower and is the only version that discovers what the backup is missing. Record which you chose and why.
4. 4Provision the target and create the mount points before any data arrives. Same paths, same filesystem types where the service depends on them, capacity for the restored form of the data.
5. 5Install the package set at the recorded versions, from the manifest or the configuration repository. Reconstructing the version list from memory during an incident is where reconstruction usually stalls.
6. 6Decide: configuration management before or after the data restore. Running it first creates the accounts, units and paths the restore needs and is the safe order. Running it after the data restore can overwrite restored files with the templated version, and on a service whose configuration lives beside its data that overwrite is silent.
7. 7Reconcile numeric identity before anything is extracted. Compare the UID and GID recorded in the archive against what the fresh installation allocated. Align the account to the archive, or plan to re-own the tree — decide once, before files exist with two ownerships in them.
8. 8Restore into a staging path first, not into the final location. A staged tree can be inspected, checksummed and abandoned. A tree written straight into the service directory cannot.
9. 9Restore the data with the metadata flags set, then confirm the restore tool reported success and a file count consistent with what you expected. A restore exit status of 1 with most files written is a real outcome and must be read, not assumed away.
10. 10Prove the restored bytes against a checksum manifest taken before the incident, file by file. This is the step that separates a completed transfer from recovered data.
11. 11Prove the metadata survived, not only the contents. ACL entries, extended attributes, file capabilities, sparse allocation and hard links. A checksum comparison passes on a tree that has lost all five.
12. 12Install identity material from escrow: TLS certificate and private key, SSH host keys, service account credentials and API tokens. This material is not application data and is routinely absent from the archive that covers application data.
13. 13Move the staged tree into place and apply ownership and security labels before anything is started, then relabel for SELinux or AppArmor where the platform uses them.
14. 14Start the dependencies in order, checking each before starting the next: mounts, then the secret agent or credential helper, then the datastore, then the service, then anything that fronts it. Starting the service before its mount is present writes new data into the underlying directory.
15. 15Validate with a transaction that exercises the whole path — a write followed by a read-back, plus a read of a record that predates the incident. A listening port proves a binary parsed a configuration file.
16. 16Record the timings and the recovery point, then hand back deliberately. Time to first byte restored, time to service start, time to a validated transaction, and the total. That total is this service's measured recovery time.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓restic snapshots exits 0 and the row for the chosen ID carries a Time earlier than the damage time written in the incident log. The ID in the log and the ID passed to the restore command are the same string, compared character by character rather than recognised at a glance.
• ✓The restore ran with its status captured — restic restore "$SNAP" --target "$STAGE"; echo "restore exit: $?" — and the recorded line reads restore exit: 0. A Summary: line of the form Restored 6 / 7 files/dirs with a non-zero exit is a failed restore however calm the console looks.
• ✓md5sum -c run from inside the restored tree against the pre-incident manifest prints OK for every path and exits 0. One FAILED line, or exit 1, means the bytes are not the bytes whatever the restore reported.
• ✓Metadata survived, checked one command at a time: getfacl -p on the configuration file lists the extra entry rather than only the three base ones; getfattr -d on the indexed file prints the stored attribute rather than nothing; getcap on the capability-bearing binary prints cap_net_raw=ep rather than an empty line; du --apparent-size and du still disagree on the sparse image; and stat -c %h on a hard-linked path still returns a count above 1. Empty output at exit 0 is the failure mode here, not a non-zero status.
• ✓find "$STAGE" -nouser -o -nogroup prints nothing. find exits 0 whether or not it matched, so the empty output is the result and the exit status is not.
• ✓systemctl is-active prints active for each unit and exits 0, and it was run after each unit rather than once at the end, so the order the dependencies came up is recorded rather than assumed.
• ✓journalctl -u rbdr-orders.service over the restore window shows one start, not a repeating start-fail-start cycle. A unit that is restarting every thirty seconds still answers active to is-active in the moment you happen to ask.
• ✓The write probe returned HTTP 201 and the script did not take its abort branch: test "$CODE" = "201" exited 0. A response body on its own is not the check; the status code is.
• ✓The read-back of the probe record and the read of a pre-incident record both exited 0 through grep -q. The second is the one that matters: it proves the restored rows are behind the running service rather than an empty schema that answers politely.
• ✓The four timings are written down and compared against the recovery time objective, and the comparison is reported whether it was met or missed. An unreported overrun silently becomes the plan's number.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶The service was started and is wrong: stop it before anything else, then unmount the data path. A partly started service writes as it runs, and every second it runs adds records that belong to no consistent recovery point.
• ↶Roll back in the reverse of the start order — dependents, service, datastore, credential agent, mounts — so nothing restarts the thing you just stopped through a dependency.
• ↶Keep the staged tree. It is the evidence of what the archive actually contained, and deleting it to reclaim space destroys the only record of a restore that went wrong.
• ↶If the service wrote into the final path before its mount was present, those writes are underneath the mount and invisible once it is mounted. Unmount, list the directory, and move what is there aside before remounting.
• ↶If a configuration management run overwrote restored files, do not re-run it to fix the result. Restore the affected paths again from the same recovery point and disable the run until the ordering decision has been made deliberately.
• ↶If DNS, a load balancer or a monitoring target list was pointed at the rebuilt host, reverse those changes before stopping it, or clients will fail against an address that answers nothing.
• ↶Reverse the reflex mutations too: silences raised on the alerts for this service, monitor thresholds edited during triage, and any speculative restart timers added while the service was flapping.
• ↶Record what was restored, to what point, what was discarded and what was rolled back. A restore is a decision about which data survives and needs a written record.

6 · Escalation

When the runbook isn't enough, contact:

• · Every available recovery point is later than the moment the damage began: escalate to the service owner and the backup platform owner immediately. There is no clean recovery point, and the decision is now about partial recovery or reconstruction, not about this procedure.
• · The repository reports damaged packs, or a read-data verification fails: escalate to the backup platform owner and switch to the second copy of the repository. Do not repair the only copy you have while it is the only copy you have.
• · A restore completes with a non-zero exit status and missing files: escalate before starting the service. A partially restored dataset that starts is worse than one that does not, because it begins accepting writes on top of a gap.
• · Identity material cannot be recovered — the certificate private key or the service credential is not in escrow: escalate to the security owner. New issuance is a human decision and its lead time belongs in the incident timeline now, not when the service is otherwise ready.
• · The restore is on the critical path of an outage and elapsed time is exceeding the recovery time objective: escalate to the incident owner with the measured remaining work, so they can decide about degraded service instead of waiting.
• · The original host is alive and may still be writing to shared storage or answering for the same name: escalate before starting the rebuilt service. Two instances believing they own the same data is not recoverable by restarting either of them.
• · The validation transaction fails after every step reported success: escalate rather than iterating on the service. The failure is usually in the dependencies the rebuilt host was never given — trust store, resolver view, firewall policy, monitoring registration.

References

1. restic documentation, Restoring from backup
2. restic documentation, Working with repositories
3. GNU tar manual
4. rsync manual page
5. systemd.unit manual page
6. NIST SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems