Restore a deleted file

1 · Prerequisites

Confirm every item is in place before any state change.

• The absolute path of the file, taken from whoever reported it rather than inferred from a directory listing. A restore to a path one character away from the intended one looks exactly like a restore that did nothing.
• Read access to the repository or snapshot store holding the recovery points, and the credential needed to open it, obtained from somewhere other than the host you are recovering.
• The name of the person or team who owns the content, and a way to reach them during this procedure. They are the only party who can confirm that what came back is the version they expected.
• Knowledge of whether the target path sits inside a running service's data directory, and the identity of whoever is authorised to stop that service.
• A staging directory on the target host, outside every service data directory, with free space for what the restore will unpack.
• The ownership, mode and security context the consumer of the file expects, or a sibling file in the same directory to read them from.
• Somewhere to write the record this procedure produces: the recovery point used, the RPO it paid, and how the file went away.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Confirm the file is actually missing, and confirm the path. sudo ls -la "$(dirname "$TARGET")" and sudo stat "$TARGET". A file that was renamed, a path with a transposed character, and a genuine deletion all reach you as "it is gone", and only the first two are cheap to fix.
• · Establish when the file was last correct. The directory's mtime brackets the deletion, the service journal usually names it, and the person reporting it can say when they last opened the file. You need a time before you look at any recovery point, because every later decision is a comparison against it.
• · Establish how the file went away. A human rm, a deployment that rewrote a directory, a log rotation, a failing disk and a compromised account all produce a missing file and all imply different recovery points. This question is not post-incident tidying; it decides how far back you have to go.
• · Determine whether the target path is inside a running service's data directory. sudo fuser -v "$(dirname "$TARGET")" and sudo lsof +D "$(dirname "$TARGET")". If a process holds that directory open, this procedure aborts here and does not resume until the service has been stopped by someone authorised to stop it.
• · Confirm the repository opens and lists recovery points covering the damage window. sudo -E restic snapshots --tag daily must exit 0 and print at least one row. Exit 12 with Fatal: wrong password or no key found means the credential is wrong, not that the repository is empty, and that is a key-recovery problem rather than this procedure.
• · Confirm the staging directory exists, has free space, and is not inside any service data directory. df -h "$RB". Restoring into the path you are trying to protect is how a recovery becomes an outage.
• · Confirm you can reach the content owner now. A restore you cannot get confirmed is a restore you cannot close, and discovering that after the file is in place means the file sits there unvalidated.
• · Record the start time. date -Is | tee "$RB/record.txt". The elapsed time of a single-file recovery is one of the few recovery numbers most estates measure often enough to trust.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Open the written record before you touch the repository. Path, reported time, damage time, who reported it, who owns it. Everything the procedure decides gets appended to this file, because the questions asked afterwards are all about the decisions, not about the copy.
2. 2Fix the damage time from evidence, not from the report. Read the parent directory's mtime, the service journal around it, and the audit log if the host keeps one. Write the resulting timestamp into the record. This is the reference point every subsequent step is measured against.
3. 3List the recovery points and read the Time column, not the bottom row. sudo -E restic snapshots --tag daily. Listings are sorted oldest first, so the newest sits at the bottom and the eye lands on it. That position is a rendering artefact, not a recommendation.
4. 4Select the newest recovery point whose time is earlier than the damage time, and write the ID into the record before restoring anything. This is the decision the runbook exists for. The newest copy is right for a deletion noticed immediately and wrong for a corruption that has been backed up on schedule for a week, and the two arrive on the phone sounding identical.
5. 5Decision point — restore in place, or restore beside and let the owner compare. Restore in place only when the path is currently absent, the consumer is not running, and the owner has already described the content they expect. Restore beside whenever the path still holds something, whenever the file is contested, and whenever the owner needs to look before you commit.
6. 6Decision point — is this a configuration file whose restoration changes running behaviour? If a service reads the path, putting an older version back is a configuration change, not a file copy. Treat it as one: know which settings differ, know who approved the newer values, and schedule the reload rather than letting the next unrelated restart discover it.
7. 7Restore into the staging directory, never onto the live path first. sudo -E restic restore "$SNAP" --target "$STAGE". The staged copy is what makes the comparison possible, and the comparison is what turns "I restored a file" into "I know what the restore changed".
8. 8Read the lines the restore prints, and its exit code. The first line names the snapshot and the timestamp it is unpacking; the summary line names how many files arrived. A non-zero exit, or a summary reading Restored 6 / 7, means part of the tree did not come back and the staged copy must not be installed.
9. 9Preserve whatever is currently at the target path before writing anything. sudo cp -a "$TARGET" "$RB/evidence/". cp -a keeps mode, ownership and timestamps, and that metadata is most of the evidence about how the file came to be wrong.
10. 10Compare the staged copy against the live path. sudo diff -u "$STAGE$TARGET" "$TARGET". An empty diff on a file you believed was damaged means the recovery point you chose already contains the damage; go back one row and repeat. On a genuine deletion the comparison is against the preserved neighbour and the checksum instead.
11. 11Place the file with the ownership, mode and security context the consumer needs. Use install rather than cp: cp onto an existing path writes through the existing inode and leaves whatever owner and mode were already there, producing a byte-perfect file the service still cannot open.
12. 12Have the owner confirm the content is the version they expected. Show them the file, or the diff, or the specific values they named earlier. "A file exists at that path" is a filesystem observation; only the owner can say whether the recovery worked.
13. 13Record the recovery point used, the RPO it paid, and how the deletion happened. The RPO paid is the interval between the chosen recovery point and the damage time, and everything legitimately written into the file inside that window is gone. Say so explicitly before anyone is told the incident is closed.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓grep -c "chosen recovery point" "$RB/record.txt" prints 1 and exits 0. The record names the damage time, the recovery point ID, and the reason that point was chosen in preference to the newest one.
• ✓grep -m1 "restoring snapshot" "$RB/record.txt" exits 0 and the timestamp on the line it prints is earlier than the recorded damage time. That line is restic naming the recovery point it actually unpacked, which is the only cheap check that you took the one you chose.
• ✓grep -c "^Fatal:" "$RB/record.txt" prints 0. Any Fatal: line in the captured restore output means the staged tree is incomplete and must not be installed, whatever the summary line says.
• ✓sudo stat -c '%n owner=%U:%G mode=%a size=%s' "$TARGET" exits 0 and prints the ownership and mode the consumer expects, not the staging directory's defaults.
• ✓sudo sha256sum "$STAGE$TARGET" "$TARGET" exits 0 and prints the same digest on both lines. Two different digests mean install placed something other than the copy you compared.
• ✓On an SELinux host, sudo restorecon -nv "$TARGET" exits 0 and prints nothing, meaning the label already matches policy; any line it prints is a relabel it would have made, and the consumer would have failed on that label.
• ✓If the path still held something when you started, sudo test -e "$RB/evidence/$(basename "$TARGET")" exits 0 and sudo stat -c '%U:%G %a %y' "$RB/evidence/$(basename "$TARGET")" prints the metadata the live file carried before the change.
• ✓If a service consumes the path, sudo systemctl reload rbdr-appd exits 0 AND sudo journalctl -u rbdr-appd --since "-2 min" --no-pager shows the service reading and parsing the file. Active: active (running) is a claim about the unit, not about the file.
• ✓The content owner has stated, in writing, that the content is the version they expected — naming a value, a row, a date or a setting, not merely acknowledging that a file is present. This is the only line here with no command and no exit code, and the only one that closes the procedure.
• ✓grep -c "RPO paid" "$RB/record.txt" prints 1 and exits 0. The record states the interval this recovery paid and names what was written into the file inside that window and is therefore gone.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶If the restored content turns out to be the wrong version, put the preserved copy back with install using the metadata recorded before the change, and start again from the recovery point selection step rather than reaching for the next snapshot down the list.
• ↶If the file was placed beside the live path for comparison, remove the .rbdr-restored sibling once the decision is made. A stray restored copy in a configuration directory is read by the next person as an intentional file.
• ↶If a service was reloaded to pick up the restored file and its behaviour changed for the worse, roll the configuration back first and the file second; the running service is the urgent half.
• ↶Undo the mutations made by reflex: an alert silence taken out at the start, a monitoring check disabled to stop the noise, a speculative service restart. These outlive the incident and are invisible in the record unless you write them down.
• ↶Remove the staging directory once the evidence artefacts have been copied out of it. A staged restore left in a home directory is an unlabelled copy of production data with none of the repository's protections.
• ↶Whatever the outcome, leave the record in place. A recovery that was rolled back is a more useful record than one that worked, and it is the only account of why the first recovery point was rejected.

6 · Escalation

When the runbook isn't enough, contact:

• · A process holds the target directory open and nobody present is authorised to stop the service: escalate to the service owner. Do not restore into a data directory underneath a running process, and do not stop the service on your own judgement to make the restore possible.
• · No recovery point predates the damage time: escalate to whoever owns the backup schedule. This is a retention or a schedule failure, and the answer is a decision about how much loss is acceptable, which is not the responder's to make.
• · The deletion looks deliberate, or the account that performed it is not one that should have had access: stop, escalate to security, and preserve the current state. A restore overwrites the evidence of how the file went away.
• · The owner cannot confirm the restored content, or disputes it: escalate rather than closing. An unconfirmed restore that is quietly declared finished is how a wrong version reaches production and stays there.
• · The file is one of many missing from the same directory, or the same pattern appears on other hosts: escalate immediately. Single-file recovery is the wrong procedure for what is now a different incident.
• · The restore prints ciphertext verification failed or exits non-zero, or the two checksums disagree: escalate to whoever owns the repository. A repository that cannot return one file has said something about every other file it holds.
• · The repository refuses the credential with Fatal: wrong password or no key found: escalate to whoever holds the escrowed passphrase. Every byte may be present and still unreadable, and no amount of retrying on this host changes that.

References

1. restic documentation, Restoring from backup
2. restic documentation, Working with repositories
3. restic documentation, Troubleshooting
4. install(1) manual page
5. cp(1) manual page
6. stat(1) manual page
7. diff(1) manual page
8. restorecon(8) manual page