Recover an accidentally deleted dataset

1 · Prerequisites

Confirm every item is in place before any state change.

• Administrative access on the host that still holds the affected path, and on whatever runs the schedule — the two are frequently not the same machine, and the schedule is the one that matters first.
• A written inventory of every job that reads this dataset and writes it somewhere else: mirrors, offsite sync, storage replication, repository prune, lifecycle expiry, and the configuration-management or GitOps loop that would re-enable any of them.
• Read credentials for each repository and bucket that might hold a recovery point, distinct from the production identity that performed the deletion.
• Free capacity on a separate path for a full restore beside the original, sized from the last known dataset size rather than from what remains.
• Agreement, in advance, that stopping a scheduled synchronisation during an incident does not need a change ticket. If that agreement does not exist, obtaining it is the first escalation, not the first step.
• A named data owner who can state whether a recovered copy is the right copy, and who is reachable now.
• A place to record timings, decisions and identifiers as you go, because the recovery point you choose has to be defensible afterwards.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Stop every scheduled job that could copy the current state onward, before you read a single log. This is the one pre-check that is also an action, and it is deliberately first. In the measured capture a single scheduled rsync -a --delete run removed the operator's deleted file from the mirror as well, with exit code 0: Elapsed time between the mistake and the loss of the only other copy: one scheduled interval. Nobody had to make a second mistake.
• · Confirm the jobs are stopped rather than merely disabled at next boot. A stopped timer with an unmasked unit is one systemctl start, one configuration-management run or one reboot away from firing. Check the system timers, root and service crontabs, the backup platform's own scheduler, and the storage array's replication schedule separately; they fail independently and each one is sufficient on its own.
• · Confirm at least one copy retains history rather than reproducing current state. A mirror, a storage replica and a continuous sync all converge on whatever the source looks like now, so after the deletion they hold the deletion. If every copy in the inventory is of that kind, stop and escalate before spending time on a recovery that has no source.
• · Confirm free capacity for a full restore beside the original. The staging restore is what makes every later step reversible; if it does not fit, that constraint changes the plan and needs to be known now rather than at 80% of a transfer.
• · Confirm the credentials you will use are not the identity that performed the deletion, and are read-only wherever the platform allows it. On object storage the measured mc rm from a readwrite identity exited 0, printed Created delete marker, and left the object out of every ordinary listing while its bytes stayed stored. An investigation run with the same rights can quietly add more markers.
• · Confirm the deletion is confined to this dataset. Look at the siblings — adjacent directories, other prefixes in the bucket, other volumes on the host. A deletion that stops neatly at one path is usually a mistake; one accompanied by renames, new extensions or unreadable files elsewhere is a different incident with a different first responder.
• · Record the start time and the last known good state. The capture's 09:00 source tree recorded orders.csv md5: 9eb4e2ad8e08e1dcaaf87ababab964b0 before anything was deleted. A digest taken before an incident is worth more than any amount of reconstruction afterwards, and noting whether one exists takes ten seconds.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Stop the scheduled synchronisation first, and investigate second. Every mirror, offsite sync, replication task and repository prune that touches this dataset. Investigation costs minutes; a scheduled run costs the copy you were about to recover from, and it does not need anybody to make a second mistake.
2. 2Mask the units you stopped, and write down which ones were enabled and when each last fired. The mask is the difference between "stopped" and "stopped and staying stopped" while a configuration-management run is in flight. The list of what you masked is also the rollback list.
3. 3Freeze the surviving state before you work on it. A read-only remount, a filesystem snapshot, or a copy of what remains. Everything after this point reads that frozen state, so an investigative mistake costs nothing and the original evidence survives the recovery.
4. 4Establish when the deletion happened, from the filesystem and the logs rather than from the report. The newest surviving modification times bound the window from below; the journal, the audit log and the object store's access log name the actor and the moment. The time somebody noticed is not the time it happened, and the gap between them decides how many scheduled runs have already gone.
5. 5Establish what else the deletion touched. Count what is missing against the last known inventory, and check the paths nobody mentioned. Scope is what tells you whether this is a recovery or a security incident, and it is cheaper to answer now than after the restore has overwritten the evidence.
6. 6Sort every copy into one of two kinds: it retains history, or it reproduces current state. Snapshots, repository archives, versioned objects and offline media retain history. Mirrors, replicas and continuous sync targets reproduce current state and after the deletion they hold the deletion. Only the first kind is a source for this recovery, and knowing which is which is the whole reason the schedule was stopped.
7. 7On object storage, list versions rather than objects. A delete on a versioned bucket that carries no version id writes a zero-byte delete marker above the object; the bytes stay stored and the ordinary listing returns nothing. The measured listing showed the marker at v2 DEL and the intact 38-byte object still at v1 PUT beneath it. An empty ls is not evidence of absence on a versioned bucket.
8. 8Select the recovery point explicitly, and never take "the latest". The one you want is the newest recovery point whose contents predate the deletion window — not the newest that exists, which may already have been taken after it. Name it in the record by repository, identifier and timestamp, and state what is lost between that point and now, because that gap is the cost of the recovery and somebody other than you has to accept it.
9. 9Decision point — restore beside, or restore in place. Restore beside by default: it costs capacity and a comparison step, and it leaves every option open. Restore in place only when the path is unused by any live consumer, capacity genuinely will not hold a second copy, and a frozen copy of the current state already exists. Record which you chose and why.
10. 10Restore into the staging path, with the ownership and permission model preserved. A restore that arrives owned by the wrong user is a second incident dressed as a success, and it is discovered by the application rather than by you.
11. 11Compare the restored tree against what survived. File counts, sizes and digests, and an explicit list of what the restore brings back that is currently absent. This is where an incorrectly chosen recovery point shows itself, while nothing has been moved into the live path yet.
12. 12Validate with the data owner against a business question, not a process question. "The restore completed" is a statement about a transfer. "The 27 August invoices are present and the ledger totals match the finance report" is a statement about the data. Only the second one ends the recovery, and only the owner can make it.
13. 13Cut over deliberately, once and with a named authoriser. Place the recovered data, set ownership, permissions and security labels to match the surviving siblings, then restart the consuming service and watch it read the data rather than watch it start.
14. 14Re-enable the schedule last, and watch the first run. Run the synchronisation in dry-run mode first and read what it proposes to delete. Re-enabling a mirror before the source is correct simply propagates the current state again, which is the same mechanism that caused the loss.
15. 15Record the recovery point, the decision at each decision point, the timings and the sign-off. The chosen recovery point is the part somebody will question in a month, and the record is the only answer.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓No timer, cron entry, platform schedule or replication task that writes this dataset is enabled, and the list of what was masked matches the list in the record.
• ✓The deletion window is stated as a bounded interval with the evidence for each bound — the newest surviving modification time below it, the log entry or access record above it.
• ✓The chosen recovery point is named by repository, identifier and timestamp, and its timestamp is inside the interval before the deletion, not merely "recent".
• ✓On object storage, a listing that shows versions returns the object versions and any delete markers; the absence of the object from an ordinary listing is explained by a marker rather than treated as loss.
• ✓The restored tree matches expectation by file count and by digest on a sample the data owner chose, not on a sample you chose.
• ✓Ownership, permissions and security labels on the recovered data match the surviving siblings, checked by listing rather than assumed from the restore tool.
• ✓The consuming service reads the recovered data through its normal path — a query that returns rows, a report that renders, a job that completes — and the data owner has confirmed the result in business terms and by name.
• ✓The first re-enabled synchronisation run was executed in dry-run mode and proposed no deletions, and the subsequent real run was watched to completion.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶The staging restore is itself the rollback. Nothing was overwritten, so abandoning the recovery costs the staging capacity and nothing else; delete the staging tree only after sign-off.
• ↶If the restore was made in place and is wrong, the frozen copy taken before the restore is the only way back. That copy exists only if it was taken, which is why freezing the surviving state is a numbered step rather than a suggestion.
• ↶Unmask and re-enable every unit you masked, from the list you wrote down. The mask is a mutation you made and it is invisible until the night it silently skips a backup.
• ↶Re-enable any monitoring silence or alert route you suppressed while working. A silence outliving the incident is the most common reflex mutation left behind by a recovery.
• ↶Revert any speculative service restart, mount option or permission change made while investigating, and say so in the record even when it turned out to be harmless.
• ↶If a version or a delete marker was removed on the object store during the investigation, record the identifier, the time and the identity. That action is not reversible and the next responder must not have to discover it.

6 · Escalation

When the runbook isn't enough, contact:

• · Every copy in the inventory reproduces current state and none retains history: escalate immediately. This is a capability limit rather than a procedural problem, and the honest early statement is far cheaper than a day spent looking for a recovery point that was never taken.
• · The deletion is not confined to the dataset, or the pattern suggests a deliberate act: escalate to security before restoring anything. A restore overwrites the evidence about what else happened, and preserving a copy first is a decision somebody else owns.
• · The only recovery point that predates the deletion also predates a schema, format or credential change: escalate to the application owner. Recovering data the current application cannot read is a second outage, not a recovery.
• · The object versions themselves are gone rather than hidden behind delete markers: escalate to the storage platform owner. A missing marker is a listing question; a missing version is a retention or lifecycle question, and the answer is not on this host.
• · The recovery is on the critical path of an outage and is exceeding the agreed recovery time: escalate to the incident owner with the measured remaining work, so that partial service on older data can be somebody's explicit decision.
• · Nobody can authorise the cut-over: escalate rather than cutting over to tidy up. A validated staging restore with the schedule still stopped is a safe state and can be left in it for hours.

References

1. rsync(1) manual page
2. Amazon S3 User Guide, Working with delete markers
3. Amazon S3 API Reference, ListObjectVersions
4. MinIO Object Storage documentation
5. restic documentation
6. systemd.timer(5) manual page
7. NIST SP 800-184, Guide for Cybersecurity Event Recovery