Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-mirror-propagation~50 min

The offsite mirror faithfully reproduced the ransomware encryption

Reported symptoms

  • ●At 08:20 the finance team reports that no file on the shared drive will open and that every filename has gained a `.locked` suffix
  • ●The offsite copy on `rbdr-mirror-host` synchronised normally at 06:04 that morning and the monitoring tile for the synchronisation job has been green every day for eleven months, including this one
  • ●The operator who opens the incident expects a short recovery, because the offsite copy is held at a second site on separate hardware and separate power
  • ●At 08:41 a listing of `/srv/rbdr-mirror/finance` shows `orders.csv.locked` and 3,179 other encrypted names, with no unencrypted file among them
  • ●No earlier version of any file exists anywhere on the mirror host: the destination holds exactly one copy of each name and it is the copy the 06:04 run wrote
  • ●The synchronisation job did not fail, did not warn, did not slow down noticeably and produced no alert of any kind
  • ●The incident is initially handled as a possible second intrusion, on the theory that the attacker reached the mirror host as well, which costs forty minutes before the timestamps are read

Evidence

  • · A measured propagation capture on the same rsync 3.4.1 build the job runs shows the destination reproducing an in-place encryption of the source on the next scheduled run, with the mirrored ledger reduced to a `Salted__` header and a closing count of zero surviving plaintext copies
  • · The job log for the 06:04 run records 3,180 files created, 3,180 files deleted and 2,041,118,720 bytes transferred in seven minutes and forty-one seconds, against a historical norm of eight to forty files and roughly 60 MiB
  • · The exit status recorded by the job wrapper and read by the monitor is 0, and the monitor exports exactly two series: the last exit status and the age of the last successful run
  • · Every destination file carries a modification time inside the 03:41 to 03:58 window in which the source was encrypted, because `rsync -a` preserves modification times from the source
  • · Every destination file carries an inode change time inside the 06:04 to 06:11 synchronisation window, which is the time at which the local kernel wrote the file and which no remote process can set
  • · The site firewall flow log for the night contains no session from the office VLAN to `rbdr-mirror-host`; the only session between the two hosts is the mirror host outbound SSH connection opened at 06:04:09 by the pull job itself
  • · The mirror host holds one directory for this share, no `--backup-dir` target, no filesystem snapshots and no versioned object store behind it
Diagnosis and resolutionclick to reveal

Root cause

Two defects, and the second is why the first was allowed to run to completion. The immediate cause is the definition of the destination. `rsync -a --delete` does not copy events, it makes a destination match a source: files present only at the destination are removed, and files whose name or content differs are replaced. Under that definition an operator deletion, a truncation and an in-place encryption are all simply differences to be resolved, and resolving them is the job working correctly. Because the destination retained no prior version — no `--backup-dir`, no snapshot, no object versioning — resolving the difference consumed the only remaining copy of the pre-attack state. The control that should have caught it is the monitoring. It watched the job result: the exit status of the last run and the age of the last successful run. Both of those stayed correct throughout, because the run genuinely succeeded. Neither is a function of what the run did. The morning of the incident differed from every previous morning by three orders of magnitude in files deleted, files created and bytes transferred, and every one of those numbers was written into the job log by rsync and read by nobody. The two defects compose. History at the destination without monitoring would have made the incident survivable and slow. Monitoring on change volume without history would have made it visible and still unrecoverable. Having neither meant the estate had one recovery point, and the schedule overwrote it at 06:04.

Remediation

Treat the destination as evidence until the pre-attack state is located. Stop the timer on the mirror host first, because the next scheduled run will re-apply the same transformation to anything restored beside it, and take a read-only copy of the encrypted destination before touching it — it is the best surviving record of what the source held at 03:58. Then search for a copy the mirror never touched. In this estate that meant the quarterly export written to a separate repository, which was eleven days old, and the finance workstations, which held locally cached copies of four of the current spreadsheets. Recovery was assembled from those two sources plus re-keyed transactions, and the loss window was eleven days rather than nine hours. Rebuild the arrangement before restoring into it. Add history at the destination: `rsync(1)` documents `--backup` with `--backup-dir=DIR`, which stores every displaced or deleted file under a per-run directory on the receiving side instead of discarding it, and that alone converts this incident into a directory copy. Add a ceiling on destructive change: `rsync(1)` documents `--max-delete=NUM`, which skips all further deletions once NUM is exceeded, warns, and exits with error code 25. On 3,180 deletions a ceiling of 200 would have stopped the run before it finished. Longer term, move the offsite copy from a mirror to a repository that stores snapshots as separate, additive objects, so that a new snapshot cannot rewrite an older one, and keep the mirror only for the failure classes it actually answers.

Verification

Verify the reconstruction against the two defects, not against the job. For history: encrypt a file in a scratch tree on the source, let the scheduled run execute, and confirm the pre-change bytes are readable from the run backup directory at the destination and that their checksum matches the value recorded beforehand. A destination that cannot produce yesterday on demand has not been fixed. For the ceiling: delete more files than the configured `--max-delete` value in the scratch tree, run the job, and confirm the run reports skipped deletions and a non-zero status, and that the destination still holds the files. For the monitoring: confirm the job exports files created, files deleted and bytes transferred as separate series, that the deletion series has an alerting rule keyed to its own recent history rather than to a fixed number, and that firing it does not require a human to open a dashboard. Prove it by tripping it deliberately. For the isolation claim: confirm from the firewall that the office VLAN still cannot open a session to the mirror host, and that the mirror host key on the source is restricted to the read-only transfer command.

Prevention

**Write down which failure classes the copy answers.** A mirror is strong against device, host and site loss and has no defence against deletion, corruption or encryption, because it is obliged to reproduce them. An arrangement that cannot name the second list has been assembled rather than designed. **Keep at least one copy the source cannot modify.** Pull-based transfer protects the destination *host* from a compromised source, and this estate proved that: the attacker never reached the mirror host. It does nothing for the destination *data*, because the pull still applies the deletions. Only history at the destination, or a repository under a retention lock that the backup credential cannot shorten, protects the data. **Monitor the change, not the outcome.** Exit status and last-success age are liveness checks for the schedule. Add files created, files deleted, bytes transferred and total object count at the destination, and alert on a deviation from each series own recent behaviour. Every backup incident in this course was preceded by a metric that had already moved. **Put a ceiling on destructive change and let it fail closed.** `--max-delete` is a single option that converts an unattended catastrophic run into an alert. **Retain long enough to survive detection latency.** Ransomware is typically found hours to weeks after it lands. A retention that covers one day covers no realistic detection window.

Reported symptoms

At 08:20 the finance team reports that nothing on the shared drive will open. Every name has gained a .locked suffix.

The share lives on rbdr-fileserver, on the office VLAN. The offsite copy lives on rbdr-mirror-host at the second site, which pulls the share every morning at 06:00 with rsync -a --delete over SSH into /srv/rbdr-mirror/finance.

The operator who opens the incident at 08:26 expects this to be short. The offsite copy is on separate hardware, separate power and a separate site, and it synchronised at 06:04 this morning. The monitoring tile has been green every day for eleven months.

At 08:41 they list the destination. It holds orders.csv.locked and 3,179 other encrypted names, and no earlier version of anything.

Evidence provided

The propagation behaviour was not inferred after the fact. It was measured beforehand on the same rsync build the job runs, and the capture’s third night is this incident in miniature: the source is encrypted in place, the scheduled run executes, and the destination is made to match.

Data-loss riskthe measured run — source encrypted, destination made to match
$ rsync -a --delete src/ mirror/
--- source after the attack ---
src/orders.csv.locked
src/reports/q3.txt.locked
--- mirror after that run ---
mirror/orders.csv.locked
mirror/reports/q3.txt.locked

Whether the mirrored ledger is still business data the capture settles by reading its first bytes rather than by asking the job.

Read-only / Safethe mirrored ledger, read as bytes
$ od -c mirror/orders.csv.locked
--- is the mirrored ledger still readable business data? ---
0000000   S   a   l   t   e   d   _   _   < 370 351 357 256   N   J   t
0000020 363 225   5 016   V   r 034   .  \0 220 274   ] 345 237 033   &

plaintext copies of the ledger remaining anywhere: 0

The capture’s closing sentence is the one to carry into the incident: the mirror did exactly what it was configured to do, on schedule, with exit code 0, and it did it to the only other copy of the data.

The second piece of evidence is this morning’s job log. Nothing in it is wrong.

Read-only / Safethe 06:04 run, as the job wrapper recorded it
$ tail -12 /var/log/rbdr-mirror/sync-2026-08-27.log
Number of files: 3,214 (reg: 3,180, dir: 34)
Number of created files: 3,180 (reg: 3,180)
Number of deleted files: 3,180 (reg: 3,180)
Number of regular files transferred: 3,180
Total transferred file size: 2,041,118,720 bytes

sent 121,946 bytes  received 2,043,551,802 bytes  4,431,721.19 bytes/sec
total size is 2,041,118,720  speedup is 1.00
started 06:04:11  finished 06:11:52
rsync exit status: 0

Illustrative output

A normal run for this share moves eight to forty files and about 60 MiB. The monitor consumed one line of that log: the exit status.

The third piece is a negative — what the destination does not contain — together with two timestamps that separate the source’s clock from the mirror host’s.

Read-only / Safepreserved modification time against local change time
$ stat -c '%n  Modify:%y  Change:%z' /srv/rbdr-mirror/finance/orders.csv.locked
orders.csv.locked  Modify:2026-08-27 03:47:12 +0000  Change:2026-08-27 06:07:41 +0000

Illustrative output

/srv/rbdr-mirror/ contains one directory for this share. There is no --backup-dir target, no filesystem snapshot and no versioned object store behind it. The firewall flow log for the night records no session from the office VLAN to rbdr-mirror-host; the only session between the two hosts is the mirror host’s own outbound SSH connection at 06:04:09.

Work the evidence before reading on

  1. The run created 3,180 files and deleted 3,180 files. Which line of the job log would a monitor have had to read to see that, and which line did it actually read?
  2. The destination modification times fall inside the attack window and the change times fall inside the synchronisation window. What does each of those two facts establish on its own?
  3. The measured run ended at exit 0. What would rsync have had to observe in order to exit anything else?
  4. If the job had run hourly instead of daily, what would have been different at 08:41?

Root cause

The destination was defined as a copy of the present

rsync -a --delete does not copy events. It makes a destination match a source: entries present only at the destination are removed, and entries whose name or content differs are replaced. Under that definition an accidental deletion, a truncation and an in-place encryption are all differences to be resolved, and resolving them is the job working correctly.

Because the destination retained no prior version, resolving the difference consumed the only surviving copy of the pre-attack state. Nine hours of exposure became total loss at 06:11:52.

The monitor watched the job result, not the change

The check exported two series: the exit status of the last run, and the age of the last successful run. Both stayed correct, because the run genuinely succeeded. Neither is a function of what the run did.

That the job succeeded does not mean the data can be restored — the two are independent, and this run is the proof. The numbers that were functions of what happened — files created, files deleted, bytes transferred — were written into the log by rsync and read by nobody. Each moved by three orders of magnitude at 06:04.

The defects compose. History without monitoring makes the incident survivable and slow; monitoring without history makes it visible and still unrecoverable. Neither leaves one recovery point, overwritten on schedule.

The hypothesis the evidence eliminates

The incident was first handled as a second intrusion — the attacker reached the mirror host too — which is the reasonable reading of an encrypted offsite copy, and it cost forty minutes.

The timestamps rule it out. rsync -a preserves modification times from the source, so the destination files carry the source’s encryption times of 03:41 to 03:58. Their inode change times are 06:04 to 06:11, set by the local kernel when the file was written and settable by no remote process. The destination bytes were written by the synchronisation, not by anything acting on the mirror host at 03:47.

Reachability confirms it. The mirror runs as a pull, so the mirror host opens the connection; the office VLAN cannot open a session to it at all, and the flow log for the night shows none.

Resolution

Stop the timer on the mirror host before anything else; the next scheduled run will re-apply the same transformation to whatever is restored beside it. Take a read-only copy of the encrypted destination — it is the best surviving record of what the source held at 03:58.

Then locate a copy the mirror never touched. Here that was the quarterly export in a separate repository, eleven days old, plus locally cached spreadsheets on four finance workstations. Recovery was assembled from those and re-keyed transactions; the loss window was eleven days, not nine hours.

Rebuild before restoring into it. Add --backup --backup-dir= pointing at a per-run directory, which rsync(1) documents as storing every displaced or deleted file on the receiving side instead of discarding it. Add --max-delete=200, which rsync(1) documents as skipping all further deletions once the limit is exceeded, warning, and exiting with error code 25 — on 3,180 deletions it stops the run.

Verification

Encrypt a file in a scratch tree on the source, let the scheduled run execute, and read the pre-change bytes back out of the run backup directory at the destination; their checksum must match the value recorded beforehand.

Delete more files than the configured ceiling, run the job, and confirm it reports skipped deletions with a non-zero status and that the destination still holds them.

Confirm the job now exports files created, files deleted and bytes transferred as separate series, that the deletion series alerts against its own recent history rather than a fixed threshold, and that firing it pages somebody. Trip it deliberately to prove it.

Confirm from the firewall that the office VLAN still cannot reach the mirror host, and that the mirror host’s key on the source is restricted to the read-only transfer command.

Prevention

Name the failure classes the copy answers, and the ones it does not. This mirror answered device, host and site loss. It was obliged to reproduce deletion, corruption and encryption.

Keep one copy the source cannot modify. Pull-based transfer protects the destination host. Only history at the destination, or a repository under a retention lock the backup credential cannot shorten, protects the destination data.

Monitor change volume, not outcome. Exit status and last-success age are liveness checks for the schedule. Files created, files deleted, bytes transferred and destination object count are the series that move before the loss becomes permanent.

Put a ceiling on destructive change and let it fail closed. One option converts an unattended catastrophic run into an alert.

Retain longer than the detection latency. Ransomware is found hours to weeks after it lands; a destination that covers one day covers no realistic detection window.