Demonstrate rsync deletion and ransomware propagation

Objective

You are going to build the most common “backup” in production — a scheduled rsync -a --delete to another directory — and then destroy the data it holds twice, without a single command failing or an exit code other than 0.

Then you will fix it: the same job with --backup --backup-dir, both injuries repeated, and the casualties restored from the retained history and checked against a baseline recorded before anything happened. The lab ends with a verified restore and a measured restore time.

Architecture

flowchart TD
    S["rbdr-src/\ninvoice.txt orders.csv reports/q3.txt"] -->|"rsync -a --delete\nthe scheduled run, exit 0"| M["rbdr-mirror/\ndefined as: whatever the source holds now"]
    E1["day 2: rm invoice.txt"] --> S
    E2["day 3: openssl enc in place, rename .locked"] --> S
    M --> Q{"is the pre-injury state\nanywhere at the destination?"}
    Q -->|"plain --delete"| N["NO. One interval after the injury,\nplaintext copies remaining: 0"]
    Q -->|"--backup --backup-dir"| H["rbdr-history/RUN/\ndisplaced files retained, restorable"]

The two events differ; the destination does not distinguish them. The only thing that changes the outcome is whether it may keep what it displaced.

Requirements

• A Linux host with rsync and openssl. The capture behind this lab ran on Ubuntu 26.04 LTS, kernel 7.0.0-29-generic, with rsync 3.4.1 (protocol version 32) and OpenSSL 3.5.5.
• A writable home directory. No root, no daemon, no second host. Everything is created under $HOME/rbdr-lab-04, and every name carries the rbdr- prefix so cleanup can be asserted.
• One shell for the whole lab, because later tasks use variables set in Task 1. The capture used the shorter names src/ and mirror/, so the transcript output shows src/... where your run shows rbdr-src/....

Scenario

An estate runs one job from a crontab entry at 01:00: rsync -a --delete from a production directory to a directory on another host. It has run cleanly for two years, and monitoring that watches job completion has never fired.

On day two an operator deletes a file that is still needed. On day three the source is encrypted in place. Nobody touches the job, the destination or the schedule. You will reproduce both nights.

Tasks

Task 1 — Record the pre-lab state and build the source tree

BASE="$HOME/rbdr-lab-04"
SRC="$BASE/rbdr-src"
MIRROR="$BASE/rbdr-mirror"
HISTORY="$BASE/rbdr-history"

mkdir -p "$SRC/reports" "$MIRROR" "$HISTORY"

# Pre-lab state. Cleanup diffs against this file, so cleanup is provable.
ls -A "$HOME" | grep -v '^rbdr-' | sort > "$BASE/rbdr-prestate.txt"

printf 'invoice 1001 due 2026-09-30\n'   > "$SRC/invoice.txt"
printf 'order_id,amount\n1,100\n2,250\n' > "$SRC/orders.csv"
printf 'q3 revenue summary\n'            > "$SRC/reports/q3.txt"

( cd "$SRC" && find . -type f | sort )
( cd "$SRC" && md5sum invoice.txt orders.csv reports/q3.txt ) \
  | tee "$BASE/rbdr-baseline.md5"
rsync --version | head -1

$ find src -type f | sort; md5sum src/orders.csv

--- 09:00  source tree ---
src/invoice.txt
src/orders.csv
src/reports/q3.txt
orders.csv md5: 9eb4e2ad8e08e1dcaaf87ababab964b0

rbdr-baseline.md5 is the only record of what the ledger contained before the lab starts. Every restore claim below is checked against it, not against memory.

Task 2 — Night 1: establish the mirror and prove it is faithful

rsync -a --delete "$SRC/" "$MIRROR/"
echo "rsync exit: $?"

( cd "$MIRROR" && find . -type f | sort )
diff -r "$SRC" "$MIRROR" && echo "OK mirror is byte-identical to the source"

$ rsync -a --delete src/ mirror/

  mirror/invoice.txt
mirror/orders.csv
mirror/reports/q3.txt
the mirror is a faithful copy. So far this looks like a backup.

The capture’s own note is the trap in seven words: So far this looks like a backup. Every check passes — diff -r is silent, a restore is a cp, and if the source disk died tonight this directory would answer the incident in minutes. That much is a real control. The error is stopping the sentence at “we have a copy” instead of finishing it with “of the current state”.

Task 3 — Day 2: one deletion, one scheduled run, and a failed restore

rm "$SRC/invoice.txt"

rsync -a --delete "$SRC/" "$MIRROR/"     # the scheduled 01:00 run
echo "rsync exit: $?"

( cd "$MIRROR" && find . -type f | sort ) | tee "$BASE/rbdr-propagation.txt"

# The failing case: try to restore the deleted file from the mirror.
cp "$MIRROR/invoice.txt" "$SRC/invoice.txt"
echo "restore-from-mirror exit: $?"

$ rsync -a --delete src/ mirror/

--- mirror now ---
mirror/orders.csv
mirror/reports/q3.txt
invoice.txt recoverable from the mirror? NO - the mirror deleted it too

Elapsed time between the mistake and the loss of the only other copy:
one scheduled interval. Nobody had to make a second mistake.

The cp fails with No such file or directory and exit status 1. That failure is the lab’s proof that the check has teeth: the same restore drill run before 01:00 would have succeeded.

The capture’s elapsed-time note names the real exposure window. The mistake alone did not cost the data; the mistake plus the interval did. Shortening that interval — the reflex from availability work — makes this case worse, because it is the only period in which the destination still holds the pre-injury state.

Task 4 — Day 3: encrypt every source file in place

PASS='rbdr-not-a-real-ransom-key'

find "$SRC" -type f ! -name '*.locked' -print0 |
while IFS= read -r -d '' f; do
  openssl enc -aes-256-cbc -pbkdf2 -salt -pass pass:"$PASS" \
    -in "$f" -out "$f.locked"
  rm -- "$f"
done

( cd "$SRC" && find . -type f | sort )

rsync -a --delete "$SRC/" "$MIRROR/"     # the same scheduled run
echo "rsync exit: $?"

( cd "$MIRROR" && find . -type f | sort ) | tee -a "$BASE/rbdr-propagation.txt"
od -c "$MIRROR/orders.csv.locked" | head -1 | tee -a "$BASE/rbdr-propagation.txt"
find "$SRC" "$MIRROR" -name 'orders.csv' | wc -l

$ rsync -a --delete src/ mirror/

--- source after the attack ---
src/orders.csv.locked
src/reports/q3.txt.locked
--- mirror after that run ---
mirror/orders.csv.locked
mirror/reports/q3.txt.locked

To rsync this was two deletions and two creations, and the destination was obliged to match. Whether the mirrored ledger is still business data is settled by its first bytes.

$ od -c mirror/orders.csv.locked

--- is the mirrored ledger still readable business data? ---
0000000   S   a   l   t   e   d   _   _   < 370 351 357 256   N   J   t

plaintext copies of the ledger remaining anywhere: 0

Salted__ is an eight-byte header written by openssl enc when a salt is in use, not a ledger. The MD5 recorded at 09:00 now describes a file that exists nowhere, and the capture closes with the sentence to carry out of this lab: the mirror did exactly what it was configured to do, on schedule, with exit code 0, and it did it to the only other copy of the data.

Task 5 — Rebuild the job so the destination keeps what it displaces

Reset to the day-one state and run the same mirror with --backup in effect. rsync then renames destination files it is about to replace or delete instead of overwriting or removing them, and --backup-dir collects them in a parallel hierarchy rather than beside the originals.

rm -rf "$SRC" "$MIRROR"
mkdir -p "$SRC/reports" "$MIRROR"
printf 'invoice 1001 due 2026-09-30\n'   > "$SRC/invoice.txt"
printf 'order_id,amount\n1,100\n2,250\n' > "$SRC/orders.csv"
printf 'q3 revenue summary\n'            > "$SRC/reports/q3.txt"

RUN1="$HISTORY/run-1"
rsync -a --delete --backup --backup-dir="$RUN1" "$SRC/" "$MIRROR/"

# Repeat both injuries, then run the job again into a second history directory.
rm "$SRC/invoice.txt"
PASS='rbdr-not-a-real-ransom-key'
find "$SRC" -type f ! -name '*.locked' -print0 |
while IFS= read -r -d '' f; do
  openssl enc -aes-256-cbc -pbkdf2 -salt -pass pass:"$PASS" \
    -in "$f" -out "$f.locked"
  rm -- "$f"
done

RUN2="$HISTORY/run-2"
rsync -a --delete --backup --backup-dir="$RUN2" "$SRC/" "$MIRROR/"
echo "rsync exit: $?"

( cd "$MIRROR" && find . -type f | sort )
( cd "$RUN2" && find . -type f | sort )

The mirror listing is unchanged from Task 4. $RUN2 is the difference: it holds invoice.txt, orders.csv and reports/q3.txt as they stood before the run that displaced them. The injury still propagated on schedule; it no longer consumed the previous state doing it.

Task 6 — Restore from the history and validate the restored bytes

START=$SECONDS

RESTORE="$BASE/rbdr-restore"
mkdir -p "$RESTORE/reports"
cp "$RUN2/invoice.txt"      "$RESTORE/invoice.txt"
cp "$RUN2/orders.csv"       "$RESTORE/orders.csv"
cp "$RUN2/reports/q3.txt"   "$RESTORE/reports/q3.txt"

# Redirect rather than pipe, so $? is md5sum's status and not tee's.
( cd "$RESTORE" && md5sum -c "$BASE/rbdr-baseline.md5" ) > "$BASE/rbdr-restore.txt" 2>&1
echo "md5sum -c exit: $?"
cat "$BASE/rbdr-restore.txt"

head -1 "$RESTORE/orders.csv"
echo "restore seconds: $(( SECONDS - START ))" | tee -a "$BASE/rbdr-restore.txt"

md5sum -c prints OK for all three files and exits 0. That — not the presence of the files — is what makes this a restore rather than a copy.

Task 7 — Who can reach the history

History defends against mistakes, not against an adversary who can reach it — and in a push arrangement the compromised source holds the credentials that do.

# In a push arrangement the source can write the whole destination tree,
# history included. Demonstrate the reachability locally:
test -w "$HISTORY" && echo "the identity running the job can write the history"
test -w "$MIRROR"  && echo "the identity running the job can write the mirror"

Reversing the direction is the structural fix: the backup host connects to the source and writes into storage the source holds no credentials for, so nothing on the source can propagate to a destination it cannot address. sshd(8) documents the authorized_keys options that make that inbound direction safe to leave open — command forces one fixed program whatever the client asks for, restrict disables forwarding and PTY allocation, and from limits which address may use the key.

restrict,from="198.51.100.7",command="/usr/local/bin/rbdr-source-reader" ssh-ed25519 AAAA... backup@vault

State the trade rather than glossing it: pull inverts the trust relationship, so a compromise of the backup host reaches every source it can read. That host is therefore hardened and monitored differently from the fleet it protects.

Validation

Run each line and compare against the stated string and exit code.

# 1. exits 0, prints exactly: 3
grep -c . "$BASE/rbdr-baseline.md5"

# 2. exits 0, prints: OK deletion propagated
grep -q '^\./invoice\.txt$' "$BASE/rbdr-propagation.txt" \
  || echo "OK deletion propagated"

# 3. exits 0, prints: OK ciphertext propagated
grep -q 'orders\.csv\.locked' "$BASE/rbdr-propagation.txt" \
  && echo "OK ciphertext propagated"

# 4. exits 0, prints: OK Salted__ header present
grep -q 'S   a   l   t   e   d   _   _' "$BASE/rbdr-propagation.txt" \
  && echo "OK Salted__ header present"

# 5. exits 0, prints exactly: 3
find "$RUN2" -type f | wc -l

# 6. exits 0, prints three lines ending ": OK"
( cd "$BASE/rbdr-restore" && md5sum -c "$BASE/rbdr-baseline.md5" )

# 7. exits 1, prints: cp: cannot stat ... No such file or directory
cp "$MIRROR/invoice.txt" /dev/null

Checks 2 and 7 are the failing cases. Check 2 passes only because ./invoice.txt is absent from the post-deletion mirror listing, and a cp that succeeds at check 7 means the mirror never ran with --delete.

Expected Outcome

One correctly configured job destroyed the last usable copy twice, exiting 0 each time; the same job with two extra flags retained everything needed to put it back.

• Actual RPO observed: one scheduled interval — the mirror job’s own period, and nothing else. The capture records the elapsed time between the mistake and the loss of the only other copy as one scheduled interval. Nobody had to make a second mistake. In the scenario that is 24 hours because the crontab entry says 01:00; it is a property of the schedule, never of rsync.
• Actual restore time: printed by Task 6 as restore seconds:. Record your own number — the capture measured no restore, so none is quoted here.
• Restore validated: md5sum -c returned OK for all three files against checksums recorded in Task 1, before any injury.

Troubleshooting

Cleanup

cp "$BASE/rbdr-prestate.txt" /tmp/rbdr-prestate.txt
rm -rf "$BASE"

ls -A "$HOME" | grep -v '^rbdr-' | sort > /tmp/rbdr-poststate.txt
diff /tmp/rbdr-prestate.txt /tmp/rbdr-poststate.txt \
  && echo "OK home matches the state recorded in Task 1"

ls -d "$HOME"/rbdr-* 2>/dev/null \
  && echo "FAIL rbdr- objects remain" \
  || echo "OK no rbdr- objects remain"

rm -f /tmp/rbdr-prestate.txt /tmp/rbdr-poststate.txt

The diff must be silent and both OK lines must print. Because every object carried the rbdr- prefix, the second check is an assertion, not a hope.

Production notes

• Write two lists beside every mirror job: the failure classes it answers, and the classes needing something else. This one is strong against device, host and site loss and has no defence against deletion, truncation, silent corruption or encryption, because it is obliged to reproduce them.
• Alert on the age of the oldest recoverable version, not on job completion. Both injuries here ran green.
• Decide the retention of a --backup-dir hierarchy at the same time you add the flag. Unpruned history is how this remediation becomes an outage of its own.
• Where the destination filesystem supports snapshots, taking one before the run is cheaper on write-heavy trees. The ordering is the whole control: a snapshot taken after the run preserves the damage instead of the data.

What You Learned

• A destination defined as “whatever the source holds now” reproduces every logical injury to the source, one scheduled interval later, with no defect anywhere in the arrangement.
• Both injuries exited 0, so monitoring that watches job success stayed green through both losses.
• The interval is the exposure window, not the safety margin. Shortening it improves the physical-loss case and degrades the logical one.
• --backup --backup-dir supplies the missing input — the destination’s own past — paid for in capacity and retention work, and it is what made the Task 6 restore possible at all.
• A restore is proved by checksums, not by file presence. md5sum -c against a pre-incident baseline is the difference between a copy and a recovery.
• Direction decides blast radius. In a push arrangement the compromised source reaches the history, the snapshots and the pruning job; in a pull arrangement it holds no route to any of them.