Mirror propagation: deletion, corruption and ransomware

Having separated what rsync is — a program that makes a destination match a source — from what it is not, the remaining question is what that definition costs when the source has a bad day. The answer requires no defect anywhere. A three-night capture on rsync 3.4.1 shows a correctly configured mirror consume the only surviving copy of a file twice, for two entirely different reasons, without a single command failing.

Night 1: a faithful copy is exactly what makes it feel like a backup

The source tree in the capture is deliberately dull: three files under src/, one a ledger called orders.csv whose MD5 is recorded at 09:00 as 9eb4e2ad8e08e1dcaaf87ababab964b0. The job is the one almost every estate has somewhere — rsync -a --delete from production to a directory on another host, at 01:00.

$ rsync -a --delete src/ mirror/

  mirror/invoice.txt
mirror/orders.csv
mirror/reports/q3.txt
the mirror is a faithful copy. So far this looks like a backup.

The capture’s own note — “So far this looks like a backup” — is the whole problem in seven words. Every property a reader checks on night one holds: names match, contents match, and a restore is a cp away with no repository to walk and no chain to reassemble. If the production disk dies tonight, this directory answers the incident in minutes.

That is a real control. The error is stopping the sentence at “we have a copy” instead of finishing it with “of the current state” — the mirror is a function of one input, whatever the source contains when the job runs, so nothing at the destination records yesterday.

Day 2: the elapsed time was one scheduled interval

On day two an operator removes src/invoice.txt — an ordinary mistake, made once, of the kind that happens in every estate every month. No second mistake follows: nobody touches the mirror, edits the job or logs into the destination host. The schedule arrives.

$ rsync -a --delete src/ mirror/

--- mirror now ---
mirror/orders.csv
mirror/reports/q3.txt
invoice.txt recoverable from the mirror? NO - the mirror deleted it too

Elapsed time between the mistake and the loss of the only other copy:
one scheduled interval. Nobody had to make a second mistake.

The elapsed-time note is the line to read slowly, because it names the real exposure window: the mistake alone did not cost the data, the mistake plus the interval did, and the interval came from a schedule nobody chose with this scenario in mind.

This is where an intuition built on availability engineering points exactly the wrong way. There a shorter interval is strictly better — less lag, less to lose when the primary dies. Against a logical injury it is strictly worse, because it shortens the only period during which the destination still holds the pre-injury state. Nightly gives an operator a day to notice; a continuous stream gives them seconds.

The arithmetic is worth doing once, because it is the only place a number in this lesson is allowed to come from. Take a stated architecture: one mirror, no history at the destination, a fixed interval T, and an injury landing at an arbitrary moment inside it. The destination then holds the pre-injury state for whatever remains of T and for nothing after that, so the window in which the estate can still recover averages half an interval and can never exceed one. Change the architecture to retain K dated directories of displaced files and the same window becomes roughly K intervals, because the injury now has to age out of a history rather than out of a single run. Both figures fall out of the schedule and the retention count. Neither is a property rsync supplies, and neither survives being quoted without the assumptions that produced it.

Notice also what is absent from the transcript: an error. The run did what --delete exists to do, so monitoring that watches job completion saw a healthy run, because it was one. The signal that would have separated the three nights is not the exit status but a property of the run itself — how many destination entries it removed. Night one removed nothing, because there was nothing at the destination to remove; day 2 removed one; day 3 removed two and created two. A monitor recording those counts would have shown a step change on both incident nights while the job went on exiting 0. That is a different instrument from the one most estates own, and it has to be built deliberately, because no transfer tool volunteers a metric whose whole purpose is to make its own successful run look alarming.

Day 3: Salted__ is where the ledger used to be

On day three the source is not deleted but rewritten: every file encrypted in place and renamed .locked, leaving src/orders.csv.locked and src/reports/q3.txt.locked. To rsync that is two deletions and two creations, and the destination must be made to match.

$ rsync -a --delete src/ mirror/

--- mirror after that run ---
mirror/orders.csv.locked
mirror/reports/q3.txt.locked

The mirror now holds a perfect copy of ciphertext. Whether that is still business data the capture settles by reading the ledger’s first bytes.

$ od -c mirror/orders.csv.locked

--- is the mirrored ledger still readable business data? ---
0000000   S   a   l   t   e   d   _   _   < 370 351 357 256   N   J   t

plaintext copies of the ledger remaining anywhere: 0

Salted__ is an eight-byte header, not a ledger, and the MD5 recorded at 09:00 on day one now describes a file that exists nowhere. The capture’s count — plaintext copies of the ledger remaining anywhere: 0 — was reached by a job that behaved correctly on all three nights, and the transcript ends with the sentence worth carrying out of this lesson: “The mirror did exactly what it was configured to do, on schedule, with exit code 0, and it did it to the only other copy of the data.”

Giving the destination a memory: --backup-dir, snapshots, versions

The cheapest change is to stop letting the destination discard anything. With --backup in effect, rsync renames destination files about to be replaced or deleted rather than overwriting or removing them, and --backup-dir collects them in a parallel hierarchy rather than beside the originals with a suffix.

SRC=/srv/data/
DEST=/srv/mirror/
HISTORY=/srv/mirror-history

# Whatever the destination would replace or delete is renamed under $HISTORY.
rsync -a --delete \
  --backup --backup-dir="$HISTORY/$(date -u +%Y-%m-%dT%H%M)" \
  "$SRC" "$DEST"

Against the day-2 event that arrangement still removes invoice.txt from mirror/, but the file lands in the dated directory, where it can be copied back; against day 3 it retains the plaintext orders.csv the encrypted version displaced. The cost is capacity and pruning. The history grows with the churn of the source rather than its size, so a tree under heavy daily rewrite can outgrow the tree it protects, and nothing in the command above prunes a dated directory once it has been written. A --backup-dir mirror with no separate retention job is a capacity incident on a delay fuse.

The second change moves the history under the destination filesystem instead of into the transfer: the destination snapshots the mirror directory before the run, then lets the run proceed unchanged. On write-heavy trees that is cheaper, because copy-on-write stores changed blocks rather than whole displaced files. It costs a filesystem that supports it, and carries the ordering rule that matters more than the tool — a snapshot taken after the run preserves the damage rather than the data.

The third replaces the flat destination with one that versions natively: an object store retaining previous versions of every key, or a repository that writes a new snapshot per run and never rewrites the old one. History becomes a property of the destination rather than the job, so nobody turns retention off by editing the rsync command. The cost is that restoring means asking the store for a version, a path that has to be exercised before an incident.

A fourth option is worth naming because it is the one operators reach for first and the one that disappoints most predictably. rsync(1) documents --max-delete=NUM, which will not delete more than a stated number of files in a run, and as a blast-radius limiter it is genuine: a bulk event that would otherwise remove ten thousand entries is capped at the threshold rather than carried through in full. What the option cannot do is encode intent. The two incidents in the capture removed one file and two files respectively, so a threshold set high enough to survive an ordinary release would have passed both without comment, and one set low enough to catch them would fire on the first legitimate cleanup an application performs. It bounds how large an accident can get in a single run. It does not tell the destination which runs were accidents, which is the input the whole lesson is about.

None of these changes the interval; they change what it costs. The mirror still reproduces the injury on schedule, but no longer consumes the previous state doing it.

The direction of the connection decides who can destroy what

History alone does not close every class. On day three the source was under an attacker’s control, and if that host also initiates the transfer, holds the destination credentials and can write anywhere in the destination tree, then everything retaining the history is reachable from the compromised machine — the --backup-dir hierarchy, the snapshots, and the job that prunes them.

The structural fix is to reverse the direction. In a pull-based arrangement the backup host connects to the source and writes into storage the source has no credentials for; the source cannot enumerate, modify or delete the destination, because it never holds a session on it. sshd(8) documents the authorized_keys options that make that inbound direction safe to leave open: command forces one fixed program whatever the client requests, restrict disables forwarding and PTY allocation, and from limits which address may use the key.

restrict,from="198.51.100.7",command="/usr/local/bin/backup-source-reader" ssh-ed25519 AAAA... backup@vault

The reasoning behind that inversion is not rsync’s, and it is older than any of these tools. NIST SP 800-184 frames cybersecurity event recovery around the case in which the production environment itself is the thing that can no longer be trusted, and once that is the premise the conclusion is mechanical: every control the compromised host can reach is a control the incident already owns. The --backup-dir hierarchy, the snapshot schedule, the pruning job and the --max-delete threshold are all controls of exactly that kind while the source holds the session, because each of them lives somewhere the source has a credential for. Reversing the direction is the only change in this lesson that moves them outside that reach rather than merely making them larger.

The trade should be stated rather than glossed. Pull inverts the trust relationship, so a compromise of the backup host reaches every source it can read, which is why that host is hardened and monitored differently from the fleet it protects. It also moves work: someone has to hold an inventory of what the backup host is entitled to read, keep the from addresses accurate as the estate changes, and notice when a source stops answering — none of which the source-initiated arrangement required, because a source that stopped running the job simply stopped appearing. What the inversion buys is the property the capture makes unarguable: an event on the source, of any kind, cannot propagate to a destination the source cannot address.

What to take from this

• Night one produced a faithful copy — the transcript’s words are “the mirror is a faithful copy. So far this looks like a backup.” Fidelity to the present is what makes a destination without history feel safe.
• After one deletion the day-2 run answered invoice.txt recoverable from the mirror? NO - the mirror deleted it too, and the capture records the elapsed time as one scheduled interval, with nobody making a second mistake.
• After in-place encryption the mirrored ledger began Salted__ and the capture’s closing count was plaintext copies of the ledger remaining anywhere: 0. The day-one MD5 9eb4e2ad8e08e1dcaaf87ababab964b0 now describes a file that exists nowhere.
• Nothing failed. On rsync 3.4.1 the job “did exactly what it was configured to do, on schedule, with exit code 0” all three nights, so monitoring that watches job success stayed green through both losses.
• The one scheduled interval in the capture is the only period in which the destination still holds the pre-injury state, so shortening it improves the physical-loss case and degrades the logical one.
• --backup-dir, snapshot-then-mirror and versioned destinations each supply the missing past: any of them would have kept the invoice.txt the day-2 run removed and the plaintext orders.csv that day 3 replaced with Salted__, paid for in capacity and retention work. Pull-based transfer answers the separate question of whether the day-3 source can reach that history at all.

Cross-course references

• Ceph & Distributed Storage for Production Sysadmins — Part XXIII (Replication) explains how a cluster holds identical current state across failure domains, the contract this mirror signs at a smaller scale, and the reason a replicated pool answers device and host loss while answering nothing in the day-2 or day-3 transcripts above.
• Observability for Production Sysadmins — Part XX (Alert Quality) covers why a signal that never changes carries no information, exactly the monitoring failure here: the exit code stayed 0 through a deletion and an encryption event, so alerting on job success would have fired at neither.
• Linux for Production Sysadmins — Part XXXVI (Scheduled Operations) is where the interval in this lesson actually gets set, and the capture shows the schedule, not the tool, deciding how long the estate had between the mistake and the loss of its last copy.

Quiz