Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-replication-propagation~50 min

The deletion reached the second site before anyone noticed the first

Reported symptoms

  • ●At 15:40 an engineering lead reports that the 2024 acquisition dataset is missing from the project share, and the tree /srv/rbdr-projects/acq-2024 is absent from rbdr-primary
  • ●The same tree is absent from /srv/rbdr-dr/projects on the second site, which every plan in the estate describes as the place the data also lives
  • ●The operator who ran a cleanup at 14:22 removed one directory they believed was a duplicate staging copy, and expected the second site to still hold the original
  • ●The replication tile has been green all day, the agent has not failed, warned or fallen behind, and the lag series never left its normal band
  • ●A listing of the destination at 15:52 shows every current project directory and no earlier version of anything, so there is no state on that host older than the last applied change
  • ●The bridge opens on the theory that something at the second site removed the tree independently, because that site runs its own scheduled cleanup of aged project directories
  • ●The estate architecture register lists the second site under the heading Recovery with the annotation second copy of all project data, and the backup coverage report contains no repository job covering /srv/rbdr-projects at all

Evidence

  • · The replication agent journal on rbdr-dr records the unlink of acq-2024 applied at 14:22:07, four seconds after the primary audit record of the operator removal at 14:22:03, so the destination reached its post-deletion state inside one propagation interval rather than at any nightly boundary
  • · The lag series rbdr_replication_lag_seconds held between 2 and 6 seconds across the entire window and never deviated, because the deletion was propagated correctly, promptly and exactly as configured
  • · A measured propagation capture on the same rsync 3.4.1 build the agent invokes shows a mirror established as a faithful copy on the first night and the same mirror discarding an operator-deleted file on the next run, with the capture recording the file as not recoverable from the mirror and naming one scheduled interval as the elapsed time
  • · The destination volume holds exactly one copy of each path: no backup directory target, no filesystem snapshot, and no versioned object store behind it, which is why nothing on that host predates the last applied change
  • · The architecture register section titled Recovery lists rbdr-dr-site as its only entry for project data, while the section titled Availability lists the load balancer, the standby application nodes and nothing else
  • · The backup coverage report enumerates 41 protected paths and /srv/rbdr-projects is not among them; the last change to that report, dated 2025-07-14, removed the path with the comment covered by DR replication
  • · The change log kept on the second site for 2026-08-27 records no interactive session, no scheduled cleanup run and no process other than the replication receiver touching /srv/rbdr-dr/projects, and every write to that path in the whole day carries the receiver process id
Diagnosis and resolutionclick to reveal

Root cause

Two defects. The first destroyed the data in four seconds; the second is the reason there was nothing else to fall back on. The immediate cause is what continuous replication is. The agent does not copy events, it makes the destination match the source, and `rsync -a --delete` removes entries that exist only at the destination. Under that definition an operator removing a directory is a difference to be resolved, and resolving it is the agent working correctly. Because the destination retained no prior state — no backup directory target, no snapshot, no object versioning — resolving that difference consumed the only other copy. The lag series is the proof rather than the alibi: it stayed between 2 and 6 seconds, which is precisely how quickly the mistake became permanent. The control that should have caught it is the architecture register. It has a section headed Recovery and a section headed Availability, and the second site was written into the wrong one. Everything that site actually provides — a warm copy of current state, continuity when the primary is unreachable, a place to run from — is availability. Recovery requires history: the ability to produce a state the source no longer holds. The second site was never able to do that and was never asked whether it could. The misfiling was not a documentation defect that sat harmlessly in a wiki. It was read and acted upon. In July 2025 the backup coverage report dropped /srv/rbdr-projects with the comment covered by DR replication, and the estate 3-2-1 review passed afterwards by counting the replica as the second copy. One wrong heading removed the share from the backup estate and then certified the result. The defects compose. History at the destination without a corrected register would have made this incident a directory copy. A corrected register without history would have commissioned a repository. Having neither meant the share had one copy that could answer a deletion, and it was deleted at 14:22:03.

Remediation

Stop the replication agent for this share before anything else. It is the mechanism that will re-apply the deletion to whatever is placed back beside it, and while it runs no restore into the source tree is safe. Then look for a copy the agent never touched. In this estate that meant three places and none of them was the second site: the workstation of the analyst who had assembled the dataset, which held a local working copy eleven days old; the mail archive, which held the four deliverable documents as attachments; and the object store behind the reporting pipeline, which held derived extracts but not the sources. Recovery was assembled from those, and roughly nine weeks of revisions were not recovered. Rebuild the arrangement before restoring into it. Give the destination history: `rsync(1)` documents `--backup` with `--backup-dir=DIR`, which stores every displaced or deleted entry under a directory on the receiving side instead of discarding it. Add a ceiling on destructive change: `rsync(1)` documents `--max-delete=NUM`, which skips all further deletions once NUM is exceeded, warns, and exits with error code 25. Then commission an actual repository for this share, on a target the primary credential cannot rewrite, with a retention that covers the time it plausibly takes somebody to notice a directory is missing. Finally, correct the register, and correct it as a class rather than as one row. Every entry under Recovery must name the mechanism by which it can produce a state the source no longer holds. An entry that cannot answer that question moves to Availability.

Verification

Verify against the two defects, not against the agent. For history: remove a marker file from a scratch subtree on the primary, let the agent propagate the change, and read the pre-change bytes back out of the destination history directory. Their checksum must match the value recorded beforehand. A destination that cannot produce the previous state on demand has not been fixed. For the ceiling: delete more entries than the configured `--max-delete` value in the scratch subtree, let the run execute, and confirm it reports skipped deletions with a non-zero exit status while the destination still holds the entries. For the repository: restore the whole share from the new repository into an isolated path and compare it to the live tree, then delete a directory on the primary and restore only that directory from a recovery point taken before the deletion. Both must be performed, because the second is the case this incident was. For the register: re-run the estate 3-2-1 review with the corrected headings and confirm it now reports the share as having a single independent copy until the repository backfills. A review that still passes on the old count has not been re-run, it has been reprinted.

Prevention

**Classify every copy by the failure it answers, then file it under the heading that matches.** Replication answers loss of a host, a rack, a room or a site. It cannot answer deletion, corruption or encryption, because it is obliged to reproduce them. Those are two different columns and a register with one column will eventually put a replica in it. **Require every Recovery entry to name its history mechanism.** Snapshots retained for a stated window, a repository with a stated retention, versioned objects with a stated expiry. An entry that cannot name one is an availability control that has been filed in the wrong place, and this incident is what that costs. **Treat a coverage exemption as a change with a blast radius.** Removing a path from the backup estate on the strength of a sentence about another system deserves the review a production change gets, including the question of what the other system does with a deletion. **Read low replication lag as exposure, not as health.** A four-second lag is four seconds of protection against a mistake. The tighter the coupling, the more the estate depends on history existing somewhere the coupling does not reach. **Keep history at the destination even when a repository exists.** A per-run backup directory on the receiving side costs one option and converts this class of incident into a copy operation.

Reported symptoms

At 15:40 an engineering lead reports that the 2024 acquisition dataset has gone from the project share. /srv/rbdr-projects/acq-2024 is absent from rbdr-primary, and absent from /srv/rbdr-dr/projects at the second site.

The share is replicated continuously: a change-triggered agent watches the tree and invokes rsync -a --delete on the affected subtree within seconds. The tile has been green all day.

The operator who ran a cleanup at 14:22 removed one directory they believed was a duplicate staging copy, expecting the second site to hold the original. The architecture register lists that site under Recovery, annotated “second copy of all project data”. At 15:52 the destination is listed: every current project directory, and no earlier version of anything.

Evidence provided

The propagation behaviour was measured beforehand on the same rsync build the agent invokes. Its first night ends with “So far this looks like a backup” — a claim the capture does not leave standing.

Data-loss riskthe operator deletion, applied to the destination
$ rsync -a --delete src/ mirror/
--- mirror now ---
mirror/orders.csv
mirror/reports/q3.txt
invoice.txt recoverable from the mirror? NO - the mirror deleted it too

Elapsed time between the mistake and the loss of the only other copy:
one scheduled interval. Nobody had to make a second mistake.

In this estate the interval is not a night.

Read-only / Safethe receiver journal across the deletion
$ journalctl -u rbdr-replicate --since '2026-08-27 14:22:00'
14:22:03  primary  audit: operator removed projects/acq-2024 (4,118 entries)
14:22:05  rbdr-dr  change detected for subtree projects/acq-2024
14:22:07  rbdr-dr  applied: 4,118 deletions, 0 created, 0 bytes, exit 0
14:22:07  rbdr-dr  rbdr_replication_lag_seconds 4

Illustrative output

The destination holds nothing older than that applied change: no backup directory target, no snapshot, no versioned object store. The register that sent everyone there says otherwise.

Read-only / Safethe architecture register, as it stood at 14:22
$ sed -n '41,45p' /srv/rbdr-docs/rbdr-architecture-register.txt
Recovery
rbdr-dr-site      second copy of all project data
Availability
rbdr-lb-01/02     service front end
rbdr-app-03/04    standby application nodes

Illustrative output

The second site’s change log for the day records no interactive session, no cleanup run, and no writer other than the replication receiver.

Work the evidence before reading on

  1. The lag series never left its normal band. Is that a reason to exclude replication as the mechanism of loss, or a reason to include it?
  2. The second site runs its own cleanup of aged directories. Which single piece of evidence removes it, and what would it have shown had the cleanup been responsible?
  3. What would the second site have to be able to do for the Recovery heading to be correct?
  4. If the agent ran nightly instead of on change, what would have been different at 15:52 — and what would have been identical?

Root cause

The destination was defined to hold only the present

The agent does not copy events. It makes a destination match a source, and rsync -a --delete removes entries that exist only at the destination. Under that definition an operator removing a directory is a difference to be resolved, and resolving it is the agent working correctly.

Because the destination retained no prior state, resolving that difference consumed the only other copy. The mistake became permanent at 14:22:07.

The register filed the second site under the wrong heading

Everything that site provides is availability: a warm copy of current state, continuity when the primary is unreachable, somewhere to run from. Recovery requires history — the ability to produce a state the source no longer holds. The second site could never do that, and nobody asked whether it could.

The misfiling was read and acted on: in July 2025 the coverage report dropped /srv/rbdr-projects with the comment “covered by DR replication”, and the 3-2-1 review then passed by counting the replica as copy number two.

The hypothesis the evidence eliminates

The bridge opened on independent modification at the second site — reasonable, since that site runs its own cleanup of aged project directories.

Its change log eliminates it: for 2026-08-27 it records no interactive session and no cleanup run against that path, and every write there carries the replication receiver’s process id. The receiver journal supplies the positive account — 4,118 deletions applied at 14:22:07, creating nothing and transferring no bytes, four seconds after the primary audit record. A local cleanup would have left its own actor in that log, and nothing upstream four seconds earlier.

Resolution

Stop the replication agent for this share first: while it runs, anything restored into the source tree is a change to propagate, and anything placed at the destination is a difference to remove.

Then find a copy the agent never touched: here, an analyst workstation holding a working copy eleven days old and a mail archive holding four deliverables as attachments. Nine weeks of revisions were not recovered.

Rebuild before restoring into it:

RUN=$(date -u +%Y%m%dT%H%M%SZ)
HIST="/srv/rbdr-dr/projects/.rbdr-history/$RUN"
rsync -a --delete --backup --backup-dir="$HIST" --max-delete=500 \
  /srv/rbdr-projects/ /srv/rbdr-dr/projects/

rsync(1) documents --backup-dir as storing every displaced or deleted entry on the receiving side rather than discarding it, and --max-delete as skipping further deletions past the limit and exiting with error code 25. Then commission a repository on a target the primary credential cannot rewrite.

Verification

Remove a marker file from a scratch subtree on the primary, let the agent propagate, and read the pre-change bytes back out of the destination history directory; the checksum must match the value recorded beforehand.

Delete more entries than the configured ceiling and confirm the run reports skipped deletions with a non-zero exit status while the destination still holds them.

Delete a directory on the primary and restore it alone from a recovery point taken before that deletion — the case this incident was — as well as the whole share into an isolated path.

Re-run the 3-2-1 review against the corrected register.

Prevention

Classify every copy by the failure it answers. Replication answers loss of a host, a rack or a site; it cannot answer deletion, corruption or encryption, because it is obliged to reproduce them.

Require every Recovery entry to name its history mechanism — retained snapshots, a stated repository retention, versioned objects with a stated expiry. An entry that cannot name one belongs under Availability.

Treat a coverage exemption as a production change. Dropping a path from the backup estate on the strength of a sentence about another system deserves a review.

Read low lag as exposure. Four seconds of lag is four seconds in which a mistake is still undoable from the second copy.