Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-chain~50 min

The incremental chain replayed without error and produced an incomplete tree

Reported symptoms

  • ●Finance reports on 2026-08-27 that a batch import corrupted the July ledger, and asks for the file tree /srv/rbdr-orders as it stood on 2026-07-21
  • ●The restore replays the archives the repository holds for that date, exits zero, and writes 41 GiB into /srv/rbdr-restore in eighteen minutes
  • ●The nightly wrapper rbdr-restore-check runs against the restored tree and reports PASS on all four of its assertions
  • ●Finance opens the recovered tree and finds ledger-2026-07.csv absent entirely, although it was created on 2026-07-15 and was never deleted on the source
  • ●orders.csv is present, is a plausible size, opens cleanly, and is missing every row added between 2026-07-19 and 2026-07-21
  • ●app.conf carries its most recent content, which makes the tree look current and delays the suspicion that anything is missing
  • ●No error, warning or non-zero exit was recorded anywhere in the restore log, the wrapper output or the scheduler history

Evidence

  • · A reproduction of the chain mechanism using GNU tar listed-incremental archives: replaying level 0 and level 2 with level 1 absent produced a directory with the newest app.conf and a silently truncated orders.csv, and reported no error
  • · The restore log for the production job records two archives applied, rbdr-orders-2026-07-01-L0.tar and rbdr-orders-2026-07-21-L2.tar, and exit status 0
  • · The repository holds no rbdr-orders-2026-07-19-L1.tar; the prune log for 2026-08-23 records its removal as rank 5 of 5 against a level-1 keep count of 4
  • · The retention policy in /etc/rbdr-archive/retention.conf sets keep counts per level independently: 12 monthly level-0, 4 weekly level-1, 45 nightly level-2
  • · The four assertions in rbdr-restore-check are: the restore exited zero, the target directory is non-empty, seven named files are present, and the total size is within 20 percent of the previous run
  • · The file list inside rbdr-restore-check was written in March 2026 and does not name ledger-2026-07.csv, which did not exist then
  • · The manifest recorded when the level-2 archive was written lists 8,412 files including ledger-2026-07.csv, and records orders.csv at 2,304 lines
  • · sha256sum --check of the restored tree against that manifest reports orders.csv FAILED and ledger-2026-07.csv FAILED open or read
Diagnosis and resolutionclick to reveal

Root cause

Two defects, and the second is why the first survived to reach an incident. The immediate cause is a missing chain member. A level-2 archive is a delta against the level-1 that preceded it, which is a delta against the level 0. Replaying level 0 and level 2 with level 1 absent applies both archives correctly and produces a tree that never existed: files created during the level-1 window are missing altogether, and files touched in both windows return at their level-2 content, which makes the result look current rather than damaged. GNU tar does not object, because it was asked to apply the archives it was given and it did exactly that. The level-1 archive was removed by a retention policy that had permitted the removal. `/etc/rbdr-archive/retention.conf` expresses retention per level and prunes each level independently against its own keep count: 12 monthly level-0 archives, 4 weekly level-1 archives, 45 nightly level-2 archives. The three windows were sized for the cost of each tier and never intersected on paper. In practice the shortest of them governs everything: with only four weeklies retained, no level-2 archive older than about 28 days still has a parent, even though 45 of them are kept. Seventeen nights of the level-2 series were unrestorable and the policy contained no rule that could say so. The control that should have caught this is `rbdr-restore-check`, and it tested the wrong object. All four assertions describe the restored tree: exit status, non-empty directory, seven named files present, total size within 20 percent of the previous run. Not one compares the tree to a property recorded when the backup was taken. A file that comes back short is present. A file that never comes back at all is invisible to a hard-coded list written in March, before that file existed.

Remediation

Recovering the requested data is a search for a surviving parent, not a repair of the broken chain. A level-1 archive cannot be reconstructed from the archives on either side of it. Enumerate what the repository actually holds for the set and read the chain files written at backup time rather than inferring the chain from filenames: ```bash RBDR_REPO=/srv/rbdr-archive ls -1 "$RBDR_REPO"/rbdr-orders-*.tar | sort cat "$RBDR_REPO"/rbdr-orders-2026-07-21.chain ``` Every level-2 archive between 2026-07-14 and 2026-08-01 names the same missing parent, so none of them can serve the request. The nearest fully intact chain is the level-0 of 2026-08-01 with the level-1 of 2026-08-02 and the level-2 of the wanted night, which reaches back only to 2026-08-02 and is later than the corruption. For this incident the recoverable position is the level-0 archive of 2026-07-01 replayed alone, which predates the corrupted import and is internally complete. It loses three weeks of change, and that loss must be stated to the requester as a number rather than absorbed silently. Whatever is restored is then checked against the manifest recorded with the archive before anyone is told the data is back: ```bash RBDR_TARGET=/srv/rbdr-restore RBDR_MANIFEST=/srv/rbdr-archive/manifests/rbdr-orders-2026-07-01.sha256 cd "$RBDR_TARGET" && sha256sum --check "$RBDR_MANIFEST" ``` Then stop the policy from removing another parent. Raise the level-1 keep count to cover the full level-2 window before the next prune runs, and re-run the prune in its dry-run mode to confirm nothing further is scheduled for removal.

Verification

A restore is verified against something recorded at backup time. Three checks, in this order: Every member named in the chain file is present in the repository before the restore starts, and the restore refuses to run if one is missing. This is the check that turns the failure from silent into loud, and it costs one pass over a text file. `sha256sum --check` of the restored tree against the manifest written with the archive reports zero FAILED lines and zero unreadable files, and the file count in the manifest equals the file count in the tree. A short file fails its checksum; an absent file fails to open. Both are detected without anyone having remembered its name. The recorded content properties match: 2,304 lines in `orders.csv`, 8,412 files in the tree, and the newest record identifier equal to the one the manifest recorded. A structurally perfect restore of the wrong recovery point matches none of these. Finally, `rbdr-chain-audit` reports zero orphaned level-2 archives across the whole repository, which is the assertion that would have found the seventeen unrestorable nights before an incident needed one of them.

Prevention

**Retention must be expressed over chains, not over archives.** A policy that prunes each level against its own count will always be governed by its shortest window, and the shortfall is invisible because the archives it strands are still on disk and still listed. Either derive the level-1 and level-0 windows from the level-2 window, or make the prune refuse to remove any archive still named by a chain file. **Write a chain file at backup time and audit it.** The chain is knowledge the backup job has and the restore does not. A three-line text file listing the members costs nothing and turns an unrestorable recovery point into a failed audit: ```bash set -euo pipefail RBDR_REPO=/srv/rbdr-archive RBDR_CHAIN="$RBDR_REPO/rbdr-orders-2026-07-21.chain" missing=0 while read -r member; do if [ ! -f "$RBDR_REPO/$member" ]; then printf 'MISSING CHAIN MEMBER: %s\n' "$member" >&2 missing=$((missing + 1)) fi done < "$RBDR_CHAIN" [ "$missing" -eq 0 ] || exit 1 ``` **Record a manifest with every backup and verify against it.** File count, per-file checksum, and one or two content properties such as a line count or a newest-record identifier. This is the single change that converts the restore check from a plausibility inspection into a test. **Never verify a restore by listing filenames you remembered.** A hard-coded list cannot contain a file created after it was written, which is exactly the class of file a broken chain loses first. **Shorten the chain.** Three levels between a recovery point and its full is a design choice, and every additional level multiplies the number of objects that must all survive retention, media and operator error together.

Reported symptoms

On 2026-08-27 finance reports that a batch import corrupted the July ledger and asks for /srv/rbdr-orders as it stood on 2026-07-21.

The restore runs. Eighteen minutes, 41 GiB, exit status zero. The nightly wrapper rbdr-restore-check runs against the result and reports PASS on all four of its assertions.

Finance opens the tree and cannot find ledger-2026-07.csv, created on 2026-07-15 and never deleted on the source. orders.csv is there, opens cleanly, is a plausible size, and is missing every row added between 19 and 21 July. app.conf carries its most recent content, which makes the tree read as current and delays any suspicion.

Nothing anywhere reported an error.

Evidence provided

The team reproduced the mechanism in a scratch environment, using GNU tar’s listed-incremental format because the chain is explicit and one member can be removed by hand.

Data-loss risklevel 0 and level 2 replayed with level 1 missing
$ tar -xf L0.tar && tar -xf L2.tar
--- correct restore: replay L0, then L1, then L2 ---
orders.csv:
  ORDER-1001,4500.00
  ORDER-1002,1250.00
app.conf  : config v2

--- now L1 is unreadable: retention removed it, or its media failed ---
after L0 only:
  orders.csv: ORDER-1001,4500.00 
  app.conf  : config v1
skipping L1 (missing) and applying L2:
  orders.csv: ORDER-1001,4500.00 
  app.conf  : config v2

Read the last two lines together. app.conf is at its newest content and orders.csv has lost a row, which is the production symptom exactly.

Read-only / Safeeach level pruned against its own count
$ cat /etc/rbdr-archive/retention.conf
# levels are pruned independently, each against its own keep count
level0.schedule = monthly
level0.keep     = 12

level1.schedule = weekly
level1.keep     = 4

level2.schedule = nightly
level2.keep     = 45

# prune log, 2026-08-23
# level1 rbdr-orders-2026-07-19-L1.tar removed (rank 5 of 5, keep=4)

Illustrative output

Read-only / Safethe restored tree against the manifest recorded at backup time
$ sha256sum --check /srv/rbdr-archive/manifests/rbdr-orders-2026-07-21.sha256
./app.conf: OK
./orders.csv: FAILED
sha256sum: ./ledger-2026-07.csv: No such file or directory
./ledger-2026-07.csv: FAILED open or read
sha256sum: WARNING: 1 listed file could not be read
sha256sum: WARNING: 1 computed checksum did NOT match

Illustrative output

The four assertions in rbdr-restore-check are: the restore exited zero, the target directory is non-empty, seven named files are present, and the total size is within 20 percent of the previous run. The list of seven was written in March 2026.

Work the evidence before reading on

  1. Two archives were applied and the restore exited zero. What does a zero exit status from tar -x actually assert?
  2. app.conf came back current and orders.csv came back short. What distinguishes the two files, and what does that tell you about which files a chain gap loses?
  3. Each level’s keep count was chosen sensibly for its own tier. Compute the window each one delivers, and then compute the window the set delivers.
  4. All four assertions passed. What object was each one measuring, and what would each have had to compare against instead?
  5. The obvious alternative is that ledger-2026-07.csv had already been deleted on the source before the backup ran, and the restore is faithful. Which single artefact settles that, and in which direction?

Root cause

The alternative had to be eliminated first

If the source had not contained ledger-2026-07.csv on 2026-07-21, the restore would be correct and this would be an application data-loss event with a different response. The manifest recorded when the level-2 archive was written settles it: 8,412 files including ledger-2026-07.csv, and orders.csv at 2,304 lines. The source had both. The backup captured both. The restore did not produce them.

That is why the manifest is evidence and a file listing is not. A listing taken today describes today; a manifest written beside the archive describes the moment the archive was written, which is the only moment a restore can be judged against.

The retention policy expired a level nothing could replace

A level-2 archive is a delta against the level-1 before it, which is a delta against the level 0. rbdr-orders-2026-07-19-L1.tar was removed on 2026-08-23 as rank 5 of 5 against a keep count of 4.

The three windows were each sized for the cost of their own tier and never compared. Twelve monthlies reach back a year, forty-five nightlies six weeks, four weeklies four. The set delivers the shortest, so every level-2 archive older than about 28 days had already lost its parent — seventeen nights of apparently healthy recovery points that no restore could use.

The restore check inspected the tree instead of the backup

All four assertions describe the restored directory: exit status, non-empty, seven names present, size within 20 percent. None refers to anything recorded when the backup was taken, so none can detect a difference between what was captured and what came back.

A short file is present. A file absent from a hard-coded list is invisible, and that list could not have named ledger-2026-07.csv because the file did not exist when it was written. The check had reported PASS for five months and had never been capable of failing for this reason.

Resolution

A level-1 archive cannot be reconstructed from the archives on either side of it, so the work is to find a surviving chain rather than repair the broken one.

RBDR_REPO=/srv/rbdr-archive
ls -1 "$RBDR_REPO"/rbdr-orders-*.tar | sort
cat "$RBDR_REPO"/rbdr-orders-2026-07-21.chain

Every level-2 archive between 2026-07-14 and 2026-08-01 names the same absent parent, so none of them serves the request. The recoverable position is the level-0 archive of 2026-07-01 replayed alone: internally complete, earlier than the corrupted import, and three weeks short of what was asked for. That shortfall is reported to finance as a number and a date, not absorbed.

The restored tree is then checked against the manifest written with that archive before anyone is told the data is back:

RBDR_TARGET=/srv/rbdr-restore
RBDR_MANIFEST=/srv/rbdr-archive/manifests/rbdr-orders-2026-07-01.sha256
cd "$RBDR_TARGET" && sha256sum --check "$RBDR_MANIFEST"

Raise the level-1 keep count to cover the full level-2 window, and re-run the prune in dry-run mode to confirm nothing else is scheduled for removal.

Verification

The chain file for a recovery point names every member, and all of them are present in the repository before a restore starts. A missing member aborts the restore instead of producing a tree.

sha256sum --check against the backup-time manifest reports zero FAILED lines and zero unreadable files, and the manifest’s file count equals the tree’s. A short file fails its checksum; an absent file fails to open. Neither depends on anyone remembering its name.

The recorded content properties match: 2,304 lines in orders.csv, 8,412 files, and the newest record identifier the manifest recorded.

rbdr-chain-audit reports zero orphaned level-2 archives across the repository — the assertion that would have surfaced seventeen unrestorable nights before an incident needed one of them.

Prevention

Express retention over chains, not over archives. A policy that prunes each level against its own count is always governed by its shortest window, and the archives it strands remain on disk and in the listing. Derive the parent windows from the child window, or make the prune refuse to remove any archive still named by a chain file.

Write a chain file at backup time. The chain is knowledge the backup job has and the restore does not.

set -euo pipefail
RBDR_REPO=/srv/rbdr-archive
RBDR_CHAIN="$RBDR_REPO/rbdr-orders-2026-07-21.chain"
missing=0
while read -r member; do
  if [ ! -f "$RBDR_REPO/$member" ]; then
    printf 'MISSING CHAIN MEMBER: %s\n' "$member" >&2
    missing=$((missing + 1))
  fi
done < "$RBDR_CHAIN"
[ "$missing" -eq 0 ] || exit 1

Record a manifest with every backup and verify against it. File count, per-file checksum, and one or two content properties. This is the single change that turns a plausibility inspection into a test.

Never verify a restore against a list of filenames somebody remembered. Such a list cannot contain a file created after it was written, which is the first class of file a chain gap loses.

Shorten the chain. Every additional level multiplies the number of objects that must survive retention, media and operator error together for one recovery point to remain usable.