Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-monitoring-blind~50 min

The dashboard is green and the newest usable recovery point is three weeks old

Reported symptoms

  • ●A routine recovery request at 08:14 for svc-ledger as it stood on Tuesday is refused by the repository within a minute: the restore of Tuesday snapshot exits 1 after reconstructing six of the seven paths the snapshot claims and 59.401 MiB of the 60.000 MiB it records
  • ●Monday fails identically, so does Sunday, and so does every snapshot back to 2026-08-07; the first recovery point that comes back whole is the one taken on 2026-08-06, twenty-two days before the request
  • ●The backup dashboard has read Backup health 100% every morning of those twenty-two days, including the morning of the request, and no alert has fired against the estate in that window
  • ●The nightly backup job on rbdr-backup-02 has exited 0 every night throughout, and a snapshot exists for every one of those nights with the expected paths and the expected size
  • ●The first hypothesis on the call is that the backups quietly stopped some weeks ago and the dashboard is reporting a stale scrape
  • ●Nobody can state when a recovery point for this system was last restored and checked against the source, and the runbook does not record it
  • ●The same failure reproduces on a second host with an independent restore target, so this is not a fault on the machine the restore was attempted from

Evidence

  • · Three ages computed per system from the repository itself - recovery point age, verification age and restore-proof age - report 9h, 22d and 118d for rbdr-db-01, and across 214 systems the newest recovery point is at most 27h old while verification age is at least 21d on all 214 and no restore proof has ever been taken for 61 of them
  • · The dashboard displays one number for the whole estate, defined as the percentage of backup jobs that exited 0 in the last 24 hours; on the morning of the request that ratio was 214 of 214, which is a true statement and the only statement the metric is capable of making
  • · The weekly restic check --read-data job on rbdr-backup-02 exited 1 on 2026-08-09, on 2026-08-16 and on 2026-08-23, writing each failure to /var/log/rbdr/verify.log; its exit status is not an input to the dashboard metric, which is defined over backup jobs only
  • · The verification output names one pack and one blob every time, and the identifier restic recomputed from the bytes it read is not the name the pack is filed under
  • · restic snapshots lists a snapshot for every night in the window with the expected tags, paths and sizes, so the recovery points exist and are the expected shape
  • · The failing restores all print ignoring error for the same path, naming the same blob in the same pack, and every snapshot that references that pack fails while every snapshot that predates it restores whole
  • · The repository deduplicates, so the affected chunks were written once on the night of 2026-08-06 and every later snapshot references them rather than rewriting them
Diagnosis and resolutionclick to reveal

Root cause

Two defects. The first made twenty-two days of recovery points unusable; the second is the reason it took a recovery request to find out. **The immediate cause.** Every recovery point taken since 2026-08-07 fails verification. One data pack, written during the run that produced the 2026-08-07 snapshot, does not read back as what it is filed under, and `restic check --read-data` has said so in the same words on three consecutive Sundays. Because the repository deduplicates, the chunks in that pack were written once and are referenced by every snapshot afterwards instead of being written again, so a single unreadable pack propagated forward through every subsequent recovery point and stopped exactly at the boundary where it was created. That is why the cut line is sharp, why it falls on a date rather than on a retention interval, and why the newest usable recovery point is three weeks old rather than the oldest one being lost. Whether the pack was damaged in flight or after it was written cannot now be established, and that is the second defect showing through the first: nothing read the repository's data between the write and the restore request except a job whose result nobody received. **The control that should have caught it.** The estate's only backup instrument is a dashboard displaying the percentage of backup jobs that exited 0 in the last 24 hours. That number was 100% throughout, and it was correct throughout. A job exit status records that a process ran and returned zero. It cannot express how old the newest recovery point is, how long it has been since a recovery point passed a check that reads the stored data, or how long it has been since one was restored and compared against its source. Those three ages are the measurements that would have shown this on 2026-08-09, and none of them is derivable from the quantity the dashboard aggregates. The control did not degrade or misreport; it was answering a different question, correctly, for twenty-two days.

Remediation

Establish the cut line before deciding anything else. Restore each retained snapshot into a scratch target and record which ones complete, because the boundary between usable and unusable recovery points is the fact every subsequent decision depends on and it is cheap to measure directly. Preserve the repository before repairing it. It is simultaneously the evidence and, for every snapshot older than the cut, a working recovery source. Do not run `prune` or `forget` first: both rewrite repository state, and both make the damage permanent and unattributable. Recover the affected data from whichever source has the shortest gap - a second copy of the repository on independent storage if one exists and passes `restic check --read-data`, the live source if it still holds the data, or the 2026-08-06 recovery point plus whatever the application's own logs can replay forward. Then take a fresh backup so the affected chunks are written again, and verify that one by reading it. Report the exposure honestly and in the units the business uses: for twenty-two days this system had one usable recovery point, and that condition was visible in the repository the whole time. Then fix the instrument, because the repository fault will recur and the dashboard will report it the same way. Publish recovery point age, verification age and restore-proof age per system, and alert on each of the three separately.

Verification

`restic check --read-data` on the repaired repository exits 0 and prints `read all data` with the full pack count, not merely the snapshot count. A restore of the newest recovery point into a scratch target exits 0, restores the full file count and the full byte count, emits no `ignoring error for` lines, and the restored tree is compared against reference digests taken from the source with `md5sum -c` exiting 0. The restore exit code and the digest comparison are read together; neither is accepted alone. Every retained snapshot is restored and checked, not only the newest, because packs are shared between snapshots and one passing restore proves only that one snapshot avoids the damaged pack. The new dashboard is verified by breaking it deliberately: stop the verification job on one system and confirm that system's verification age climbs and alerts, while the backup jobs continue to exit 0. An instrument that stays green through that test has the defect this incident was about. A reader who has never seen the estate can open the dashboard and state, for any named system, how old its newest recovery point is, how long since one passed a data-reading check, and how long since one was restored and compared.

Prevention

**Report three ages per system, not one percentage per estate.** Recovery point age, verification age and restore-proof age answer three different questions, and no aggregate over job exit statuses answers any of them. A single number across 214 systems also hides the distribution: 214 of 214 green is the same pixel as 213 of 214. **Alert on the age of the last successful data-reading check.** The weekly job failed three times in a row and the estate learned nothing, because its exit status went to a log file rather than to an instrument. An age-based alert would have fired on 2026-08-09. **Make the verification job's result a first-class signal.** A control whose output nobody receives is not a control; it is a record that can be read afterwards to establish how long the problem was visible. **Measure restore-proof age, and accept that "never" is a value it can take.** Sixty-one systems in this estate had never had a recovery point restored and compared. That is not a monitoring gap to be filled later - it is the number the dashboard most needed to show. **Remember that a green job says the job ran.** It does not say a recovery point is readable, and in a deduplicating repository it does not even say the recovery point is independent of the last one. **Instrument the boundary, not just the newest point.** Because deduplication carries chunks forward, one bad pack makes every later recovery point unusable at once. Testing only the newest snapshot finds this; testing only that a snapshot exists never will.

Reported symptoms

At 08:14 finance asks for svc-ledger as it stood on Tuesday. The repository on rbdr-backup-02 refuses within a minute: Tuesday’s snapshot restores six of its seven paths and exits 1.

Monday fails the same way. So does Sunday, and every snapshot back to 2026-08-07. The first one that comes back whole is 2026-08-06 — twenty-two days before the request.

The dashboard has read Backup health 100% every one of those mornings, including this one. Nothing has alerted. The call opens on the theory that the backups stopped weeks ago and the scrape is stale.

Evidence provided

The first useful move is to stop asking the dashboard and compute the ages from the repository.

Read-only / Safethree ages per system, next to the one number the estate was watching
$ rbdr-age-report --repo /srv/rbdr-repo-prod
system            recovery_point_age   verification_age   restore_proof_age
rbdr-db-01                        9h                22d                118d
rbdr-ledger-01                    9h                22d                  --
rbdr-app-01                      11h                22d                 84d

214 systems  newest recovery point at most 27h old
           verification age 21d or greater on 214 of 214
           restore proof never taken on 61 of 214

dashboard    Backup health 100%  (214/214 backup jobs exited 0, last 24h)

Illustrative output

The dashboard’s number is one expression, and the estate has never had another:

backup_health_percent =
  100 * count(backup_job_last_exit_code == 0) / count(backup_job_last_exit_code)
  over the last 24 hours, across all systems

The recovery points are present and are the expected shape. A listing proves it:

Read-only / Safethe recovery points exist, with the expected tags, paths and sizes
$ restic snapshots
ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
3e349a12  2026-08-28 13:27:03  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
2 snapshots

The weekly verification job has been saying the rest of it since 2026-08-09, into /var/log/rbdr/verify.log, in the same words each Sunday.

Data-loss riskthe weekly job that reads the packs, and what it has printed three times
$ restic check --read-data
[0:00] 100.00%  2 / 2 snapshots
read all data
pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f contains 2 errors: [blob 9a6d59cfa25fce43e433aff4b16bb04d240c3730b027ae677bcb15719e44a436: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed unexpected pack id 39e18fa9bf2f4acad3899cbe025e82aad203d9cfa1ee30017b1667a8d1e9bdc7]
[0:00] 100.00%  7 / 7 packs
Service impact possiblewhat every snapshot after the cut line does when restored
$ restic restore 3e349a12 --target /srv/rbdr-restore-check
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors

Snapshots taken on or after 2026-08-07 fail on that path. Snapshots taken before it restore whole.

Work the evidence before reading on

  1. The dashboard says 100% and twenty-two days of recovery points are unusable. Which of those two statements is false?
  2. Write down what a backup job’s exit code can and cannot tell you. Then read the three ages again and say which of them it could produce.
  3. The cut falls on a date, not on a retention boundary. What does the repository do that would put a boundary there?
  4. The call’s theory is that backups stopped. Which single piece of evidence above ends that theory?
  5. The verification job failed on 2026-08-09. Where did that result go, and who was supposed to receive it?

Root cause

One unreadable pack, carried forward by deduplication into every later recovery point

A data pack written during the run that produced the 2026-08-07 snapshot does not read back as the name it is filed under. Repository objects are stored under a name derived from a hash of their contents, so unexpected pack id 39e18fa9… is restic saying the bytes it read no longer hash to 2c3be6d1c7….

That alone would cost one night. What made it cost twenty-two is deduplication. The repository stores each chunk once; a later backup adds only the chunks that changed, and the capture measures the effect — a second backup of 60 MiB added 2.062 KiB. Every snapshot after 2026-08-07 therefore references the affected chunks rather than rewriting them, so one bad pack propagated forward through every recovery point taken since and stopped precisely at the boundary where it was created.

Whether the pack was damaged in flight or afterwards can no longer be established, and that is the second defect showing through the first: nothing read the repository’s data between the write and this morning except a job whose result nobody received.

A percentage of job exit statuses cannot express an age

The dashboard was not broken, stale or misconfigured. It reported that 214 of 214 backup jobs exited 0 in the last 24 hours, which was true every morning.

Resolution

Establish the cut line first. It is cheap to measure and every later decision depends on it.

export RESTIC_REPOSITORY=/srv/rbdr-repo-prod
RBDR_OUT=/srv/rbdr-restore-check
mkdir -p "$RBDR_OUT"
while IFS= read -r RBDR_SNAP; do
  if restic restore "$RBDR_SNAP" --target "$RBDR_OUT/$RBDR_SNAP" > /dev/null 2>&1; then
    printf '%s restored\n' "$RBDR_SNAP" >> "$RBDR_OUT/rbdr-cut-line.txt"
  else
    printf '%s FAILED\n' "$RBDR_SNAP" >> "$RBDR_OUT/rbdr-cut-line.txt"
  fi
done < "$RBDR_OUT/rbdr-snapshot-ids.txt"

Copy the repository before repairing it. Do not run prune or forget first: both rewrite repository state, and both make the damage permanent and unattributable.

Recover from whichever source has the shortest gap — a second copy on independent storage that passes restic check --read-data, the live source if it still holds the data, or the 2026-08-06 point plus whatever the application’s own logs replay forward. Then take a fresh backup so the affected chunks are written again, and read that one back.

Report the exposure in the units the business uses: for twenty-two days this system had one usable recovery point, and the condition was visible in the repository throughout.

Verification

restic check --read-data on the repaired repository exits 0 and prints read all data with the full pack count, not merely the snapshot count.

The restore then exits 0 with the full file and byte counts and no ignoring error for lines, and the restored tree is compared against reference digests taken from the source:

Read-only / Safewhat a usable recovery point looks like on the same comparison
$ md5sum -c rbdr-reference.md5
  ./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK
>>> md5sum -c exit code: 0

Read the exit code and the digest result together, never one without the other, and repeat for every retained snapshot: packs are shared, so one passing restore proves only that one snapshot avoids the damaged pack.

Then verify the replacement instrument by breaking it on purpose. Stop the verification job on one system and confirm that system’s verification age climbs and alerts while its backup jobs keep exiting 0. A dashboard that stays green through that test has the defect this incident was about.

Prevention

Report three ages per system, not one percentage per estate. Recovery point age, verification age and restore-proof age answer three different questions, and no aggregate over exit statuses answers any of them.

Alert on the age of the last successful data-reading check. The weekly job failed three times and the estate learned nothing, because its result went to a log file rather than to an instrument.

Measure restore-proof age, and let “never” be a value it can hold. Sixty-one systems here had never had a recovery point restored and compared. That is the number the dashboard most needed to show.

A green job says the job ran. It does not say a recovery point is readable, and in a deduplicating repository it does not say the newest point is independent of the last one.

Test the boundary, not only the newest snapshot. Deduplication carries chunks forward, so one bad pack invalidates every later recovery point at once — and confirming that a snapshot exists will never find it.