Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-catalogue~50 min

The data is all there and nothing can tell you what it is

Reported symptoms

  • ●A firmware update bricks the local RAID controller on rbdr-backup-01 at 04:12; the host is rebuilt from the configuration-management repository by 06:30, the scheduler starts cleanly, and the platform console comes up showing zero clients, zero jobs and zero snapshots
  • ●Nothing is wrong with the storage: df on the repository mount reports the same 4.1 TiB used as the figure written into the previous evening's shift handover, and all sixty-one repository directories are present
  • ●Each repository directory is named by the UUID the platform allocated when that client was registered, and nothing on the repository mount records which business system any UUID belongs to
  • ●At 09:00 finance asks for the previous Tuesday's copy of the orders system and the platform cannot answer, because which repository and which snapshot is a catalogue question and the catalogue database rbdr_catalogue no longer exists
  • ●Restore requests submitted through the platform return an empty result set rather than a storage error, which reads to the operators on the bridge like the backups are gone
  • ●The monthly coverage report has read 100 per cent for fourteen consecutive months and cannot be regenerated to check that figure, because it was generated from the same database that is missing
  • ●The first hypothesis on the bridge is that the repositories went with the array; the second is that they survived but cannot be opened, because the keys are assumed to have lived in the catalogue

Evidence

  • · restic snapshots run directly against one repository with no platform process involved lists the snapshots with their identifiers, times, originating host, tags, paths and sizes, and exits 0 — the repository describes itself
  • · restic restore of the older snapshot into a scratch target exits 0 having restored 7 files/dirs and 60.000 MiB, and md5sum -c against digests taken from the source before the repository existed exits 0 with every path reporting OK
  • · restic check --read-data against the same repository prints read all data and 100.00% 7 / 7 packs, then no errors were found, and exits 0, so the stored bytes are intact and not merely present
  • · borg list against a second repository names its archives with their timestamps and fingerprints at exit 0, and borg check --verify-data on that repository returns exit code 0
  • · The secrets manager still holds one entry per repository UUID and each entry opens its repository; the restore above was performed with a passphrase read from it, so no key was lost with the catalogue
  • · The platform job definitions, recovered from the configuration-management repository, number 61, and a recursive search of the job directory, /etc/cron.d and /etc/systemd/system for rbdr-backup-01 or rbdr_catalogue matches no file at all
  • · The coverage report query begins FROM jobs JOIN clients, so every row it is capable of emitting starts life as a job that already exists; a system with no job produces no row rather than a red row
  • · The platform documentation's install guide carries a paragraph headed Protecting the catalogue that recommends a scheduled dump to a location outside the platform; no such dump has ever been scheduled on this host
Diagnosis and resolutionclick to reveal

Root cause

Two defects, and the second one is the reason the first survived fourteen months. The immediate cause is that the catalogue was held in a separate database that nothing backed up. The platform stored its clients, jobs, retention classes, run history and the mapping from business system name to repository UUID in a PostgreSQL database on the host's local array. Every backup job the platform ran protected something else. No job named the platform host, no job named the database, and no scheduled dump existed anywhere on the machine. When the array went, the repositories were untouched — they were on a separate mount — and the only thing lost was the index. The data is all there. Nothing can tell you what it is. The control that should have caught this is the monthly coverage inventory, and it is keyed on the tool. Its rows are produced by enumerating the platform's own job table and joining out to the clients those jobs name. Every row it can emit therefore begins life as a job that already exists, which makes the report structurally incapable of reporting a system that has no job: such a system contributes no row, and an absent row reads as nothing at all rather than as a gap. The platform's own state was exactly such a system. Fourteen months of 100 per cent coverage was not a measurement of the estate; it was the job table counting itself. The report was also stored in the catalogue, so the only artefact that could have raised the question was inside the thing it failed to protect. The plausible alternative on the bridge — that the repositories were damaged or lost by the same event — is eliminated by reading them. `restic check --read-data` re-reads and re-hashes every pack rather than merely describing the repository's structure, and on this repository it printed `read all data`, `100.00% 7 / 7 packs` and `no errors were found` at exit 0. `borg check --verify-data` returned exit code 0 on the second repository. A restore then completed at exit 0 and the restored tree matched digests taken from the source. Damaged repositories do not do that.

Remediation

Mount the repository storage read-only and take an inventory before anything else. The repositories are now the only surviving record of what was backed up, and every subsequent step reads from them. Suspend all retention. `restic forget`, `restic prune` and `borg prune` exist to destroy recovery points on purpose, and with the catalogue gone the platform no longer knows which policy governs which repository. A scheduler that restarts and applies a default policy to sixty-one repositories will turn a recoverable incident into an unrecoverable one. Rebuild the index from the repositories themselves, because a repository is self-describing. The captured `restic snapshots` listing carries the snapshot identifier, the time, the originating host, the job's tag, the paths and the size — the columns the catalogue's snapshot table held. Walking every repository and recording what it reports reconstructs that table without the platform. Serve the outstanding restore request directly, out of band, from the repository the reconstruction identifies. Do not wait for the platform to be well. Recover from independent sources what the repositories cannot describe: the UUID to business-system mapping from the client registrations committed to the configuration-management repository, retention classes from the written policy, and restore-test history from the ticket system. Then state plainly what is not recoverable — the platform's own run history and its coverage attestations — rather than letting a reconstructed database imply a provenance it does not have. Finally, create the job that did not exist: a scheduled dump of the catalogue, written to a location that does not require the catalogue to find.

Verification

The rebuilt catalogue is reconciled against an independently derived list of production systems — the asset inventory, not the job table — and every UUID either resolves to a named system or is explicitly recorded as unidentified. A reconstruction checked only against itself repeats the original defect. A restore is performed from a repository chosen by the rebuilt catalogue, into a scratch target, and the restored tree is compared against reference digests. In the capture this is the step that separates a repository that lists from a repository that restores: `Restored 7 files/dirs (60.000 MiB)` at exit 0, then `md5sum -c` at exit 0 with every path reporting OK. Read the two results together. The new catalogue dump is restored into a scratch database and queried before the job is declared to work. A dump that has never been restored is a file, not a recovery point. The catalogue dump is located and read with the platform stopped and the catalogue unavailable, which is the only state in which it will ever be needed. The regenerated coverage report is driven from the asset inventory as its left side, and the number of systems with no job is a figure the report prints, not a row it omits. On first run that figure is non-zero, which is how you know the report now works.

Prevention

**Treat the backup platform as a protected system with a named owner.** Its catalogue, its configuration and its credentials are production state. The platform will not volunteer this; every platform's default view of the estate is the list of things it was told to protect. **Store the catalogue dump where it can be found without the catalogue.** A dump held only inside a repository whose UUID the catalogue is needed to resolve is a circular dependency, and it fails at exactly the moment it is wanted. **Build coverage from the asset inventory outward.** Enumerate systems from a source the backup platform does not own, then left-join the jobs. A report that iterates jobs can only ever report 100 per cent, whatever is missing. **Keep the repositories self-describing.** Set the host, the tags and the paths at backup time so that the repository listing carries them, as the capture's snapshot table does. That property is what made this incident recoverable at all. **Keep the UUID to business-system mapping in version control**, committed at client registration, so it survives independently of the platform that allocated it. **Rehearse the loss of the platform, not only of a client.** Restoring a file when the platform is healthy exercises none of the machinery this incident needed.

Reported symptoms

At 04:12 a firmware update bricks the local RAID controller on rbdr-backup-01 and takes the system array with it. The host is rebuilt from the configuration-management repository by 06:30. The scheduler starts cleanly. The console comes up showing zero clients, zero jobs and zero snapshots.

The storage is fine. df on the repository mount reports the same 4.1 TiB used as last night’s shift handover recorded, and all sixty-one repository directories are present. Each is named by the UUID the platform allocated at client registration, and nothing on the mount says which business system any UUID belongs to.

At 09:00 finance asks for last Tuesday’s copy of the orders system. The platform returns an empty result set, which on the bridge reads as the backups are gone. Two hypotheses follow: the repositories went with the array, or they survived but cannot be opened because the keys were in the catalogue.

Evidence provided

The first move is to leave the platform out of it and talk to a repository directly.

Read-only / Safea repository, asked what it contains, with no platform process involved
$ restic snapshots
ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
3e349a12  2026-08-28 13:27:03  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
2 snapshots

Identifier, time, originating host, the job’s tag, the paths, the size: the columns the catalogue’s snapshot table held. The passphrase came from the secrets manager, which holds one entry per repository UUID.

Read-only / Safea restore, still with no platform involved, checked against digests taken from the source
$ restic restore 3fe43af4 --target /work/restore
restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00
>>> exit code: 0

--- comparing the restored tree against the 09:00 checksums ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK
>>> md5sum -c exit code: 0
Read-only / Safenot described, read: every pack re-read and re-hashed
$ restic check --read-data
using temporary cache in /tmp/restic-check-cache-1847567736
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found
>>> exit code: 0

A second repository, a Borg one, answers the same way.

Read-only / Safeit names its own archives, with their times and fingerprints
$ borg list /work/repo
  day1                                 Fri, 2026-08-28 13:58:06 [fae87fac26949fa0ab2994a62819407fce5de3418d8d3bc23d4608fd466988c0]
day2                                 Fri, 2026-08-28 13:58:07 [85d3e533e094eb96663fd26a148b14737667bfc4251eea6b7144827ea2d279b8]
>>> exit code: 0
Read-only / Safeand the same repository re-read: verification prints nothing and exits 0
$ borg check --verify-data /work/repo
  >>> exit code: 0

Then the platform’s job definitions, recovered from the configuration-management repository, are searched for the one job whose absence the incident turns on.

Read-only / Safesixty-one jobs, and not one of them names the platform or its database
$ ls /srv/rbdr-config/jobs | wc -l; grep -rlE 'rbdr-backup-01|rbdr_catalogue' /srv/rbdr-config/jobs /etc/cron.d /etc/systemd/system
61
>>> grep exit code: 1 - no file matched

Illustrative output

The report that read 100 per cent for fourteen months came from this query:

SELECT c.hostname, j.name, max(r.finished_at) AS last_success
  FROM jobs j
  JOIN clients c ON c.id = j.client_id
  LEFT JOIN runs r ON r.job_id = j.id AND r.status = 'ok'
 GROUP BY c.hostname, j.name;

Work the evidence before reading on

  1. Two hypotheses were offered on the bridge. Which single command above eliminates the first, and why is a listing not enough to do it?
  2. The restore succeeded. What does that tell you about where the keys were, and which hypothesis does that retire?
  3. The snapshot listing has six columns. Which of them did the catalogue also hold, and which did only the catalogue hold?
  4. Read the SQL. Under what circumstances can that query emit a row for a system that has no backup job?
  5. Fourteen months of 100 per cent. What population was that a percentage of?

Root cause

The catalogue lived in a database that no job named

The platform kept its clients, jobs, retention classes, run history and the mapping from business system to repository UUID in a PostgreSQL database on the host’s local array. Every job it ran protected something else. No job named the platform host, no job named the database, and no dump was scheduled anywhere on the machine — the install guide’s Protecting the catalogue paragraph was never acted on. The repositories were on a separate mount and were never at risk. The only thing lost was the index.

The coverage inventory was keyed on the tool

Its rows come from the job table, joined out to the clients those jobs name, so every row it can emit begins life as a job that already exists. A system with no job contributes no row, and an absent row reads as nothing rather than as a gap. The platform’s own state was precisely such a system: it could never have appeared, not as a red row, not at all. The report was measuring the job table against itself — and it was stored in the catalogue, inside the thing it failed to protect.

Resolution

Mount the repository storage read-only and inventory it before anything else: the repositories are now the only surviving record of what was backed up.

Suspend all retention. restic forget, restic prune and borg prune exist to destroy recovery points on purpose, and the platform no longer knows which policy governs which repository.

Rebuild the index from the repositories themselves. restic documents a global --json flag for scripting, which turns the listing above into something a rebuild can consume:

REPOROOT=/srv/rbdr-repos
CATALOGUE=/srv/rbdr-rebuild/rbdr-catalogue.tsv
mkdir -p "$(dirname "$CATALOGUE")"
: > "$CATALOGUE"
for REPO in "$REPOROOT"/restic-*; do
  export RESTIC_REPOSITORY="$REPO"
  restic snapshots --json \
    | jq -r --arg repo "$REPO" '.[] | [$repo, .short_id, .time, .hostname, (.tags // [] | join(",")), (.paths | join(","))] | @tsv' \
    >> "$CATALOGUE"
done
for REPO in "$REPOROOT"/borg-*; do
  printf '# %s\n' "$REPO" >> "$CATALOGUE"
  borg list "$REPO" >> "$CATALOGUE"
done

Serve the 09:00 request directly from the repository the reconstruction identifies, without waiting for the platform to be well.

Recover from independent sources what the repositories cannot describe: the UUID mapping from the client registrations committed to the configuration-management repository, retention classes from the written policy, restore-test history from the ticket system. Then state plainly what is not recoverable — the run history and the coverage attestations — rather than letting a rebuilt database imply a provenance it lacks.

Last, create the job that did not exist: a scheduled pg_dump of the catalogue, written where it can be found without the catalogue.

Verification

Reconcile the rebuilt catalogue against the asset inventory, not against the job table. Every UUID resolves to a named system or is recorded as unidentified.

Restore from a repository the rebuilt catalogue selected, into a scratch target, and compare the tree against reference digests. In the capture, Restored 7 files/dirs (60.000 MiB) at exit 0 and md5sum -c at exit 0 are read together; either alone proves less than it appears to.

Restore the new catalogue dump into a scratch database and query it before declaring the job to work; a dump that has never been restored is a file. Do it with the platform stopped, the only state in which it will ever be wanted.

Regenerate the coverage report from the asset inventory as its left side. The number of systems with no job must be a figure the report prints. On first run that figure is not zero, which is how you know it works.

Prevention

Treat the backup platform as a protected system with a named owner. Its catalogue, configuration and credentials are production state. No platform volunteers this: its view of the estate is what it was told to protect.

Store the catalogue dump where it can be found without the catalogue. A dump inside a repository whose UUID the catalogue must resolve is a circular dependency that fails at the moment it is wanted.

Build coverage from the asset inventory outward. Enumerate systems from a source the platform does not own, then left-join the jobs. A report that iterates jobs can only report 100 per cent, whatever is missing.

Keep the repositories self-describing. Set host, tags and paths at backup time so the listing carries them, as the capture’s does. That property is the only reason this incident was recoverable.

Keep the UUID to business-system mapping in version control, committed at registration, so it survives the platform that allocated it.

Rehearse the loss of the platform, not only the loss of a client. Restoring a file while the platform is healthy exercises none of the machinery this incident needed.