Rebuilding a backup catalogue

Recovering one file straight from the storage with nothing but the tool and the key, which is where the previous lesson left a platform outage, works because a repository can be read without the platform that wrote it. That answers the emergency and not the estate. Sooner or later somebody has to ask which recovery points exist for the other four hundred systems, and that is the catalogue’s question: what exists, and where are its bytes. Lose it and you are holding data you cannot find. Whether that costs an afternoon or costs everything was decided in the on-disk format, years before anyone opened an incident ticket.

What the catalogue is actually asked for

Strip the interface away and a catalogue answers two questions. What recovery points exist? and where are the bytes of the one I want? Everything else the product renders — the calendar view, the search box, the per-client summary — is presentation over those two answers.

$ restic snapshots

ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
3e349a12  2026-08-28 13:27:03  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
2 snapshots

Six columns, and each one is doing different work. The ID is the address a restore is given. The time is what a recovery objective is measured against. The host and the paths are the claim that this recovery point came from a particular system and covered a particular part of it. The tag daily is a label a human chose. The size is a capacity figure.

Two of those columns are worth staring at, because they are where this lesson ends up. The host reads 8211a08b55c3, which is a container identifier — a string generated by a runtime, meaningful to nothing outside it. Nowhere in the repository is there a sentence saying that this host was the orders API, or a staging box, or the machine somebody built to test a theory in March. And the tag daily describes a schedule that lives in a cron entry on a machine that may no longer exist. The catalogue carries the strings. The meaning of the strings was always somewhere else.

Derived state and the database that is the only copy

Backup platforms fall into two families here, and the distinction is not a detail of implementation. It decides what a catalogue-loss incident costs.

In the first family the catalogue is derived state. The stored objects carry enough description of themselves that the mapping from recovery points to bytes can be recomputed by reading them. The index exists so that a restore does not have to do that reading every time; it is an accelerator over information that is already present. restic and Borg both work this way, and Borg demonstrates it in the least ambiguous manner available: the upstream procedure for rolling a repository back to an earlier transaction begins by deleting the repository’s own bookkeeping files, and the capture followed it.

$ borg list /work/aorepo

day1                                 Fri, 2026-08-28 13:58:09 [edf6f20cb15b5a2ed56ed0a8f4abe6e02e4d6a1e305729d1febb01608482195f]
day2                                 Fri, 2026-08-28 13:58:09 [dc55669922a0cbf88e0c74390976f69dd69c7ae81f90d3ea22b977c4b24c35e7]
day3                                 Fri, 2026-08-28 13:58:09 [06305e70a6a1ea7dc8c56e68eae5290ad7c52a4f22b559e8c46ac03b6398ad6a]

Three archives, full fingerprints, and no restoration of the deleted files because no backup of them existed or was needed. An archive extracted after this listing exited 0. A format whose index was authoritative could not have a documented recovery procedure whose first step is to remove it.

In the second family the catalogue is a database outside the objects. A master or media server holds it; clients send data through that server to disk pools, buckets or tape; and what lands there is numbered fragments carrying no statement of what they contain. Which client, which backup, which fragment order, which volume, which retention or legal hold — all of it exists in exactly one place. That design buys fast browsing across billions of files and policy evaluation that never touches storage, and it is a legitimate trade. What it also buys is a single point of failure that is not the data, and the honest way to hold it is that the objects are unaddressable without the database.

The question that separates the families is short enough to put in a procurement document, and almost nobody asks it: if the catalogue were lost tomorrow and every stored object survived untouched, what is the documented recovery, and what does it cost at our object count?

The scan is priced in objects, not terabytes

The instinct is to estimate a rebuild from the size of the repository, because size is the number everyone knows. It is the wrong number. Two repositories holding the same forty terabytes rebuild at wildly different costs if one is two hundred thousand large objects on a local disk and the other is two million small ones in a bucket that bills per request, and neither figure appears on a datasheet.

Which means the rebuild duration is a measurement you owe yourself, taken on a copy, before the day you need it. It is cheap to take.

REPO=/srv/backup/repo
export RESTIC_REPOSITORY="$REPO"

PACKS=$(find "$REPO/data" -type f | wc -l)
START=$(date +%s)

restic repair index

echo "rebuilt the index over $PACKS packs in $(( $(date +%s) - START )) seconds"

Record the two numbers together, because one without the other does not transfer. Eleven minutes over four thousand packs tells you something about tomorrow’s repository; eleven minutes on its own tells you nothing. Then run the structural check and the reading check afterwards, in that order: an index rebuilt from the objects describes the repository as it now is, and only the checks turn that description back into a statement about what can be restored.

What comes back, and what never does

Run the scan and the listing returns. Look carefully at what is in it, because the gap between that and a working recovery service is where the second half of the incident lives.

Retention intent is the clearest case. The measured Borg repository was pruned under a policy, and the policy did not come from the repository.

$ borg prune --list --dry-run --keep-daily=1 /work/repo

Keeping archive (rule: daily #1):            day2                                 Fri, 2026-08-28 13:58:07 [85d3e533e094eb96663fd26a148b14737667bfc4251eea6b7144827ea2d279b8]

>>> exit code: 0

--keep-daily=1 decided which recovery point survived, and it was typed into the invocation. The repository contributed the archives and the timestamps; the rule that ranks them lived in a wrapper script, a systemd unit, a configuration management role, or a person’s shell history. Rebuild the index a hundred times and none of those come back. The same is true of restic, where forget is handed its policy at each run. A rebuilt catalogue can list every recovery point you hold and be silent on which of them you had promised to keep for seven years.

Client mapping is the second case, and the transcript hands it over: the host column read 8211a08b55c3. The repository knows a string that identified the machine to itself. It does not know that the machine was the payment gateway, or which of four hosts with similar names is the one under discussion, or that the name was reused for something unrelated after a rebuild. In an estate where the recorded names are container ids, cloud instance ids, or hostnames from an autoscaling group, the mapping from those strings to systems people can name is external by construction.

Human-readable naming is the third, and it is subtler than it sounds. The Borg archives were day1, day2, day3; the restic snapshots carried the tag daily. Those strings survive any rebuild, because they are in the repository. What does not survive is the schedule that gave them meaning — which job wrote day2, whether daily meant the 02:00 run or the pre-deployment one, whether a tag was retired last year and reused this year for something different. Someone standing in front of a recovered listing at 03:00 needs to choose a recovery point, and every column they would use to choose is a label whose interpretation lives outside the system that stored it.

The catalogue is a system in the inventory

All of which produces one operational rule, and it is unusual among the rules in this course because it costs almost nothing to adopt.

The catalogue is small, irreplaceable and rarely inventoried, which is a combination that only ever ends one way. It belongs in the estate inventory as a system in its own right — an owner, a documented recovery procedure, a stated recovery objective, and its own recovery points on independent storage. The external record that survives it should hold the things a scan cannot return: the retention policy as it is actually invoked, the mapping from recorded host strings to named systems, what each tag and archive name means, and the version of the tool that reads the objects.

Then monitor its recovery point age, which is the only signal that tells you whether that record is still true. Not whether the dump job exited 0 — the age of the newest usable dump, computed from the artefact itself.

DUMP=/srv/catalogue-dumps/latest.dump
MAX_AGE_HOURS=26

WRITTEN=$(stat -c %Y "$DUMP")
AGE_HOURS=$(( ( $(date +%s) - WRITTEN ) / 3600 ))

echo "backup_catalogue_recovery_point_age_hours $AGE_HOURS"
if [ "$AGE_HOURS" -gt "$MAX_AGE_HOURS" ]; then
  echo "catalogue recovery point is $AGE_HOURS hours old, over the $MAX_AGE_HOURS hour threshold" >&2
  exit 1
fi

The 26-hour threshold is not a property of any tool. It is the stated consequence of a daily dump schedule plus one missed run, chosen so a single failure is visible before a second one hides it — change the schedule and the number changes with it. And the check must run somewhere that survives the loss of the backup platform, because a catalogue-age alert evaluated by the platform it describes goes quiet at exactly the moment it had one job.

Production discipline

1. Ask the rebuild question before the platform is bought, and keep the answer in writing. If the catalogue were lost and every object survived, what is the documented procedure and what does it cost at our object count? A vendor who cannot answer has answered.
2. Measure the scan on a copy and record the duration next to the object count. The measured restic repository held a 60 MiB tree across seven packs; a rebuild pays once per object, so seven and seven million are different architectures described by the same sentence.
3. Write down what a rebuild cannot return, at backup time. The Borg capture kept day2 because --keep-daily=1 was typed into the prune invocation, and the restic catalogue recorded its client as 8211a08b55c3 — retention intent and client identity both lived outside the repository.
4. Keep catalogue recovery points on storage that fails independently of the platform and of the data. A standby that copies current state also copies a bad migration or a wrong-policy retention run, which is how catalogues are usually lost.
5. Put the catalogue in the inventory and alert on the age of its newest recovery point, from somewhere the platform’s failure does not take down. Age computed from the artefact, with the threshold justified by the schedule that produces it.

Cross-course references

• PostgreSQL for Production Sysadmins — Part XIII (Backup, Archiving and Point-in-Time Recovery) is this lesson’s problem inside a database: a directory of archived WAL segments is bytes without the metadata saying which base backup they continue and on which timeline, which makes that metadata the catalogue this lesson tells you to protect and record separately from the data it describes.
• Observability for Production Sysadmins — Part XVIII (Alerting Rules) is where the catalogue recovery point age from the closing script becomes a rule with a threshold and an owner, rather than a number a script prints into a log nobody reads.
• Observability for Production Sysadmins — Part XC (Meta-Monitoring) covers the constraint that makes that alert worth anything: a check on the backup catalogue must be evaluated by something the backup platform’s failure does not also remove, which is the same independence argument this lesson applies to where the catalogue dumps are stored.

Quiz