Losing the catalogue: metadata recovery

The previous lesson damaged stored objects and watched the damage surface: a pack that failed its hash, a restore that came back with six files of seven. This lesson takes nothing away from the data at all. Every byte the backup system has ever written is still on disk, still readable, still matching whatever checksum it carries. What disappears is the thing that says which recovery point those bytes belong to. Some designs survive that with an afternoon of scanning. Others do not survive it at any price, and the difference was decided in the on-disk format long before anyone had an incident.

flowchart TB
    subgraph SelfDescribing["Self-describing repository"]
        SO["Stored objects with embedded mapping"] --> Scan["Scan surviving objects"] --> RI["Rebuilt index"] --> RestoreA["Restorable recovery points"]
    end
    subgraph ExternalCatalogue["Externally catalogued repository"]
        OO["Opaque stored objects"] --> LC["Lost authoritative catalogue"] --> Orphaned["Bytes survive but recovery points are orphaned"]
    end

The same physical survival therefore has two different outcomes. In the first design the catalogue is derived state and recovery costs a scan; in the second it is irreplaceable recovery material and must be protected independently.

The question worth asking before a platform is chosen

There is one question that separates backup products more sharply than deduplication ratios or compression algorithms, and almost nobody asks it during an evaluation: if the index or catalogue database were lost tomorrow, and every stored object survived untouched, could the data be recovered, and how long would it take?

It is not a trick question and it is not hypothetical. The catalogue is a database. It lives on a disk, it is written by a process, it has a schema, and it fails the way databases fail — a filesystem full at the wrong moment, a storage array replaced by someone who believed the backup server held nothing of value, a restore of the backup server itself from an image that predates half the recovery points. The stored data survives all of that quite happily, because it is large, boring and rarely touched. The small, hot, constantly rewritten metadata is exactly the part most likely to be lost.

Answers fall into two families. In a self-describing repository, the index is an accelerator: it says where each object lives so a restore does not have to look, but the mapping it holds is also present in the stored objects themselves, and can therefore be recomputed by reading them. Losing the index costs a scan. In an externally catalogued design, a database outside the object store holds the only copy of the mapping between recovery points and objects; the objects are opaque numbered blobs with nothing in them that says which backup, which client or which file they belong to. Losing the database orphans the data — the bytes remain, and nothing on earth can say what they are.

Four sub-questions turn the general one into something a vendor can answer. Is the mapping present in the stored objects, or only in the database? Is the rebuild a documented, self-service procedure, or a support engagement? How long does that rebuild take at your object count, not at the count in the datasheet? And what else does the catalogue hold that the objects do not — retention state, legal holds, which volume of a tape library holds which fragment, which client an archive belongs to?

Derived state, measured: a Borg repository without its index

Borg is a clean instance of the first family, and the evidence for that comes from an unlikely place: the upstream procedure for rolling a repository back to an earlier transaction, which the append-only capture followed after a compromised client deleted three archives. Step one of that procedure is to delete the repository’s own bookkeeping files without ceremony. That in itself is the answer to the rebuild question — a format whose index was authoritative could not have a documented recovery procedure that begins by removing it.

$ rm -f /work/aorepo/hints.* /work/aorepo/index.* /work/aorepo/integrity.*

--- first attempt to read the rolled-back repository ---
Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)

--- archives after the rollback ---
day1                                 Fri, 2026-08-28 13:58:09 [edf6f20cb15b5a2ed56ed0a8f4abe6e02e4d6a1e305729d1febb01608482195f]
day2                                 Fri, 2026-08-28 13:58:09 [dc55669922a0cbf88e0c74390976f69dd69c7ae81f90d3ea22b977c4b24c35e7]
day3                                 Fri, 2026-08-28 13:58:09 [06305e70a6a1ea7dc8c56e68eae5290ad7c52a4f22b559e8c46ac03b6398ad6a]

--- and can the recovered archive still be extracted? ---
>>> extract exit code: 0

Read the sequence in order, because it contains two separate lessons about derived state.

The repository’s index, hints and integrity files were removed and the repository still knew everything. All three archives came back with their full fingerprints, and the archive that was extracted afterwards exited 0 with its contents intact. Nothing was restored from a backup of those files, because there was no backup of those files; Borg regenerated what it needed from the segment log that had never stopped being authoritative.

The first read attempt nevertheless failed, and the reason is the second lesson. The refusal came from the client, not the repository: the local cache and security directory still described the post-attack state, which was newer than what now sat on disk, and Borg treats a repository that has apparently travelled backwards in time as a possible attack. That state was also derived, and it too was fixed by deletion rather than restoration — the capture cleared ~/.cache/borg and ~/.config/borg/security and the listing above is what followed. Derived state does not only fail to help during a recovery. It can actively block one, and the fix is to recognise it as disposable.

restic belongs to the same family and spells the rebuild out as a command rather than leaving it implicit, which makes it the better one to put in a runbook. The sequence below is worth writing down before it is needed, because the alternative is reading about repair index for the first time in an error message, during an incident, from a shell that is already under time pressure.

REPO=/srv/backup/repo
export RESTIC_REPOSITORY="$REPO"

# Rebuild the index by reading the header of every pack file.
restic repair index

# Then re-establish what the two levels of check each prove.
restic check
restic check --read-data

The ordering matters: an index rebuilt from the packs describes the repository as it now is, so the structural check and the reading check are what turn that description back into a statement about recoverability. Why the rebuild is possible at all is a property of the on-disk format in both tools.

etcdutl snapshot status: a file that describes itself in four values

The same distinction appears in systems that are not backup products at all, and etcd is the cleanest small example. An etcd snapshot is a single file containing a complete copy of the keyspace, and everything needed to describe that file is inside it. There is no catalogue anywhere; the tooling reads the description straight out of the snapshot.

$ etcdutl snapshot status /work/backup.db --write-out=table

┌──────────┬──────────┬────────────┬────────────┬─────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ VERSION │
├──────────┼──────────┼────────────┼────────────┼─────────┤
│ 8c66d749 │       54 │         53 │      45 kB │   3.7.0 │
└──────────┴──────────┴────────────┴────────────┴─────────┘

snapshot file size: 48K

Four values and a version: a hash, a revision, a key count and a size, read out of a file that arrived without any surrounding system. The keyspace had 53 keys written into it and the status reports 53; the revision counter stood at 54; the version column reports 3.7.0, the server’s storage version, while the tooling that produced and read the file was 3.7.1. That is the complete inventory of what the snapshot says about itself, and it is enough to answer the only question that matters when a file turns up in a bucket six months later: is this the recovery point I think it is?

It is enough only if the answer was written down somewhere else at backup time. A hash the file computes about itself, compared against nothing, proves that the file is internally consistent — not that it is the file you intended to keep, not that it has not been replaced, and not that a truncated copy did not overwrite the good one. The comparison needs an external record, and the external record is the part teams skip.

SNAP="/srv/etcd/snapshots/etcd-$(date -u +%Y%m%dT%H%M%SZ).db"

etcdctl snapshot save "$SNAP"
etcdutl snapshot status "$SNAP" --write-out=table | tee "$SNAP.status"
sha256sum "$SNAP" >"$SNAP.sha256"

The .status and .sha256 files are the point of that sequence, and they are only worth writing if they leave the machine — into a ticket, a configuration repository, an object store in a different account, anywhere that survives the loss of the host that produced them.

Two limits of that self-description are visible in the same capture and are easy to over-read. The snapshot does not carry cluster identity: restoring it with an identical --initial-cluster reproduced the original cluster id 1c45a069f3a1d796, while restoring the same file with a different membership produced 80d54574493dd420. And the status output describes the snapshot, never the gap after it — a key written after the snapshot was absent from the restored cluster, which is the RPO, not a defect. Self-describing means the file explains itself. It does not mean the file explains your estate.

There is one more piece of recovery metadata that is not in any file, and the capture found it the way people find it in production.

$ etcdctl snapshot status /work/backup.db

Manages etcd node snapshots

Usage:
etcdctl snapshot [command]

On this version status and restore are no longer subcommands of etcdctl; they live in the etcdutl binary. A runbook that still spells the command the old way does not fail during a quiet review — it fails at the first line of a control-plane recovery, in front of an audience. The name and version of the tool that reads your recovery points is metadata about those recovery points, and it goes stale on the vendor’s schedule rather than yours.

When a database outside the objects is the only map

The second family is where catalogue loss becomes existential rather than slow. The pattern is easy to recognise once named: a media or master server runs a catalogue database; clients send data through it to disk pools, object storage or tape; and the objects that land there are numbered fragments with no embedded statement of what they contain. Everything that turns those fragments back into a recovery point — which client, which backup, which time, which fragment order, which volume of which tape library, which retention or legal hold applies — exists in exactly one place.

That design is not a mistake. It buys fast browsing across billions of files, policy evaluation without touching storage, and cross-client deduplication accounting that would be impractical to recompute. What it also buys is a single point of failure that is not the data, and the honest way to hold it is to treat the catalogue as a production database with its own recovery objectives, its own backup schedule, and its own proven restore.

Two rules follow, and they are rules rather than suggestions because the failure they prevent is unbounded. The catalogue is backed up separately from the data, on its own schedule, because its rate of change and its recovery objective are different. And the catalogue copy lives somewhere that fails independently of the data, because a single event that removes both leaves an intact repository nobody can address.

The scenario that closes the gap is the one nobody schedules. Restore tests overwhelmingly exercise the happy path — the catalogue is up, it lists the recovery points, one is selected and restored. That test never touches the question this lesson asks. Rehearsing catalogue loss means starting from storage plus whatever external records exist, and measuring how long it takes to reach a listable, restorable state. The measurement is the deliverable; a rebuild that is documented but takes eleven hours at your object count is a different architecture from one that takes twenty minutes, and both are described by the same sentence in the datasheet.

Production discipline

1. Ask the rebuild question in writing before a platform is chosen, and keep the answer. If the index were lost and every object survived, what is the documented procedure and what does it cost at our object count? A vendor who cannot answer has answered.
2. Treat the catalogue as production state with its own backup, on its own schedule, on storage that fails independently of the data. Its change rate and its recovery objective are not the data’s, and a single event that takes both leaves an intact repository nobody can address.
3. Record the identity of every recovery point outside the system that stores it. For the measured etcd snapshot that is hash 8c66d749, revision 54, 53 total keys and 45 kB — worthless inside the file it describes, decisive when held in a ticket or a repository somewhere else.
4. Record the recovery tooling and its version alongside the recovery points. etcdctl snapshot status returned a usage message on etcd 3.7.1 because status and restore moved to etcdutl; a runbook naming the old command fails at the first line of a control-plane recovery.
5. Rehearse the catalogue-loss scenario and time it. The Borg capture deleted hints.*, index.* and integrity.*, cleared the client cache and security directory, and got day1, day2 and day3 back with an extract at exit code 0 — evidence that exists because someone ran it, not because a format document promised it.

Cross-course references

• Kubernetes for Production Sysadmins — Part LXVIII (etcd Backup) covers the cluster-level procedure that surrounds the snapshot file examined here, including how often to take one and where to send it; this lesson supplies the narrower point that etcdutl snapshot status is the whole of the metadata the file carries, so the hash, revision and key count have to be recorded outside it at backup time.
• PostgreSQL for Production Sysadmins — Part XIII (Backup, Archiving and Point-in-Time Recovery) is the same problem in a database: a directory of archived WAL segments is only a recovery point in combination with the base backup and the metadata saying which base backup those segments continue, which makes that metadata exactly the catalogue this lesson tells you to protect separately.
• Observability for Production Sysadmins — Part XCI (Backup Strategy) is where the external record from rule 3 should land, because a hash and a revision written only onto the host that produced them do not survive the incident they exist for; the monitoring estate is the system best placed to hold and alert on that record.

Quiz