When the backup platform is the thing that failed

The previous lesson took the catalogue away and left every stored object exactly where it was. This one takes away the machine that was holding the catalogue, the scheduler that filled it, the registration list that decided which hosts were allowed to write into it, and the key material that made any of it readable. Every service in a mature estate has a recovery procedure written by the people who run it, and most of those procedures name the backup platform as the source they would recover from. The platform itself usually has no such procedure, because it is not a service in that sense — it is the plan, and nobody writes a plan for losing the plan.

flowchart TB
    Store["Surviving backup storage"] --> Probe["Direct one-file restore"]
    Binary["Standalone format-aware binary"] --> Probe
    Escrow["Independent key escrow"] --> Probe
    Probe --> Decision{"Known-good data path?"}
    Decision -->|"No"| StorageIR["Investigate storage, format and key failure"]
    Decision -->|"Yes"| Rebuild["Rebuild isolated control plane"]
    Config["Escrowed catalogue, schedules, registrations, retention and keys"] --> Rebuild
    Rebuild --> Policy["Restore retention before attaching storage"] --> Attach["Attach surviving repositories read-only"] --> Restore["Resume prioritised restores"]

The direct probe separates a platform outage from a data-path failure. The ordering after it is equally important: restoring retention policy before the surviving repository is attached prevents a fresh default from pruning the history the rebuild exists to recover.

Five things the platform holds that are not the data

A backup platform is described, bought and defended in terms of the data path. Take an inventory of what the platform actually holds, though, and the data is the part that lives somewhere else — on a datastore, in a bucket, on a shelf of tapes. What sits inside the platform is five categories of state, and every one of them is a separate recovery problem.

The catalogue or index is the mapping from recovery points to stored objects: which archive was taken from which host, at which time, covering which paths, and where its pieces landed. The previous lesson established the question to ask of it — whether it can be rebuilt from the stored objects or whether its loss orphans them — and the answer is a property of the on-disk format, decided long before the incident.

The schedule and policy definitions decide what is protected at all. Losing them costs nothing on the night, because a schedule that is not running does not break an existing restore. It costs a great deal on the following nights, and the expensive part is not that backups stopped: it is that nobody can state which hosts were in scope, so the rebuilt platform protects the set someone remembers rather than the set that existed.

The client registrations and their credentials are the identities each protected host used to authenticate. Rebuild the platform and every client is a stranger to it; the agents keep trying, the platform keeps refusing, and the inventory of who was allowed to write what has to be reconstructed before a single new backup succeeds. The credentials pointing the other way belong in this category too — the ones the platform used to reach object storage, tape libraries and remote repositories.

The retention configuration decides how long each recovery point survives. This is the item with a sharp edge: a freshly installed platform carries default policy, and a default policy applied to a surviving datastore can begin removing recovery points that the previous configuration was keeping. A rebuild that reconnects storage before retention is restored to what it was is capable of destroying the history it was rebuilt to reach.

The encryption key material is the one that turns a platform outage into a permanent loss. If the platform generated per-client keys and stored them in its own database, that database is not configuration — it is on the data path, and the stored objects are unreadable without it. If the keys were always held outside, the platform is merely a convenience. Which of the two you have is not a detail; it is the whole difference between an inconvenient week and an unrecoverable estate.

Together these five are small. They are kilobytes to a few megabytes of configuration, they change rarely, and none of them arrive with the software. Reinstalling a backup platform from its packages produces a platform that is running, healthy, monitored, and knows nothing at all.

The first move is one file, and it happens without the platform

When the platform goes down, the instinct is to start rebuilding it, because that is the visible broken thing. That is the wrong order. Before anything is rebuilt, one question has to be answered, and it changes every decision that follows: is the stored data reachable at all without the platform?

The way to answer it is not to reason about the architecture. It is to recover a single file. Pick something small from a recent recovery point, put it in a scratch directory, and compare it against a checksum you already hold. What that attempt needs is a path to the storage, a binary that understands the format, and the key. What it deliberately does not use is the platform.

$ restic snapshots

ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
3e349a12  2026-08-28 13:27:03  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
2 snapshots

Two recovery points came back with their times, their originating host, their tags and their paths. That is a menu, and it was produced by a tool reading files. The next command turns the menu into bytes on disk, and the checksum comparison after it is the part that matters: the transcript below restores the whole three-file tree because the tree is 60 MiB, but restic restore takes --include to narrow a restore to a single path, so the same move on a real repository need not materialise anything large.

$ restic restore 3fe43af4 --target /work/restore

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00
>>> exit code: 0

--- comparing the restored tree against the 09:00 checksums ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK
>>> md5sum -c exit code: 0

Read what that took. A storage path, one binary, one passphrase, and a snapshot identifier that the storage itself supplied. Of the five categories above, the catalogue was consumed and it came out of the repository; the schedules, the client registrations and the retention configuration were not on the path at all, and the key was held by the operator rather than by a platform. Nothing here proves the whole estate is recoverable — it proves one file was, which is exactly the small, fast, decisive fact the first hour of a platform incident needs.

Reading the fork the first file creates

The single-file recovery produces one of two answers, and they lead to incidents that have almost nothing in common.

If the file comes back and verifies, the data is reachable and the incident is about the platform. Restores continue by hand, at whatever rate the people and the network allow, while the platform is rebuilt on its own timeline. The pressure moves to the schedule: no new recovery points are being written, so the loss window is growing for every protected host, and that — not the dashboard — is the thing to report upward hourly. How long the rebuild takes is a function of where the five categories of state were kept and how fast they can be retrieved, and it is knowable in advance only if someone has written that down.

If the file does not come back, the incident is about the data, and rebuilding the platform stops being the priority entirely. The questions become which second copy exists, whether it was written by an independent path, and whether the key that opens it is held anywhere other than the system that just failed. A team that discovers at this point that the second copy was a replica of the first has discovered it too late; the design work that would have helped belonged in the architecture, and the only thing left to do now is to work the copies that exist.

Both branches are cheaper to walk with the answer than without it, and the answer costs one file and a few minutes. That is why it goes first, ahead of the rebuild, ahead of the vendor call, and ahead of the incident bridge filling up with people who want a status.

The version constraint: the reader may need to be newer than the writer

A repository is not a pile of files that any release of a tool can read. The format carries a version, and the tooling has to understand the version it finds. restic’s documentation describes a repository format version and a migration that raises it, and states that a repository upgraded to the newer format can no longer be read by older restic releases. BorgBackup documents borg upgrade as the command for moving a repository from an earlier format to the current one. In both cases the direction is fixed: the migration goes forward, and the tooling that reads a migrated repository must be at least as new as the tooling that performed the migration.

This turns the version into recovery material. The recovery host is frequently not the machine the platform ran on — it is a laptop, a jump host, a rescue image, or a freshly provisioned instance — and what it has is whatever its distribution packages. A repository written by a current release and opened with a two-year-old packaged binary is a plausible thing to walk into at hour one of an outage, and the failure it produces is a confusing message about the format rather than a clear statement that the binary is too old.

$ restic version

restic 0.19.1 compiled with go1.26.4 on linux/amd64

restic 0.19.1 compiled with go1.26.4 on linux/amd64 is a complete answer to “what can read this repository”, and it costs one line in a document. The practical arrangement follows from it: record the format version and the tool version beside the repository address, and keep a copy of a binary that meets the floor somewhere the recovery host can reach. restic is distributed as a single self-contained executable, which makes storing a known-good copy next to the repository — or on the same escrow media as the passphrase — a small thing to arrange rather than a project.

The reverse direction, a newer tool reading an older repository, is the ordinary case and is generally supported. “Generally supported” is not the same as verified, though, and this course only counts verification that was performed. The way to hold a claim about version compatibility is the way to hold every other recovery claim: restore something with the exact binary you intend to use, and record the date you did it.

The platform’s recovery material has to live outside the platform

Everything above collapses into one design rule. The material needed to recover the backup platform must be held somewhere that does not depend on the backup platform being available. It is a short rule and it is violated constantly, because the platform is genuinely the best place in the estate to put things, right up until the moment it is the thing that failed.

What belongs in that external record is specific, and it is short enough to fit on a couple of pages: the repository and datastore addresses; read-capable credentials for the storage, held under whatever approval the estate requires; the repository format version, the tool version that satisfies it, and where a binary of that version is kept; the custody of the key material — who can produce it, from where, and with whose sign-off; the client inventory, so a rebuilt platform can be audited against the set of hosts that actually existed; the retention configuration that was in force; and the procedure for restoring one file without the platform, with the date it was last executed against a live repository. Kept in a form that needs the platform to read it, all of that is decoration.

The probe below is the executable half of that record. It uses only the storage, a passphrase file on separate media, and the binary, and it is short enough to run monthly.

export RESTIC_REPOSITORY=/srv/backup/repo-prod
export RESTIC_PASSWORD_FILE=/media/escrow/repo-prod.pass
PROBE=/var/tmp/platform-dr-probe
mkdir -p "$PROBE"
restic version
restic snapshots --json > "$PROBE/snapshots.json"
SNAP=$(jq -r '.[-1].short_id' "$PROBE/snapshots.json")
restic restore "$SNAP" --target "$PROBE" --include /etc/hostname

Production discipline

1. Recover one file before rebuilding anything. Measured on restic 0.19.1, a repository opened at its storage path listed 2 snapshots and reported Restored 7 files/dirs (60.000 MiB) at exit code 0, with all three files matching their pre-backup checksums. That result took a path, a binary and a passphrase, and it tells you which of the two possible incidents you are in.
2. Inventory the five things the platform holds that are not the data. The catalogue, the schedules and policies, the client registrations and their credentials, the retention configuration and the key material — and beside each, the copy that lives outside the platform and the date it was last refreshed. A category with no entry in the second column is a category that will be reconstructed from memory.
3. Record the format version and the tool version, and keep the binary. restic 0.19.1 compiled with go1.26.4 on linux/amd64 is one line, and a migrated repository cannot be read by releases older than the one that migrated it. Store the line and a copy of the executable with the escrowed passphrase, not on the platform.
4. Restore retention before reconnecting storage to a rebuilt platform. A fresh installation carries default policy, and default policy pointed at a surviving datastore is capable of removing recovery points the old configuration was keeping. Reconnect read-only first, verify the policy matches what was in force, then allow the platform to manage the store.
5. Rehearse the platform outage, not only the client restore. Declare the platform unavailable for the duration of the exercise and recover a file anyway; then rebuild the platform from the external record alone and measure how long that took. Both numbers are properties of your architecture and your escrow arrangements, and neither one is a property of the product.

Cross-course references

• Proxmox VE for Production Operators — Part XIII (Proxmox Backup Server) builds the platform this lesson removes, and Part XIV (Disaster Recovery) plans around its loss; read them against the inventory above, because a surviving datastore is only one of the five categories, and the datastore definitions, the job schedules, the client identities and the key material are configured on the server rather than stored in the chunks.
• Observability for Production Sysadmins — Part LXXXIX (Observability Platform Monitoring Itself) and Part XCII (Disaster Recovery) pose the identical recursion for a different platform: a system that watches everything is the last thing anything is watching. The reason this lesson opens with a restore rather than an investigation is the reason meta-monitoring has to run somewhere else — a platform cannot be the evidence for its own availability.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XV (KMS, HSM and Key Protection) and Part XVIII (Incidents and Recovery) own the fifth category in the inventory above: where key material is held, who can produce it under what approval, and what the procedure is when the holder is the system that has just failed.

Quiz