Choosing a backup tool honestly · Backup & DR

Losing a repository’s own bookkeeping was the last failure this part had to put on the table, and with it the ground is finally prepared for the question most people arrive at Part VIII already holding: which tool should we standardise on? This lesson does not answer that. The right answer is a property of an estate — its data, its credentials, its storage, its people — and anyone who hands you a name without having seen those four things is selling something.

What can be handed over is a rubric: the questions whose answers decide it, the measured evidence for what two credible candidates answer today, and one rule about when a decision is permitted to be made at all.

Eight questions, and “which is better” is not one of them

Run every candidate through the same list. The questions are ordered roughly by how expensive the answer is to discover late.

Can it restore what you will actually be asked for, at the granularity you will be asked for it? One file, one directory, one database, one machine — the shapes differ, and a tool is only good at the shapes you need.
What does its integrity verification read, and what does that cost? A check that reads bookkeeping and a check that reads every stored byte are different claims at different prices, and both usually appear as one row.
Can the writing client be prevented from destroying history? The client holds credentials; assume they are stolen, then ask what remains.
Where does the encryption key live, and can it be escrowed? A key that travels only with the protected host is a key the disaster takes with it.
What happens when the catalogue is lost? Repositories carry an index over their stored objects, and losing it must be an inconvenience rather than an ending.
How does it behave when the repository is full? The behaviour under exhaustion decides whether the incident is a pause or a corruption.
Is it maintained, and on what release cadence? A repository format is something you must still be able to open in three years.
Can your team operate it at three in the morning? Under pressure, with the wrong person on call, using a procedure nobody rehearsed this quarter.

Notice what is absent. There is no question about which tool is faster, none about which has more backends, and none about which is more popular. Those are real properties and none of them is a recovery capability. Notice also that questions one, two, three and eight cannot be answered by reading anything at all: they are answered by running the tool against data shaped like yours and watching what it does.

A feature matrix fails on the same four rows. Both tools examined in this part would put a tick under “integrity verification” and a tick under “append-only support”, and both ticks are true. The captures in the previous lessons showed that the two properties mean measurably different things in each case. A tick records that a mechanism exists; the rubric asks what the mechanism reads, who enforces it, and what it costs — and no cell is wide enough for that.

What the verification reads, and what the answer costs

Question two separates candidates faster than any other, because every backup tool has something called a check and the checks are not comparable.

Service impact possiblerestic's reading verification on a healthy repository— EXECUTION VALIDATED on restic 0.19.1. Step 5 of the repository capture, run against the healthy two-snapshot repository; the exit-code line is the capture's own, with the two-space indentation the capture applied removed.

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-1847567736
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found

>>> exit code: 0

Borg answers the same question, and answers it differently in a way that is purely about operations.

Service impact possibleBorg's reading verification, which succeeds by saying nothing— EXECUTION VALIDATED on borg 1.4.0. Step 4 of the repository capture, run against the healthy two-archive repository; the capture recorded no output other than the exit status.

$ borg check --verify-data /work/repo

>>> exit code: 0

Both tools pass question two: each has a verification that reads the stored data rather than only the structure over it. The tick is earned in both columns. What the rubric wants is the next sentence, and the two transcripts supply it. restic printed a progress line, read all data, a count of 7 / 7 packs and the string no errors were found. Borg printed nothing at all. Both exited 0.

That difference decides how the verification is monitored, which is an operational cost attached to a real column. A tool that reports by exception gives you no event to alert on, so the only monitorable signal is the age of the last successful run, recorded by whatever wrapper invokes it. A tool that prints a success string gives you a second option and a second failure mode, since a log line can also be matched by a run that verified something other than what you assumed. Neither is better; one of them is better for the monitoring you already have, which is a question about your estate.

The cost side of question two is arithmetic and it is the same for both. A verification that reads every stored object performs the I/O of a full read of the repository, so its duration and its bill scale with repository size and with whatever the storage charges for retrieval. That is why the cadence of the reading check is a capacity decision rather than a preference, and why the honest way to state a repository’s integrity is with a date attached.

Question one — granularity — is answered the same way, by execution rather than by reading. The captures show the shape of the difference without deciding it: restic’s restore takes an explicit destination, invoked in the capture as restic restore 3fe43af4 --target /work/restore2, while Borg’s extraction was invoked as borg extract /work/repo::day1 and unpacks relative to the working directory. That is a small ergonomic divergence with a large consequence for the eighth question, and neither transcript tells you whether either tool can give you the one file your application owner will ask for at 03:10. Only restoring that one file tells you.

Two secrets, two custodians, and a client you cannot trust

Questions three and four are about an adversary rather than about an accident, and they are the two most often answered by assumption.

The append-only result from earlier in this part is the reason question three has to be phrased as “who enforces it” rather than “does it have it”. With append_only = 1 set on the repository, three borg delete commands exited 0 and the archive listing came back empty. Append-only does not refuse the deletion; the upstream documentation says as much, and what it forbids is compaction, which is why 41M of repository data remained on disk and the transaction log grew from 5, 9 and 13 to include 17, 21 and 25. The control is real — the deletion is reversible by rolling the repository back to transaction 13, which the capture did, after which the archives listed again and the extraction exited 0 — but the recovery belongs to an operator with filesystem access on the repository server. It is not available to the compromised client, and that is exactly the point.

Question four has a cleaner answer, and both tools pass it by different means. Under a repokey mode Borg stores the key in the repository directory itself, so every replica carries the encrypted key material with it, and restic’s repository holds a single file under keys/ that is the master key encrypted with a key derived from the passphrase. In both designs the passphrase is the secret that decides everything, and both tools offer a way to stop it being a single point of failure.

Configuration changea second passphrase on the same repository, held by someone else— EXECUTION VALIDATED on restic 0.19.1. Step 4 of the encryption and escrow capture, reproduced in the order the capture printed it, with the capture's two-space indentation removed.

$ restic key add --new-password-file /work/recovery-pass

saved new key with ID 66c34166d8443d16e8c5899fe3f792e749cb24fa3a90ac90bbe8348979b57d90

>>> exit code: 0

$ restic key list
ID        User  Host          Created
--------------------------------------------------
66c34166  root  17dffded9807  2026-08-28 14:04:55
*4bc6f61a  root  17dffded9807  2026-08-28 14:04:52
--------------------------------------------------

Two keys, one repository, and neither passphrase derives the other: both decrypt the same master key. That is escrow expressed as a repository operation, and it means the recovery team never has to be told the production passphrase in order to be able to open the repository. Borg reaches the same goal from the other direction, exporting the key material itself as a printable BORG PAPER KEY v1 block that a safe in another building can hold — still useless without a passphrase, which is precisely why the two are escrowed to different custodians.

The rubric does not prefer either mechanism. It asks whether the mechanism was exercised, and the escrow capture answers that in the only acceptable way.

Read-only / Safethe escrowed passphrase opening a repository whose site is gone— EXECUTION VALIDATED on restic 0.19.1. Step 5 of the encryption and escrow capture, after the production directory and the production passphrase had been destroyed; the capture's two-space indentation has been removed.

$ restic --password-file /work/recovery-pass restore latest --target /work/rec

restoring snapshot b96ba7cf of [/work/prod2] at 2026-08-28 14:04:52.481565631 +0000 UTC by root@17dffded9807 to /work/rec
Summary: Restored 3 files/dirs (38 B) in 0:00

>>> exit code: 0

recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical, using a passphrase production never held

Why the eight questions are really two facts about a repository

The rubric looks like eight independent criteria and is not. Nearly all of it falls out of two structural facts, which is why answers cluster: a tool that disappoints you on one question usually disappoints you on its neighbours.

Fact one: a repository is a store of content-addressed objects plus an index that names them. Everything the tool can do for you is bounded by what that index records. Granularity is bounded by it — a repository whose index describes files and directories can be asked for one file, and one that describes whole device images can be asked for a device. Verification cost is bounded by it too: checking the index alone is cheap because the index is small, and checking the objects means reading them, which is why the reading form of the check costs what a restore costs and no implementation can make that cheaper. Catalogue loss is the same fact seen from the other side — an index derived from the stored objects can be rebuilt from them, and an index that holds anything the objects do not is a single point of failure by construction. Behaviour when the repository is full is the last consequence: if recording a change requires writing to the store, then a store with no free space cannot cleanly record anything, including the deletions that would free space.

Fact two: the process that writes the backup holds credentials that reach that index. Deleting a recovery point is a write to the index, not an erasure of the objects, which is why the Borg measurement came out as it did — the archives left the listing while 41M stayed on disk and the transaction log grew rather than shrank. It follows that no control living inside the backup client can protect history from that client, because the client is the thing being defended against. Real enforcement has to sit where the client’s credential does not reach: in a serving process on the repository host, or in the storage layer beneath it under a retention setting that the backup account cannot shorten.

Read the eight questions again with those two facts in hand and the shape of the decision changes. You are not comparing feature sets. You are asking what a particular index records, and who can write to it.

A full repository, a lost catalogue and a release date

The three questions in the middle of the rubric are the ones that get skipped, because each describes a condition nobody is in on the day of the evaluation.

Question five was answered by execution in the previous lesson, and the general form of that answer is what matters for selection: establish, before standardising, whether the tool’s index can be reconstructed from the stored objects, how long that takes at your repository size, and what is unrecoverable if it cannot. A catalogue that is the only place some fact is written carries a different risk from a catalogue that is a cache.

Question six is worth rehearsing deliberately, because the incident it describes is common and the intuitive remediation is dangerous.

Anti-pattern: the repository is full, so remove old recovery points to make room

It is the tempting move and it is the wrong one, for two reasons this part has already measured. The first is that removing recovery points does not necessarily return space: policy evaluation and space reclamation are separate operations in both tools, and Borg’s reclamation step is exactly the one an append-only repository refuses, so a repository configured for protection will delete the archives and free nothing.

The second is that you are destroying recovery points during an incident whose cause you do not yet understand. If the repository filled because something is writing far more than it should — an encrypted-in-place dataset, a runaway export — then the recovery points you are about to remove are the ones from before it started. Add capacity, or write to a second repository, and work out what filled it afterwards.

Question seven is the one that rots fastest, so state the position with its source rather than from memory. This course’s source manifest recorded the versions from the upstream release feeds on 2026-08-28: for Borg, the newest non-beta release is 1.4.5, dated 2026-07-18, while the newest tags are 2.0.0b23 and 2.0.0.dev0 — so the 1.4 line is the stable line and 2.x is a beta that this course does not present as a production choice. The binary the captures ran against was borg 1.4.0. For restic, the manifest recorded 0.19.1 upstream against 0.18.0 in Debian 13, which is the second half of the same question: a project’s release cadence and your distribution’s packaging lag are different facts, and the version you will actually operate is the second one unless you decide otherwise and take on the upgrade yourself.

None of that makes either tool a bad choice. What it does is put a plannable cost on the table — a format migration ahead for one, a decision about packaging for the other — where it can be weighed rather than discovered.

The three-in-the-morning criterion and the evidence that settles it

The eighth question sounds like the soft one and is the most concrete of the set, because it has an unambiguous test: someone who was not involved in building the system restores something real from it, unaided, from the documentation you actually have.

Two properties measured in this part are the kind that decide it. The first is that both tools separate policy evaluation from destruction, and both will show you the decision before acting: restic’s dry run named the snapshot it would have removed, printing {3fe43af4} under Would have removed the following snapshots:, and Borg’s printed Keeping archive (rule: daily #1) for day2 alone, so the archive missing from the list was the one at risk. A tool that cannot rehearse a retention change is a tool whose retention changes are deletions nobody reviewed.

The second is that a restore’s result is its exit code, not its directory listing. The damaged-repository capture reported Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB), then Fatal: There were 1 errors, and exited 1 — leaving a file on disk with a plausible name and a plausible size and the wrong contents. Any rehearsal, and any wrapper, has to check the status and then check the bytes.

REPO=/srv/restic/web01
RECORD=/var/records/web01.md5
TARGET=$(mktemp -d /var/tmp/restore-rehearsal.XXXXXX)

if restic -r "$REPO" restore latest --target "$TARGET"; then
  ( cd "$TARGET" && md5sum -c "$RECORD" )
else
  echo "restore exited non-zero; treat the recovery point as unproven" >&2
fi

That is the whole of the selection procedure’s final step, and it is the only part of the rubric that cannot be delegated to a document. Run the candidate against a copy of your real data, at the granularity you will be asked for, with your storage backend underneath it and your on-call engineers driving. A tool that has not restored anything of yours has not been evaluated; it has been read about.

Production discipline

Score every candidate against all eight questions in writing, and record how each answer was obtained. An answer taken from documentation and an answer taken from a transcript are different kinds of evidence, and the difference belongs next to the answer.
Establish what the reading verification costs on a repository the size of yours before committing to a cadence. It performs the I/O of a full read, so the schedule is a capacity decision; state the integrity of a repository with the date of its last full verification attached.
Assume the backup client’s credentials are stolen, and name what stops them. With append_only = 1 three borg delete commands exited 0 and the listing emptied while 41M stayed on disk, so the answer is a serving process or a storage-layer setting the backup account cannot shorten, never a flag inside the client.
Escrow the key material and the passphrase separately, then prove the escrow by restoring with it. The capture added a second passphrase with restic key add, destroyed the production side, and restored a byte-identical file with a passphrase production never held.
Do not choose the tool until someone has restored real data with it, unaided. The demonstration that counts uses your data, your storage and your on-call engineers — and its result is the exit code, since a restore that exited 1 still reported Restored 6 / 7 files/dirs.

Cross-course references

Linux for Production Sysadmins — Part XLVIII (Backup Tools) surveys the tooling landscape this lesson turns into a decision, and the rubric here is what converts that survey into a choice: the same tools scored against an estate rather than against each other.
Observability for Production Sysadmins — Part XX (Alert Quality) is where question two’s operational half is settled, because a verification that succeeds silently — as borg check --verify-data did, exiting 0 with no output — has to be alerted on by the age of its last success rather than by an event that is never emitted.
Proxmox VE for Production Operators — Part XIII (Proxmox Backup Server) is the same rubric applied to a different product, whose answers on granularity, verification and client restriction differ from both tools measured here; this course has no Proxmox hardware, so those answers are read from documentation rather than executed.