Restore dependency chains and the risk of chain length

Full, differential and incremental describe how much data each run writes. The question that decides whether a recovery point is usable is the mirror image of that: for this point in time, what does the restore have to read? The answer is almost never a single object, and the set is almost never visible in the interface that lists recovery points. Underneath each row sits a set of stored objects, and it is that set — not the row — that either restores or does not.

flowchart LR
    subgraph Linear["Classic incremental chain"]
        F["Sunday full"] --> I1["Monday incremental"] --> I2["Tuesday incremental"] --> I3["Wednesday incremental"] --> RP1["Wednesday recovery point"]
    end
    subgraph Repository["Deduplicating repository"]
        RP2["Snapshot A"] --> T1["Tree A"]
        RP3["Snapshot B"] --> T2["Tree B"]
        T1 --> P1["Shared pack 1"]
        T1 --> P2["Pack 2"]
        T2 --> P1
        T2 --> P3["Pack 3"]
    end

The line makes a missing parent invalidate every later point. The repository graph makes the blast radius depend on which snapshots share the missing pack; dates alone reveal neither relationship.

A recovery point is the transitive closure of what restore reads

Start with the arrangement everyone already pictures: a full backup on Sunday, then an incremental every weekday. To recover Saturday’s state you read the Sunday full and each of the six incrementals in order — seven objects, and the recovery point exists only if all seven are readable. Change to differentials and it needs two. Nothing about the data differs; what differs is the size of the set that must survive.

Treat that set as what it literally is: the transitive closure of a dependency relation, computed outward from the recovery point. It is not defined by dates or by the order of rows in a listing but by reachability — an object is in the chain because the restore reads it, or because something the restore reads reaches it.

In a repository-based tool the closure has more levels than the weekly-full picture suggests. A restic snapshot references a tree object; each tree references further trees and data blobs; each blob lives inside a pack file. The chain for one snapshot is therefore a set of pack files, and the relation is many-to-many: one pack backs several snapshots, and one snapshot needs many packs.

The shape of the closure matters as much as its size. A classic incremental chain is a line: every link has exactly one successor, and a break at position three makes positions three through seven unreachable in a single stroke. A repository chain is a directed acyclic graph in which many recovery points share the same objects, so a broken object does not remove a neat suffix — it removes every recovery point that happens to reference it. That may be all of them or exactly one, and which cannot be read off the dates.

Chain length matters for two independent reasons. The first is time: reading n objects costs more than reading one, and across object storage the per-object latency often dominates the transfer. The second is probability. Model each object as independently readable with probability 1 - p; a chain of n objects is fully readable with probability (1 - p)^n. At an illustrative p = 0.001 — chosen for the arithmetic, not measured from any repository — seven objects give 0.993 and 180 give 0.835. The per-object reliability did not change. The chain did.

Computing the chain for a given recovery point

The listing is the wrong artefact to reason from. Two backups were taken of a 60 MiB source tree on restic 0.19.1, and the second changed exactly one file.

$ restic backup /work/prod --tag daily

using parent snapshot 3fe43af4

Files:           0 new,     1 changed,     2 unmodified
Dirs:            0 new,     3 changed,     1 unmodified
Added to the repository: 2.062 KiB (1.370 KiB stored)

processed 3 files, 60.000 MiB in 0:00
snapshot 3e349a12 saved

>>> exit code: 0

Everything else the new snapshot needs was already in the repository, written by the run before it. Then look at how the repository presents the two:

$ restic snapshots

ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
3e349a12  2026-08-28 13:27:03  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
2 snapshots

Both rows say 60.000 MiB, because that column reports the size of the source tree the snapshot represents, not what the snapshot stores or what a restore will have to read. The first backup added 60.005 MiB to the repository; the second added 2.062 KiB. Two rows identical in every visible column sit on storage that differs by four orders of magnitude in what each one contributed.

Computing the real chain therefore means asking the tool, not the listing. Name the recovery point, ask what the restore reads first — a snapshot record, a catalogue, a manifest — expand every reference it contains, and keep expanding until nothing new appears. The cardinality of the result is the chain length, the oldest write time among its members is the chain’s age, and the union of those sets across every recovery point you claim to hold is what must stay intact.

Two things make that worth writing down rather than recomputing under pressure. The answer changes whenever a backup runs, a policy is applied or a repository is copied, so a chain is a dated statement and needs a timestamp beside it. And the object set is the input to every other decision in this part of the course: what has to be verified, what may be pruned, what must be replicated to the second site, and how long a restore of that point will take. A recovery-point manifest — identifier, object count, object list or its digest, and oldest member — turns all four of those questions into lookups instead of investigations.

One unreadable object makes a listed recovery point unreachable

The consequence is not gradual. Ten bytes were overwritten in the middle of the largest pack in that repository — the file kept its name, its size of 17374653 bytes and its place in the directory listing, restic check still reported no errors were found with exit code 0, and the snapshot stayed in the listing at 60.000 MiB. Then it was restored.

$ restic restore 3fe43af4 --target /work/restore2

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors

>>> exit code: 1

Six of seven entries came back, 59.401 MiB of 60.000 MiB, and the missing file failed its checksum comparison against the source. One unreadable object did not degrade the recovery point by one-seventh — it removed the database file, which was the reason anyone would have restored it.

That is what makes chain length a risk quantity rather than a storage-efficiency detail: every object in the closure is another whose loss removes the whole recovery point.

Retention and prune are how a chain loses a link on purpose

Corruption is the accidental way to lose a link. Retention is the deliberate one, and far more common, because retention policies are written in the vocabulary of dates while chains live in the vocabulary of reachability.

The classic accident is a policy that keeps the last fourteen daily backups and, in doing so, removes the weekly full all fourteen incrementals still read. It is arithmetically correct about dates and wrong about dependencies, and it produces the state this lesson is about: fourteen listed recovery points, none reachable.

Repository tools split the operation to make this survivable: removing a recovery point removes only the snapshot record, and space is reclaimed by a separate pass that first computes which chunks remain reachable. That separation is protective only while the pass can see the whole picture. A reachability pass evaluated over a subset of the recovery points — a filtered view, a partial copy of the repository, a mirror that has not yet received the newest records — computes the right function over the wrong input, and deletes on that reasoning without reporting anything unusual.

Three failure shapes account for most of the damage, and they deserve separate names because their fixes differ. A retention policy removes a parent that a later increment still reads — fixed by making retention operate on reachability rather than on dates, which is precisely what the two-phase forget-then-prune split exists to do. A prune runs while a chain is still referenced from somewhere the prune cannot see — fixed by repository locking and by never applying retention against a partial view. And a repository ends up listing recovery points it can no longer fully materialise, which nothing fixes after the fact: it can only be detected by reading the chain, and that detection has to be scheduled, because no operation removes the stale row on its own.

Borg’s retention command makes the intent explicit, and a dry run shows precisely which recovery points a policy destroys before it destroys them:

$ borg prune --list --dry-run --keep-daily=1 /work/repo

Keeping archive (rule: daily #1):            day2                                 Fri, 2026-08-28 13:58:07 [85d3e533e094eb96663fd26a148b14737667bfc4251eea6b7144827ea2d279b8]

>>> exit code: 0

One archive is kept, and by omission the other is destroyed. That is prune working correctly: it is the one command in a backup system whose purpose is to remove recovery points. Treat every retention change as a change to the set of chains you can still complete, and have someone read the dry-run list. Deletion is also available to anyone holding the backup credentials — on borg 1.4.0, borg delete of an archive returned exit code 0 even with append_only = 1 set.

Chain-length limits, fresh fulls, and the age of the oldest dependency

Two practices usually inherited as tradition become straightforward once chains are the unit of reasoning.

A chain-length limit caps how many objects any recovery point may depend on. It is a risk control with two measurable inputs: the observed per-object failure rate of your storage over the retention window, and the service’s restore-time budget. Pick the limit that keeps (1 - p)^n above your tolerance and the object count inside that budget, and state both inputs beside the number. “Start a new chain monthly, because that is what we have always done” states neither, and goes wrong unnoticed when storage or data volume changes.

A periodic fresh full enforces the limit by producing a recovery point whose closure contains only recently written objects, so recent recovery points stop depending on old ones. In a deduplicating repository this needs care, because re-running a backup into the same repository references the existing chunks rather than rewriting them. A genuinely fresh full means new storage — a new repository, or a copy that rewrites the objects — followed by a verification that reads the data.

The limit and that verification cadence are one decision rather than two. Chain length sets how many objects have to survive; the interval between checks that actually read the data sets how long any one of them may be damaged before anyone finds out. Lengthening the chain while leaving the interval alone raises the object count and the rot window together — the path by which a repository moves from a few unreachable recovery points to most of them without any single change looking wrong.

That leads to the metric this lesson is arguing for. The useful number is not the age of the newest backup, which is always reassuring, but the age of the oldest object a recent recovery point depends on. A backup taken last night whose oldest dependency was written fourteen months ago carries fourteen months of accumulated storage risk, and no dashboard built on job timestamps shows it. A crude approximation comes from the repository’s storage:

REPO=/work/repo
OLDEST=$(find "$REPO/data" -type f -printf '%T@ %p\n' | sort -n | head -n 1)
echo "oldest stored object under $REPO: $OLDEST"

What to take from this

• A recovery point is the closure of what restore reads, not the row in a listing. restic snapshots showed both snapshots at 60.000 MiB while the first had added 60.005 MiB to the repository and the second 2.062 KiB.
• One damaged pack made a listed snapshot restore 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) and exit 1 on restic 0.19.1, with the listing row unchanged before and after the damage.
• That same repository reported no errors were found with exit code 0 while the chain was already broken, so a green structural check is not evidence that a chain can be completed.
• Deduplication makes chain age independent of backup age: on borg 1.4.0 the second archive of a 41.94 MB tree deduplicated to 613 B, with a chunk index of 23 unique chunks against 40 total chunks.
• Retention destroys recovery points by design. borg prune --list --dry-run --keep-daily=1 kept day2 and, by omission, marked the other archive for removal — and on borg 1.4.0 borg delete returned exit code 0 even with append_only = 1 set.

Cross-course references

• Observability for Production Sysadmins — Part XVIII (Alerting Rules) covers writing rules against a derived quantity rather than a raw event, which is what the chain-age metric here needs: the alert fires on the age of the oldest object a recent recovery point depends on, not on whether last night’s job finished.
• Linux for Production Sysadmins — Part XLIX (Restore) works through the restore side of file-level recovery, and the chain computation in this lesson is what tells you, before that restore starts, how many stored objects it must read successfully to finish.
• PostgreSQL for Production Sysadmins — Part XIII (Backup, Archiving and Point-in-Time Recovery) is the sharpest instance of a chain: a base backup plus every WAL segment after it, where one missing segment does not shorten recovery by a segment but caps it at the moment before the gap.

Quiz