Full, differential, incremental and incremental-forever

Independence settled where the copies live and what has to fail before all of them fail together. It said nothing about what is inside each one. Two estates can hold three independent copies apiece and still have entirely different recovery properties, because one of them stores self-contained images and the other stores one image plus a long sequence of deltas that mean nothing on their own. The scheme is not a scheduling detail. It decides how many separate stored objects have to be present, intact and correctly ordered before a restore can hand back a single file.

Four schemes, described by what a restore has to find

The four schemes are usually taught by what the backup writes. That is backwards for an operator, because the write is the part that works. Describe them by the object count on the restore side instead.

A full backup captures everything in scope on every run. A restore of any full needs exactly one artefact. That is the entire advantage, and it is a large one: no ordering to get right, no parent to locate, no other run whose failure can reach into this one. Every run reads the whole source and writes the whole volume, so the window and the storage bill scale with the estate rather than with its rate of change.

A differential backup captures everything that changed since the last full. Because every differential is measured against the same full, they do not depend on each other. A restore needs the full and the most recent differential, and that count is two whether the full was taken last night or three weeks ago. Differentials grow through the week — each re-captures everything the previous one did, plus that day’s changes — so storage cost rises toward the cost of a full while the restore dependency stays fixed at two.

An incremental backup captures what changed since the previous backup, whatever that was. This makes each run as cheap as it can possibly be, and it makes the restore depend on the whole chain from the full up to the point being recovered. With a Sunday full and nightly increments, recovering Tuesday’s state requires the full plus Monday’s and Tuesday’s increments: three objects. Recovering Saturday’s requires seven. Every one of those objects is a separate opportunity for a bad block, a failed upload, an expired retention rule or an operator error. A gap does not announce itself, either. In a capture of a GNU tar --listed-incremental chain — the format where the levels are explicit — level 0 and level 2 were applied with level 1 missing, and the restore reported no error at all. It produced a directory holding the newest app.conf beside an orders.csv that had silently lost a line, because tar was asked to apply the archives it was given and did exactly that.

Incremental-forever takes one full at the beginning of the repository’s life and never takes another. Instead of restoring by replaying a chain, the system maintains a single logical full that is continuously updated — either by merging new increments into it on the backup target, which is what a synthetic full is, or by making each new backup the full and demoting its predecessor to a reverse increment. The restore dependency stops growing with the age of the chain. What replaces it is a dependency on the process that does the merging, and on one set of stored objects that is now referenced by every recovery point you have.

NIST SP 800-34 Rev. 1 names the first three in its contingency-planning guidance. They survived the move from tape to object storage because the arithmetic did not change when the media did.

Measured: a 1.370 KiB backup that presents as a 60.000 MiB recovery point

The incremental economics are worth seeing on real output, because the size of the effect is what makes the trade-off so easy to take without noticing.

A restic repository was initialised and a first backup taken from a 60 MiB source tree of three files. That run logged no parent snapshot found, will read all files and reported Added to the repository: 60.005 MiB (60.008 MiB stored). One file was then modified and the backup was repeated.

$ restic backup /work/prod --tag daily

using parent snapshot 3fe43af4

Files:           0 new,     1 changed,     2 unmodified
Dirs:            0 new,     3 changed,     1 unmodified
Added to the repository: 2.062 KiB (1.370 KiB stored)

processed 3 files, 60.000 MiB in 0:00
snapshot 3e349a12 saved

>>> exit code: 0

Sixty megabytes of source, processed in full, produced 2.062 KiB added to the repository, 1.370 KiB of it stored. That ratio is why practical schemes end up incremental underneath: nothing else makes nightly protection of a large estate affordable in either time or storage. It is also, in the same breath, why the chain became load-bearing. Almost none of the bytes needed to reconstruct that snapshot were written by the run that created it.

Now look at how the repository presents the two runs to whoever is choosing a recovery point during an incident.

$ restic snapshots

ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
3e349a12  2026-08-28 13:27:03  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
2 snapshots

Both snapshots list at 60.000 MiB. One of them cost 60.005 MiB to write and the other cost 2.062 KiB, and the catalogue does not distinguish between them, because the Size column describes the tree the snapshot represents rather than the objects a restore of it would have to read. Both tools measured in this lesson present their catalogue this way, and both are honest about what they report. Neither reports the dependency, so an operator scanning the list sees equally self-sufficient recovery points and has no reason to think otherwise.

The chain is load-bearing, and retention is what leans on it

Two things routinely remove an object a chain still needs, and only one is an accident.

The first is damage. In the same restic capture, ten bytes were overwritten in the middle of the repository’s largest data pack, a 17374653-byte file that kept its name, its size and its place in the directory listing. The restore that followed reported Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) and exited 1. One damaged object, one lost file — and because unchanged files are stored once and referenced by both snapshots, that object is reachable from every recovery point containing the file, not only from the one that was tested.

The second is retention, which does the same thing deliberately and on a schedule. Deleting a recovery point is not a bug in a retention policy; it is the policy working. What a scheme decides is what else stops being recoverable when it happens. Under a differential scheme, removing last week’s full retires that week’s differentials with it, predictably. Under an incremental chain, removing a full or an increment from the middle can invalidate everything after it, and the tool will not necessarily warn you, because from its point of view you asked for exactly that.

Borg: a different repository, the same arithmetic

The behaviour is not a restic peculiarity. Borg reaches it by a different route and reports it in a different table, and the numbers say the same thing. Two archives were written to an encrypted Borg repository; this is the second.

$ borg create --stats /work/repo::day2 .

Archive fingerprint: 85d3e533e094eb96663fd26a148b14737667bfc4251eea6b7144827ea2d279b8
Time (start): Fri, 2026-08-28 13:58:07
Time (end):   Fri, 2026-08-28 13:58:07
Duration: 0.07 seconds
Number of files: 2
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                     Original size      Compressed size    Deduplicated size
This archive:               41.94 MB             41.94 MB                613 B
All archives:               83.89 MB             83.89 MB             41.95 MB

                     Unique chunks         Total chunks
Chunk index:                      23                   40
------------------------------------------------------------------------------

>>> exit code: 0

Read the row for this archive across: 41.94 MB original, 41.94 MB compressed, 613 B deduplicated. Read the row beneath it: two archives totalling 83.89 MB of original size occupy 41.95 MB. And read the chunk index, which is the clearest statement of the model in either tool — 23 unique chunks out of 40 total. Seventeen of the chunks in the second archive were already in the repository, so they were referenced rather than written.

Original size is Borg’s equivalent of the restic Size column: truthful about the archive’s contents, silent about its dependencies. An archive that cost 613 B is a complete view of a 41.94 MB tree only because chunks an earlier run wrote are still there.

Which brings back retention, on the tool whose command for it is the bluntest in the field.

$ borg prune --list --dry-run --keep-daily=1 /work/repo

Keeping archive (rule: daily #1):            day2                                 Fri, 2026-08-28 13:58:07 [85d3e533e094eb96663fd26a148b14737667bfc4251eea6b7144827ea2d279b8]

>>> exit code: 0

The policy keeps day2 and names nothing else, so day1 — the archive that actually wrote 41.94 MB of chunks — is what it would remove. The capture stopped at the dry run, so what happens after that is the repository model rather than a measured result: a chunk survives while some remaining archive still references it, which is the same reachability rule the restic capture records for forget and prune. The phrasing of the output is the part to take away. It reports what it keeps, and everything absent from that list is what you lose. Run it with --dry-run and read the list every time.

Incremental-forever moves the failure, it does not remove it

The chain-length problem has an obvious fix, and the fix is genuinely good: stop letting the chain grow. A synthetic full merges recent increments into the stored full on the backup target, so there is always one current logical full and the restore dependency stops climbing. Content-addressed repositories arrive there by never having had a chain.

What this does not do is remove the dependency. It changes its shape, and three consequences follow that are easy to miss because the dashboard looks better afterwards, not worse.

The merge itself becomes a component. Something has to run on the target to fold new data into the logical full, and it can fail, stall or fold in the wrong thing. Under a weekly full, a bad week is repaired by the next full. Under incremental-forever there is no next full, so a defect introduced by the maintenance process persists until something reads the data and notices.

Nothing re-reads the source, and this is the consequence with the longest reach. A periodic full is not only a backup; it is a fresh, independent observation of the source. In an incremental-forever repository a chunk written once and never changed is never re-read, so an error in the stored copy is never overwritten by a correct one. Only verification that reads the stored bytes will find it, which makes scheduled deep verification structural to the scheme rather than a nicety.

And the blast radius consolidates. One damaged pack cost one file across every snapshot referencing it. Scale that to years of recovery points on the same shared chunks, and repository-level damage stops being a property of one night’s backup — which is the previous lesson’s independence argument applied to whole repositories rather than to media.

What to take from this

• The first run logged no parent snapshot found, will read all files and added 60.005 MiB; the second logged using parent snapshot 3fe43af4 and Added to the repository: 2.062 KiB (1.370 KiB stored) for the same 60 MiB tree. After the first run, every subsequent run in that repository is incremental whether or not anyone chose a scheme.
• restic snapshots listed both runs at 60.000 MiB although one cost 60.005 MiB and the other 2.062 KiB. The catalogue reports the tree a snapshot represents, never the objects a restore of it depends on.
• A GNU tar --listed-incremental chain replayed with level 1 missing reported no error. It produced the newest app.conf beside an orders.csv that had lost a line, so a broken chain can present as a completed restore rather than as a failure.
• Borg’s second archive reported 41.94 MB original, 613 B deduplicated, with a chunk index of 23 unique chunks out of 40 total. A different repository, a different report, the same dependency on objects an earlier run wrote.
• Ten bytes overwritten in one 17374653-byte pack produced Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) and exit code 1. Damage to a shared object is reachable from every recovery point that references it.
• borg prune --list --dry-run --keep-daily=1 reported only Keeping archive (rule: daily #1): day2. Retention output names what survives, so everything absent from that list is what the run would destroy.

Cross-course references

• Proxmox VE for Production Operators — Part XIII (Proxmox Backup Server) operates a chunk-based backup repository, so the arithmetic measured here applies directly to it: nightly guest backups are cheap to write because they reference chunks earlier runs stored, and the retention policy configured there is the mechanism that removes those objects.
• Linux for Production Sysadmins — Part XLIX (Restore) is where the object count chosen in this lesson is actually paid, because the restore procedures in that part have to locate and read every object the scheme made a recovery point depend on before anything is handed back.
• Observability for Production Sysadmins — Part XVIII (Alerting Rules) covers alerting on the absence of data rather than on bad values, which is the only alert shape that catches the failure in this lesson: a chain is broken by an increment that never arrived, and a rule watching job results sees nothing at all.

Quiz