Retention schemes as patterns, not prescriptions

Every restore dependency chain in the previous lesson began at a recovery point that still existed. Retention is the mechanism that decides which points exist, and it decides by destroying the rest. Every other scheduled operation in a backup system adds something — a snapshot, a copy, a verification result. Retention alone reduces what can be recovered, it runs unattended, and it reports success identically whether it removed the right points or the wrong ones.

Deletion on a schedule, reported as success

A backup that fails is at least discoverable: the next run compares against a parent that is not there, the dashboard row goes red, someone eventually looks. A retention run that removes too much produces no error at all, because removing snapshots is exactly what it was asked to do and from the tool’s point of view nothing went wrong. The failure surfaces later, as a restore request that finds nothing to restore from, and by then the evidence of what was deleted and why is a line in a job log.

That asymmetry has two consequences. The first is that retention parameters are production configuration with the blast radius of an unrestricted delete, and deserve the review any such change gets elsewhere in the estate. The second is that the parameters have to be derived rather than inherited. A keep-7-daily, keep-4-weekly, keep-12-monthly triple is a reasonable starting shape, and it is the shape most schedules arrive with. It stops being a policy the moment nobody can say which failure in the model the twelve monthly points were sized against, because then nobody can say what is lost by cutting them to three.

The two questions that produce the numbers

The first question is how far back you might need to go. Not how far back you usually go — almost every restore comes from last night, and the common case is not the case retention exists for. The number you need is the longest plausible interval between a problem starting and somebody noticing it, because every recovery point written inside that interval may already contain the problem.

Two failure classes set that interval, and neither of them is about convenience. The first is silent corruption. Part I of this course measured a restic repository in which a damaged data pack survived restic check with no errors were found and exit code 0, and was only exposed by restic check --read-data. The practical consequence for retention is arithmetic: if a full data verification runs quarterly, the detection latency for corruption is bounded below by a quarter, and a retention depth shorter than the verification interval can delete the last known-good recovery point before anyone has established that it was the last known-good one. The second class is compromise. The relevant figure is your own time from initial access to detection, and if the retention window is shorter than it, then every recovery point still present at the moment of detection was written while the adversary was already inside.

The second question is what you are required to keep and what you are required to destroy. Retention has a floor and a ceiling, and neither is an engineering choice. The floor comes from regulation, contract and litigation hold, and backup repositories are routinely in scope for it. The ceiling comes from data protection obligations: a repository is a place where deleted records go on existing, and retention is the only mechanism that eventually makes an erasure true across the backup estate. Deduplication complicates the ceiling, because a record is not gone when the last snapshot containing it is forgotten — it is gone when the space is reclaimed and no surviving snapshot references the chunk. The borg capture behind this material shows how wide that gap can be from the other direction: after three archives had been deleted from an append-only repository the data directory still reported 41M, because append-only withholds compaction and the segments stay on disk until compaction runs. A control that keeps recovery points alive and an obligation that requires records to disappear pull the same lever in opposite directions, and no value between them satisfies either.

These two obligations frequently conflict, and the resolution is never to average them. If financial ledgers require seven years and session data must be erasable within thirty days, one repository holding both cannot satisfy both requirements, and the fix is to separate the data sets so each lives under a single policy. Only after both questions have answers does choosing parameters become a technical exercise.

--keep-daily, --keep-weekly, --keep-monthly: a shape, not a policy

Grandfather-father-son came from tape. A fixed pool of media was rotated through three generations — daily tapes reused within the week, weekly tapes reused within the month, monthly tapes held for a year — and the scheme existed because the constraint was a cupboard with a countable number of cartridges in it. The modern expression of the same shape drops the cupboard and keeps the generations: sort the snapshots newest first, and for each rule keep the most recent snapshot falling in each calendar bucket until that rule’s count is satisfied. A snapshot kept by any rule is kept; everything else falls to the removal side.

What the shape asserts is that recovery granularity should be dense near the present and sparse in the past, and that assertion is a claim about detection. Operator error is usually found within hours, so the recent tier needs many closely spaced points. Corruption and compromise are found over weeks or months, so the deep tiers need reach rather than density. When your failure model matches that gradient the traditional numbers are a reasonable starting point. When it does not, they are simply wrong: a dataset whose dominant risk is a migration quietly damaging rows over several weeks needs density in the weekly tier, not the daily one, and a dataset that is regenerated from source every morning may need almost no depth at all.

Three mechanical details decide whether the parameters mean what you think. Keep-last is time-independent — it retains the newest N snapshots regardless of when they were taken — which makes it useful where backup cadence is irregular and misleading where it is the only rule. Selection inside a bucket is by recency, not by quality: both tools keep the most recent snapshot falling in each interval, so a policy evaluated after a bad run promotes that run into the tier and discards the point it displaced, which is why a deep tier is worth little without verification that a point was good. And no policy can keep a recovery point that was never taken: a tier finer than the backup schedule simply relabels the points that exist, so a daily rule over a twice-weekly schedule yields a “daily” tier whose members are days apart. All three are visible the moment the policy is evaluated against the actual snapshot list, which is what a dry-run does.

Measured: policy evaluation is a separate command from destruction

Both tools in this part split the operation in two, and both of them will show you the decision before acting on it.

$ restic forget --keep-last 1 --dry-run

1 snapshots

remove 1 snapshots:
ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
1 snapshots

Would have removed the following snapshots:
{3fe43af4}

>>> exit code: 0

Nothing was destroyed. The policy was evaluated against the snapshot list, the affected snapshot was named by ID, and the command exited 0. The second half of the separation matters just as much: forget removes snapshot records, space is reclaimed only by prune, and a chunk is deleted only when no remaining snapshot still references it. Borg draws the same line, and prints the opposite half of the list.

$ borg prune --list --dry-run --keep-daily=1 /work/repo

Keeping archive (rule: daily #1):            day2                                 Fri, 2026-08-28 13:58:07 [85d3e533e094eb96663fd26a148b14737667bfc4251eea6b7144827ea2d279b8]

>>> exit code: 0

Restic named what it would remove. Borg named what it would keep, and said nothing whatever about day1, whose absence from the survivor list is the entire report. A reviewer who reads a keep-list as though it were a removal list, or who skims a removal list for names they recognise, approves the exact opposite of what they believe they are approving. The rule that survives both tools is to count: compare the number of surviving recovery points against the number the policy was designed to leave, and record that arithmetic where the next person can check it.

REPO=/srv/backup/prod
PREVIEW=/var/tmp/retention-preview.txt

restic --repo "$REPO" forget \
  --keep-daily 14 --keep-weekly 8 --keep-monthly 12 \
  --dry-run | tee "$PREVIEW"

The preview file is the artefact, not the terminal scrollback, and it belongs with the change record: once the real forget has run, the repository can no longer say what it used to hold. Comparing a preview taken under the current policy against one taken under the proposed policy is the cheapest review available, because the difference between the two lists is exactly the set of recovery points the change destroys, expressed as IDs and timestamps rather than as parameters.

The capacity fix that deleted a later incident’s recovery point

Nothing in the sequence that follows requires anybody to be careless, which is why it recurs. The capacity alert fires on the backup volume at 02:00. It is a real alert, the volume really is filling, and the on-call engineer really does have to act. Procurement takes weeks; changing what is in the backup set requires knowing what the data is; the one lever that is immediate, one-line and entirely within reach is retention. Monthly points go from twelve to three, the next prune brings the volume back under threshold, and the ticket is closed with the words “free space on backup volume”.

Seven weeks later a corruption is found in a dataset whose last verified-good state was five months back. The recovery points for months four through twelve no longer exist. Nobody deleted them maliciously or by accident, and no tool reported an error at any stage — the prune that removed them exited 0, exactly as designed.

This is a pattern rather than a coincidence, and the reason is organisational. The capacity problem is visible, alerting and owned by a named on-call rotation. The recovery-depth requirement is invisible, silent, and usually owned by nobody in particular. Retention is the single control that trades one for the other, it performs the trade instantly, and it is reached for under exactly the time pressure that prevents anyone from asking what the deep tier was for. The discipline that breaks the pattern is to classify any reduction in retention depth as a risk acceptance rather than a capacity action, which forces the change to name the failure class it is removing coverage for and the person who accepted that. Under genuine pressure, cut granularity before depth: dropping seven of fourteen daily points removes the most redundant tier in the scheme, while dropping nine of twelve monthly points removes three quarters of the reach. And per the mechanism above, measure the reclaim first, because the change that costs the most coverage often frees the least space.

Production discipline

1. Derive both numbers before writing either parameter. Depth comes from the longest detection latency in the failure model — the verification interval for silent corruption, the time to detect a compromise — and the floor and ceiling come from obligations. Tier counts are chosen last.
2. Run the policy in dry-run and make a person read the list. Restic printed Would have removed the following snapshots: and {3fe43af4} at exit 0; borg printed Keeping archive (rule: daily #1) at exit 0. The two tools print opposite halves of the same decision, so state which half you are reading and count the survivors.
3. Treat a retention reduction as a risk acceptance, not a capacity fix. The change must name the failure class losing coverage and the person accepting it, and under pressure it should cut granularity in the dense tier before cutting reach in the deep one.
4. Measure the reclaim before committing to it. Restic’s second backup of a 60 MiB tree added 2.062 KiB; borg’s second archive of 41.94 MB deduplicated to 613 B. Space is freed in proportion to unique data, so a deep cut on slow-changing data can cost every month of reach and return almost nothing.
5. Know the window between removal and reclamation, and never rely on it. After three deletes in the append-only borg repository the data directory still held 41M, and a rollback to transaction 13 restored the archives with extract exit code 0 — but only because compaction had not yet run.

Cross-course references

• Observability for Production Sysadmins — Part XL (Log Retention) applies the same two questions to telemetry, and the relationship is direct: a log retention window shorter than your detection latency destroys the evidence that would tell a restore engineer which recovery point predates the problem, so the two windows have to be sized against the same number.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part XXX (Pull Requests and Merge Requests) is the review mechanism this lesson depends on: the dry-run list is the diff, and a retention change that reaches production without a human reading that list is an unreviewed merge into a job whose only function is destruction.
• Proxmox VE for Production Operators — Part XIII (Proxmox Backup Server) expresses retention with the same keep-tier vocabulary at the datastore layer, which means the depth derived here is the depth entered there, and the failure model that justified it has to travel with the numbers rather than being reinvented per platform.

Quiz