restic in production: backup, retention and prune

The repository model from the previous lesson — content-addressed chunks, an index that locates them, and snapshots that reference chunks rather than own them — stops being an abstraction the moment the commands that create and destroy those references go on a schedule. What follows is restic operated rather than restic introduced: the handful of commands a production schedule actually runs, what each one changes, and the output that has to be read before the next one is allowed to start.

What restic init writes, and the single file under keys/

A repository is created once and then never again, which is exactly why the decisions taken at that moment are the ones nobody revisits. restic init writes the directory structure, generates a master key, and stores that master key encrypted under a key derived from the passphrase you supply. Everything written afterwards — packs, index files, snapshots — is encrypted with the master key, and the master key is only reachable through a passphrase.

export RESTIC_REPOSITORY=/srv/backup/repo
export RESTIC_PASSWORD_FILE=/etc/restic/passphrase
restic init
restic backup /srv/app/data --tag daily --tag app

The consequence is visible in the repository itself. A freshly initialised repository contains exactly one key file, and it is not a copy of your passphrase.

$ ls /work/repo/keys/

d5f39ef5517fab2e4d1e6dc1d9d5b4b1f1ca4eeacecde523af8b5a0c3c3d120e

One file, named for its own identifier, holding the master key wrapped in a key derived from the passphrase. Two facts follow directly. The first is that copying the repository copies this file too, so an offsite copy is complete in the sense that it needs nothing else from the origin site except the passphrase. The second is the uncomfortable half of the same fact: the repository is self-contained apart from the one thing it deliberately does not contain, and possessing every byte of it is not the same as being able to open it.

Tags, the parent snapshot, and what the second backup cost

restic backup takes paths and writes a snapshot. The two pieces of operator control worth using from the first run are tags and the parent snapshot, and the first backup demonstrates the absence of the second.

$ restic init && restic backup /work/prod --tag daily

repository initialised
no parent snapshot found, will read all files

Files:           3 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repository: 60.005 MiB (60.008 MiB stored)

processed 3 files, 60.000 MiB in 0:00
snapshot 3fe43af4 saved

>>> exit code: 0

no parent snapshot found, will read all files is the line that tells you the run has no prior snapshot to compare against for the same host and paths, so every file is opened and chunked. One file was then modified and the backup repeated, which is the shape of every night after the first one.

$ restic backup /work/prod --tag daily

using parent snapshot 3fe43af4

Files:           0 new,     1 changed,     2 unmodified
Dirs:            0 new,     3 changed,     1 unmodified
Added to the repository: 2.062 KiB (1.370 KiB stored)

processed 3 files, 60.000 MiB in 0:00
snapshot 3e349a12 saved

>>> exit code: 0

Read the two summaries as a pair. The run processed 3 files, 60.000 MiB and added 2.062 KiB to the repository, 1.370 KiB of it stored. Those are two different quantities and confusing them is the origin of most bad capacity and duration estimates. The snapshot represents a 60.000 MiB tree; the run moved a couple of kilobytes.

The parent-snapshot line is also an operational signal rather than a cosmetic one. When the parent is found, unchanged files are recognised from their metadata and are not re-read. When it is not — a renamed host, a changed path list, a snapshot that retention removed — the job silently reverts to reading everything, and a window sized for the incremental case can be badly overrun by a run that nothing reported as failed.

Tags are the cheapest structure you can impose. The --tag daily above appears in every listing, and retention policies and searches can both be scoped by it. A repository whose snapshots carry daily, weekly and pre-upgrade tags can have a policy expressed per tag; a repository whose snapshots are untagged forces every policy to be expressed by counting and dates alone.

Choosing a recovery point, then restoring it to a target

restic snapshots is the command that turns a repository into a menu. Its output is the list of recovery points you are actually choosing between.

$ restic snapshots

ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
3e349a12  2026-08-28 13:27:03  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
2 snapshots

Both snapshots report 60.000 MiB, because the size column describes the tree each snapshot represents, not what it cost to store. Choosing between them is a decision about time, and the default of taking the newest is only correct when the failure being recovered from is loss. If the failure is corruption or a bad deployment, the newest snapshot faithfully preserves it, and the operator’s job is to pick the last snapshot taken before the event — which is why the Time and Tags columns matter more during an incident than the ID column does.

Selection also has a shorthand, and the shorthand has a trap. restic restore latest resolves the snapshot for you rather than making you name an ID, which is convenient in a script and removes the one step at which a human would have looked at the menu. Repositories written to by more than one machine are common enough that forget groups its policy by host and paths unless told otherwise, and on such a repository the most recent snapshot may belong to a machine other than the one being recovered. A restore that ran cleanly against the wrong source looks exactly like a correct one. The defence is already printed: the first line of a restore names the snapshot, the paths it covers, the timestamp and the host that wrote it, and reading that line is the cheapest confirmation available that the right recovery point is being applied.

$ restic restore 3fe43af4 --target /work/restore

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00

>>> exit code: 0

That is what success looks like: 7 files/dirs, 60.000 MiB, exit code 0, and a subsequent md5sum -c against the checksums recorded at 09:00 returning OK for all three files with exit code 0. Now compare it with the same command, against the same snapshot, in a repository where ten bytes had been overwritten in the middle of the largest data pack.

$ restic restore 3fe43af4 --target /work/restore2

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors

>>> exit code: 1

The restore ran to completion, wrote a directory tree, and exited 1. Verifying what it produced gave ./app/app.conf: OK, ./app/orders.csv: OK and ./db/data.bin: FAILED, with md5sum: WARNING: 1 computed checksum did NOT match. The failed file is on disk with a plausible name and a plausible size, and nothing about the directory distinguishes it from the good one. The exit code is the result of a restore. The presence of files is not evidence, and a wrapper script that tests for the target directory instead of testing $? will report this run as a success.

forget applies a policy; prune reclaims the space

Retention in restic is deliberately two operations. forget evaluates a policy and removes snapshots, which are references. prune walks the index, finds chunks that no remaining snapshot references, and rewrites or deletes the packs that hold them. Between the two commands the repository has fewer recovery points and exactly the same size on disk.

Because a policy is the one thing in a backup system that deletes data on purpose, it gets dry-run first, every time it changes.

$ restic forget --keep-last 1 --dry-run

1 snapshots

remove 1 snapshots:
ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
1 snapshots

Would have removed the following snapshots:
{3fe43af4}

>>> exit code: 0

Would have removed the following snapshots naming 3fe43af4 is the entire value of the exercise: the policy was evaluated and printed the snapshot it matched, and the reviewer gets to decide whether that is the intended outcome before it becomes irreversible. Read that identifier against the snapshot list printed earlier — 3fe43af4 is the older of the two, so what a --keep-last 1 policy is proposing to keep is the newer one, which is what that policy should do. A policy that had instead named 3e349a12 for removal would be visible in this line and nowhere else, because once the real run has happened the snapshot that would have shown the mistake is the one that is gone.

What prune then does is more than deleting files. A pack holds many chunks, and a pack whose chunks are only partly unreferenced cannot simply be removed, so prune rewrites it: the chunks that are still needed are read out and written into a new pack, and only then is the old pack deleted. That is why the cost of prune is proportional to the repository rather than to the number of snapshots removed, why it holds an exclusive lock for its whole duration, and why it belongs on its own schedule instead of on the tail of every backup.

Two more properties of the policy engine are worth knowing before writing one. The first is the grouping already mentioned: because retention is evaluated per group rather than across the whole repository, --keep-last 1 means one snapshot per host-and-path group, not one snapshot in the repository, and a policy that was verified against a single host quietly changes meaning the day a second host starts writing into it. The second is that the keep rules combine additively rather than restrictively: a snapshot survives if any rule wants it, so --keep-daily 7 alongside --keep-weekly 5 keeps the union of the two sets and never the intersection. Both properties push in the same direction. A retention policy cannot be read for correctness on its own; it has to be evaluated against the actual snapshot population, which is precisely what a dry-run does and what reasoning about the flags does not.

Locks, stale locks, and the automated schedule

Restic takes a lock on the repository before it works on it, which is why the check output quoted elsewhere in this course opens with create exclusive lock for repository. The repository format defines two kinds. A non-exclusive lock lets concurrent readers and writers coexist, which is what allows several hosts to back up into one repository at the same time. An exclusive lock excludes everything else for its duration, and the set of operations that take one is wider than it first looks: prune needs it because it rewrites repository structure, and restic check took one in that same capture simply to guarantee that nothing moved underneath it while it was reading.

Locks are files in the repository, and they are refreshed by the process that holds them. A process that is killed, or a host that loses power or its network mid-run, leaves a lock behind with nothing refreshing it. That is a stale lock, and its effect on an automated schedule is specific: the next scheduled run cannot proceed, and it fails in a way that repeats every night at the same time until someone intervenes. restic unlock removes stale locks and is the correct remedy — after establishing that the lock really is stale.

The order matters, because the alternative failure is worse than a missed night. A lock held by a prune that is still running is not stale, and clearing it defeats the exclusion that operation depends on. Before unlocking, confirm on the host named in the failure that no restic process is still alive, and that no long-running maintenance is in flight. Then unlock, then run the backup, and record that the repository spent a night unprotected.

set -euo pipefail
STAMP=/var/lib/restic/last-read-data-check
restic backup /srv/app/data --tag daily
restic forget --keep-daily 7 --keep-weekly 5 --dry-run
restic check --read-data
date -u +%FT%TZ > "$STAMP"

With set -euo pipefail, the timestamp is written only if every command before it exited 0, which makes the file itself the answer to the question that matters during an incident: when did this repository last have its data read and verified? A schedule that swallows exit codes cannot answer it, and neither can a dashboard that only counts completed backups.

Production discipline

1. Dry-run every retention change before it deletes anything. The measured restic forget --keep-last 1 --dry-run printed Would have removed the following snapshots and {3fe43af4} while removing nothing, which is the only cheap moment to discover that a policy names the wrong survivor.
2. Check the exit code, never the directory. The damaged-pack restore wrote a tree, reported Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) and exited 1, leaving a corrupt db/data.bin on disk at a plausible size.
3. Record when the last full verification finished, not that one is scheduled. Repository integrity is reported as the age of the newest completed restic check --read-data; between two of them, undetected corruption is possible and the honest statement names a date.
4. Never estimate recovery time from backup time. On restic 0.18.0, on tmpfs, a second unchanged backup of 400 MiB completed in .73s while restoring the same 400 MiB took 1.07s on the same machine — one date, one dataset, and no figure to carry to other hardware. What carries is the direction: the gap widens as the delta shrinks against the tree.
5. Keep the passphrase off the host being protected, and know who holds it. The keys/ directory contained one file, the master key wrapped under the passphrase; without that passphrase an intact 11 MiB repository answered Fatal: wrong password or no key found with exit code 12.

Cross-course references

• Linux for Production Sysadmins — Part XXXVI (Scheduled Operations) covers how a timer or cron job propagates the exit status of what it ran, which is the mechanism that decides whether the restic exit codes in this lesson ever reach a human, and it is also where a run that hangs on a stale repository lock has to be given a timeout.
• Observability for Production Sysadmins — Part XVIII (Alerting Rules) is where the timestamp file written at the end of the schedule above becomes an alert: the rule that matters for a repository is one on the age of the last completed restic check --read-data, not one on the last backup, because backup completion is the signal this lesson shows to be insufficient.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XII (Secret Management Platforms) addresses the exact gap the keys/ listing exposes here: the repository carries its own encrypted master key but never the passphrase, so the passphrase is a managed secret that must be stored, audited and made retrievable somewhere the protected host cannot take with it.

Quiz