Backup & DRI · Recovery Objectives, Vocabulary and Failure ModelsFoundations
Seven words that are not synonyms
What you'll learn
- Define each of the seven mechanisms by the failure it can and cannot recover from
- Evaluate any protection arrangement with the two questions of independence and history
- Explain why a snapshot dies with the storage its origin lives on
- Predict what a scheduled mirror does with a deletion and with an in-place encryption
Prerequisites
Verified against restic 0.19.1 · BorgBackup 1.4.5 · rclone 1.75.0 · MinIO (S3-compatible object storage) RELEASE.2025-09-07T16-13-09Z · OpenZFS 2.4.1 · LVM2 2.03.31(2) · btrfs-progs 6.17.1 · PostgreSQL 18.6 · pgBackRest 2.59.1 · Kubernetes (k3s) and etcd k3s v1.36.3+k3s1, etcd 3.7.1 · Velero 1.18.2 · Docker Engine 29.7.2 · Proxmox Backup Server (documentation only) 4.0.10-1 · Ubuntu (host baseline) 26.04 LTS · 2026-08-28
The previous lesson took apart the sentence “we have backups” and found three separate claims inside it. This one takes apart the vocabulary those claims are made in. Ask five engineers what protects a filesystem and you will hear backup, snapshot, replica, mirror and archive used as interchangeable names for one arrangement. They are seven mechanisms with seven recovery properties, and most protection gaps are inherited from a sentence in which one word quietly stood in for another.
Independence and history, and nothing else
Two questions distinguish them.
The first is independence: when the thing being protected is destroyed — the device, the volume group, the pool, the host, the credential that reaches it — does the other copy go with it? Independence is not distance and not a different directory; it is the absence of shared fate. Two copies on one physical volume share fate, and so do two copies reachable by one compromised credential.
The second is history: does the mechanism hold a state the primary no longer has, and for how long? A mechanism with history can answer “give me this file as it was before Tuesday”. Without it, the only available answer is “as it is now” — the useless answer once now is the problem.
The two map onto the two classes of failure. Independence covers physical loss: the disk, the array, the rack, the region. History covers logical damage: the deleted directory, the truncated table, the bad migration, the encryption run by someone else. A mechanism providing one is a real control against one class and none at all against the other; a mechanism providing neither is not a backup, however its schedule is named.
Both are answered by enumeration rather than intuition. For independence, list what one event reaches: the volume group the extents are carved from, the pool that resolves the block pointers, the hypervisor presenting both datastores, the account whose credential can delete both destinations. Anything on that list touching both copies is the shared fate, and geography does not shorten it — two copies in two cities behind one control plane still share it. For history, name the oldest state the mechanism can still produce and everything that removes states from it: a retention setting, a ring of overwrites, an exception area filling up, an operator able to expire the series.
A snapshot is a view of the origin, not a second copy of it
The most commonly substituted word is worth testing first, because snapshots
genuinely do provide history. A 600 MiB volume data held a ledger, a 100 MiB
snapshot data_snap was taken, and an operator truncated the ledger on the
origin.
$ md5sum /mnt/snap/orders.csvorigin orders.csv now:
ORDER-9999,0.00
snapshot orders.csv still:
ORDER-1001,4500.00
ORDER-1002,1250.00
md5 read from snapshot : 9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded at 09:00 : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the snapshot still holds the 09:00 ledgerThat is a real capability and the whole of what a snapshot supplies: a readable point-in-time view while the origin moves on. It is enough to undo the truncation and it costs almost nothing when taken.
It has two ways of ending, and the first surprises people. Nothing was written
to data_snap; only the origin was rewritten, 10 MiB at a time, with the
snapshot’s usage read after each increment.
$ lvs -o lv_name,data_percent rbdrvg/data_snapNothing is written to the snapshot. Only the origin is rewritten.
after rewriting 10 MiB of the origin: snapshot used 10.12%
after rewriting 20 MiB of the origin: snapshot used 20.17%
after rewriting 30 MiB of the origin: snapshot used 30.22%
after rewriting 40 MiB of the origin: snapshot used 40.27%
after rewriting 50 MiB of the origin: snapshot used 50.32%
after rewriting 60 MiB of the origin: snapshot used 60.37%
after rewriting 70 MiB of the origin: snapshot used 70.42%
after rewriting 80 MiB of the origin: snapshot used 80.47%
after rewriting 90 MiB of the origin: snapshot used 90.52%
after rewriting 100 MiB of the origin: snapshot used 100.00%The relationship is close to one for one: 10 MiB of change on the origin cost
10.12 percent of a 100 MiB snapshot, 50 MiB cost 50.32, and the last
increment took it to 100.00. What fills the space is not a copy of the
dataset but the superseded version of every block the origin overwrote, so
consumption is driven by the origin’s write volume over the snapshot’s
lifetime, not by the size of the data. Sizing a snapshot is a bet on change
rate: a 600 MiB volume can exhaust a 100 MiB snapshot without growing by a byte.
Losing that bet costs the snapshot. Past 100.00 the kernel logged
device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.,
the attribute string became swi-I-s--- with I in the fifth position for
invalid, and the next mount returned can't read superblock. Note what did
not happen: the origin kept accepting writes, and nothing anyone was watching
returned an error — the same shape of failure as the green check in the
previous lesson.
The second way of ending decides the vocabulary question.
$ vgs; lvsBoth origin and snapshot are extents in ONE volume group on ONE PV.
pvs before:
PV VG PSize
/dev/loop5 rbdrvg 1020.00m
Simulating permanent loss of the underlying device:
re-attached the same (now destroyed) backing store as /dev/loop5
--- what survived? ---
vgs:
lvs:vgs printed nothing and lvs printed nothing. The snapshot did not outlive
its origin’s storage by a moment, because it was never a copy: it was a set of
copy-on-write extents in the same volume group on the same physical volume.
History and no independence makes it a control against the operator and none
against the disk.
zfs send changes nothing about the snapshot and everything about the outcome
The instructive comparison starts identically. On the ZFS pool rbdrprod, a
dataset was snapshotted at 09:00 and that snapshot serialised into a second pool
on a different device.
$ zfs send rbdrprod/ledger@0900 | zfs receive rbdrbkp/ledgerexit=0
NAME USED REFER
rbdrbkp 50.2M 24K
rbdrbkp/ledger 50.1M 50.1MThe snapshot on rbdrprod and the dataset on rbdrbkp hold the same bytes.
The snapshot mechanism is unchanged; one copy is now allocated out of a
different pool on a different device, so the destruction test answers
differently.
Production then moved on — the 09:30 truncation and the writes after it — and a second send carried the newer point across incrementally, relative to the snapshot both sides already held.
$ zfs send -i @0900 rbdrprod/ledger@0930 | zfs receive rbdrbkp/ledgerexit=0
--- snapshots now held on the backup pool ---
NAME USED REFER
rbdrbkp/ledger@0900 50.0M 50.1M
rbdrbkp/ledger@0930 0B 50.1MThe backup pool now holds two points in time. Then the destruction test is applied to production’s device.
$ zpool import--- zpool import (can ZFS find anything to import?) ---
no pools available to import
--- can the 09:00 snapshot be reached? ---
cannot open 'rbdrprod': dataset does not exist
--- what the INDEPENDENT backup pool still holds ---
NAME USED REFER
rbdrbkp 100M 24K
rbdrbkp/ledger 100M 50.1M
rbdrbkp/ledger@0900 50.0M 50.1M
rbdrbkp/ledger@0930 0B 50.1M
--- recovering the 09:00 ledger from the backup pool ---
$ zfs clone rbdrbkp/ledger@0900 rbdrbkp/restore0900
ORDER-1001,4500.00
ORDER-1002,1250.00
md5 recovered: 9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded 09:00: 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical to the 09:00 ledgerThe snapshot did not survive the pool. The replicated dataset did, and returned the 09:00 ledger byte-identical at the md5 recorded before any of this started. Both halves used the same snapshot mechanism; only the independence differed, which is why “we snapshot hourly” and “we back up hourly” are statements about different failures.
That arrangement answers both questions at once, and each side supplies a
different half. Independence comes from the receiving pool being a separate
allocation domain on a separate device. History comes from that pool keeping
@0900 once @0930 arrives — a retention decision taken on the backup side,
not a property of the transfer. Expire each point in time as the next one lands
and you keep the same two pools, the same devices and the same command while
holding exactly one state — the arrangement the next section takes apart.
The mirror did what it was told, on schedule, twice
Sync and replication fail the other question. A nightly rsync -a --delete
mirror is the arrangement most often described as a backup, and its first night
is indistinguishable from one. Then an operator deleted a file that was still
needed, and 01:00 arrived.
$ rsync -a --delete src/ mirror/--- mirror now ---
mirror/orders.csv
mirror/reports/q3.txt
invoice.txt recoverable from the mirror? NO - the mirror deleted it too
Elapsed time between the mistake and the loss of the only other copy:
one scheduled interval. Nobody had to make a second mistake.The next night the source was encrypted in place, every file renamed
.locked, and the same command ran.
$ rsync -a --delete src/ mirror/--- mirror after that run ---
mirror/orders.csv.locked
mirror/reports/q3.txt.locked
--- is the mirrored ledger still readable business data? ---
0000000 S a l t e d _ _ < 370 351 357 256 N J tThe capture’s closing count is plaintext copies of the ledger remaining anywhere: 0. Nothing malfunctioned. --delete exists to make the destination
match the source, and the mirror did exactly that, on schedule, with exit code
0. A mirror is independent — it survives the loss of the primary device — and
has no history at all, so every logical injury reaches it within one interval.
Continuous replication is the same trade with a shorter interval, which makes
the physical protection better and the logical protection worse.
The seven words, and what each one can hand back
Against those two questions the vocabulary stops being a matter of taste.
| Mechanism | Protects against | Does not protect against | History |
|---|---|---|---|
| Copy | loss of the original, if it sits elsewhere | everything after the moment it was taken | one point, until overwritten |
| Sync | loss of the device on one side | deletion, corruption and encryption, which it propagates | none |
| Replication | loss of the primary node, array or site | logical damage, reproduced within the lag | none beyond the lag |
| Snapshot | logical damage after it was taken, while its storage lives | loss of that storage, and exhaustion of its own space | yes, inside one failure domain |
| Versioning | overwrite and deletion of an object in the store | destruction of the store, or credentialled access to it | yes, per object, under lifecycle rules |
| Archive | nothing about the live system; it usually is the only copy | everything — it is a workload that itself needs backing up | history of records, not of system state |
| Backup | physical loss and logical damage within its retention | anything older than its oldest restore point | yes, and held independently |
Two rows deserve their own sentence. Archive is the word most often used as a synonym for backup and is closest to its opposite: archiving usually moves data out of the primary system, so the archived copy is frequently the only instance in existence — a single point of failure holding the records with the longest legal life. Versioning depends on who can reach the store: history an attacker’s credential can delete is history only until the attacker arrives.
Only the last row answers both, and it earns that not from a product but by being independent of the primary and retaining more than one point in time.
What to take from this
- The LVM origin and its snapshot were
extents in ONE volume group on ONE PV; after that PV was destroyed,vgsandlvsboth printed nothing. A snapshot has history and no independence. - Rewriting the LVM origin cost the snapshot almost byte for byte:
10.12percent used at 10 MiB,50.32at 50 MiB,100.00at 100 MiB, thenswi-I-s---withInvalidating snapshot: Unable to allocate exception.in the ring buffer. - After the ZFS production device was overwritten,
zpool importreportedno pools available to importand the snapshot answeredcannot open 'rbdrprod': dataset does not exist. - The independent pool held both
rbdrbkp/ledger@0900andrbdrbkp/ledger@0930, and returned the 09:00 ledger byte-identical at md59eb4e2ad8e08e1dcaaf87ababab964b0: same mechanism, different failure domain, opposite outcome. rbdrprod/ledger@0900reported0Bused when taken and47.0Mafter the live dataset moved on, with nothing written to it. Snapshot cost is driven by change on the origin.- The
rsync -a --deletemirror removed the deleted file withinone scheduled interval, then propagated the encrypted.lockedfiles, ending atplaintext copies of the ledger remaining anywhere: 0.
Cross-course references
- Linux for Production Sysadmins — Part XVI (LVM) builds the physical
volume, volume group and logical volume layering that this lesson’s snapshot
test destroyed; it is where you see why an origin and its snapshot are
extents in one allocation pool, the mechanical reason
vgsprinted nothing. - Ceph & Distributed Storage for Production Sysadmins — Part XXIII (Replication) explains what replication guarantees, namely durability against device and host loss; that is exactly the guarantee the rsync capture here satisfies while handing the deletion and the encryption straight to the second copy.
- Proxmox VE for Production Operators — Part VI (ZFS) covers snapshots on the hypervisor’s storage layer, and this lesson supplies the reason a guest snapshot is not a guest backup until a send stream has placed that point in time on a pool able to survive the first one.
Quiz
Knowledge check · 5 questions
Q1. An hourly LVM snapshot of `data` is taken as `data_snap` in volume group `rbdrvg`, which has one physical volume. That volume fails permanently. What remains recoverable?
Q2. A file server is replicated continuously to a second site with sub-minute lag, and the team calls the replica their backup. An operator deletes a directory at 14:02. What does the second site hold at 14:05?
Q3. A file is deleted by mistake in the morning, and that afternoon the array holding the live data is permanently lost. Which arrangements can still return the file? Select all that apply.
Q4. Nothing was written to the ZFS snapshot `rbdrprod/ledger@0900`, yet its reported USED grew from 0B to 47.0M.
Q5. A colleague argues that because the ZFS snapshot and the received dataset came from the same mechanism, the snapshot is a backup too. State what the capture showed and why that does not follow.
Passing score: 75%. Answers are checked in this browser.