Skip to main content
RunBook Academy

Backup & DRI · Recovery Objectives, Vocabulary and Failure ModelsFoundations

Seven words that are not synonyms

Foundation⏱ ~28 minlvm2zfsrsync

What you'll learn

  • Define each of the seven mechanisms by the failure it can and cannot recover from
  • Evaluate any protection arrangement with the two questions of independence and history
  • Explain why a snapshot dies with the storage its origin lives on
  • Predict what a scheduled mirror does with a deletion and with an in-place encryption

Prerequisites

Verified against restic 0.19.1 · BorgBackup 1.4.5 · rclone 1.75.0 · MinIO (S3-compatible object storage) RELEASE.2025-09-07T16-13-09Z · OpenZFS 2.4.1 · LVM2 2.03.31(2) · btrfs-progs 6.17.1 · PostgreSQL 18.6 · pgBackRest 2.59.1 · Kubernetes (k3s) and etcd k3s v1.36.3+k3s1, etcd 3.7.1 · Velero 1.18.2 · Docker Engine 29.7.2 · Proxmox Backup Server (documentation only) 4.0.10-1 · Ubuntu (host baseline) 26.04 LTS · 2026-08-28

Not yet marked complete on this device.

The previous lesson took apart the sentence “we have backups” and found three separate claims inside it. This one takes apart the vocabulary those claims are made in. Ask five engineers what protects a filesystem and you will hear backup, snapshot, replica, mirror and archive used as interchangeable names for one arrangement. They are seven mechanisms with seven recovery properties, and most protection gaps are inherited from a sentence in which one word quietly stood in for another.

Independence and history, and nothing else

Two questions distinguish them.

The first is independence: when the thing being protected is destroyed — the device, the volume group, the pool, the host, the credential that reaches it — does the other copy go with it? Independence is not distance and not a different directory; it is the absence of shared fate. Two copies on one physical volume share fate, and so do two copies reachable by one compromised credential.

The second is history: does the mechanism hold a state the primary no longer has, and for how long? A mechanism with history can answer “give me this file as it was before Tuesday”. Without it, the only available answer is “as it is now” — the useless answer once now is the problem.

The two map onto the two classes of failure. Independence covers physical loss: the disk, the array, the rack, the region. History covers logical damage: the deleted directory, the truncated table, the bad migration, the encryption run by someone else. A mechanism providing one is a real control against one class and none at all against the other; a mechanism providing neither is not a backup, however its schedule is named.

Both are answered by enumeration rather than intuition. For independence, list what one event reaches: the volume group the extents are carved from, the pool that resolves the block pointers, the hypervisor presenting both datastores, the account whose credential can delete both destinations. Anything on that list touching both copies is the shared fate, and geography does not shorten it — two copies in two cities behind one control plane still share it. For history, name the oldest state the mechanism can still produce and everything that removes states from it: a retention setting, a ring of overwrites, an exception area filling up, an operator able to expire the series.

A snapshot is a view of the origin, not a second copy of it

The most commonly substituted word is worth testing first, because snapshots genuinely do provide history. A 600 MiB volume data held a ledger, a 100 MiB snapshot data_snap was taken, and an operator truncated the ledger on the origin.

Read-only / Safethe origin has moved on; the snapshot has not
$ md5sum /mnt/snap/orders.csv
origin  orders.csv now:
ORDER-9999,0.00
snapshot orders.csv still:
ORDER-1001,4500.00
ORDER-1002,1250.00
md5 read from snapshot : 9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded at 09:00  : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the snapshot still holds the 09:00 ledger

That is a real capability and the whole of what a snapshot supplies: a readable point-in-time view while the origin moves on. It is enough to undo the truncation and it costs almost nothing when taken.

It has two ways of ending, and the first surprises people. Nothing was written to data_snap; only the origin was rewritten, 10 MiB at a time, with the snapshot’s usage read after each increment.

Data-loss riskrewriting the origin, and what it costs the snapshot
$ lvs -o lv_name,data_percent rbdrvg/data_snap
Nothing is written to the snapshot. Only the origin is rewritten.
after rewriting  10 MiB of the origin: snapshot used 10.12%
after rewriting  20 MiB of the origin: snapshot used 20.17%
after rewriting  30 MiB of the origin: snapshot used 30.22%
after rewriting  40 MiB of the origin: snapshot used 40.27%
after rewriting  50 MiB of the origin: snapshot used 50.32%
after rewriting  60 MiB of the origin: snapshot used 60.37%
after rewriting  70 MiB of the origin: snapshot used 70.42%
after rewriting  80 MiB of the origin: snapshot used 80.47%
after rewriting  90 MiB of the origin: snapshot used 90.52%
after rewriting 100 MiB of the origin: snapshot used 100.00%

The relationship is close to one for one: 10 MiB of change on the origin cost 10.12 percent of a 100 MiB snapshot, 50 MiB cost 50.32, and the last increment took it to 100.00. What fills the space is not a copy of the dataset but the superseded version of every block the origin overwrote, so consumption is driven by the origin’s write volume over the snapshot’s lifetime, not by the size of the data. Sizing a snapshot is a bet on change rate: a 600 MiB volume can exhaust a 100 MiB snapshot without growing by a byte.

Losing that bet costs the snapshot. Past 100.00 the kernel logged device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception., the attribute string became swi-I-s--- with I in the fifth position for invalid, and the next mount returned can't read superblock. Note what did not happen: the origin kept accepting writes, and nothing anyone was watching returned an error — the same shape of failure as the green check in the previous lesson.

The second way of ending decides the vocabulary question.

Data-loss riskthe volume group after its only PV was destroyed
$ vgs; lvs
Both origin and snapshot are extents in ONE volume group on ONE PV.
pvs before:
PV         VG     PSize   
/dev/loop5 rbdrvg 1020.00m

Simulating permanent loss of the underlying device:
re-attached the same (now destroyed) backing store as /dev/loop5

--- what survived? ---
vgs:
lvs:

vgs printed nothing and lvs printed nothing. The snapshot did not outlive its origin’s storage by a moment, because it was never a copy: it was a set of copy-on-write extents in the same volume group on the same physical volume. History and no independence makes it a control against the operator and none against the disk.

zfs send changes nothing about the snapshot and everything about the outcome

The instructive comparison starts identically. On the ZFS pool rbdrprod, a dataset was snapshotted at 09:00 and that snapshot serialised into a second pool on a different device.

Configuration changethe same point in time, written into an independent pool
$ zfs send rbdrprod/ledger@0900 | zfs receive rbdrbkp/ledger
exit=0
NAME             USED  REFER
rbdrbkp         50.2M    24K
rbdrbkp/ledger  50.1M  50.1M

The snapshot on rbdrprod and the dataset on rbdrbkp hold the same bytes. The snapshot mechanism is unchanged; one copy is now allocated out of a different pool on a different device, so the destruction test answers differently.

Production then moved on — the 09:30 truncation and the writes after it — and a second send carried the newer point across incrementally, relative to the snapshot both sides already held.

Configuration changethe second point in time, added beside the first
$ zfs send -i @0900 rbdrprod/ledger@0930 | zfs receive rbdrbkp/ledger
exit=0
--- snapshots now held on the backup pool ---
NAME                  USED  REFER
rbdrbkp/ledger@0900  50.0M  50.1M
rbdrbkp/ledger@0930     0B  50.1M

The backup pool now holds two points in time. Then the destruction test is applied to production’s device.

Data-loss riskproduction storage lost; what can still be read
$ zpool import
--- zpool import (can ZFS find anything to import?) ---
no pools available to import

--- can the 09:00 snapshot be reached? ---
cannot open 'rbdrprod': dataset does not exist

--- what the INDEPENDENT backup pool still holds ---
NAME                  USED  REFER
rbdrbkp               100M    24K
rbdrbkp/ledger        100M  50.1M
rbdrbkp/ledger@0900  50.0M  50.1M
rbdrbkp/ledger@0930     0B  50.1M

--- recovering the 09:00 ledger from the backup pool ---
$ zfs clone rbdrbkp/ledger@0900 rbdrbkp/restore0900
ORDER-1001,4500.00
ORDER-1002,1250.00
md5 recovered:     9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded 09:00: 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical to the 09:00 ledger

The snapshot did not survive the pool. The replicated dataset did, and returned the 09:00 ledger byte-identical at the md5 recorded before any of this started. Both halves used the same snapshot mechanism; only the independence differed, which is why “we snapshot hourly” and “we back up hourly” are statements about different failures.

That arrangement answers both questions at once, and each side supplies a different half. Independence comes from the receiving pool being a separate allocation domain on a separate device. History comes from that pool keeping @0900 once @0930 arrives — a retention decision taken on the backup side, not a property of the transfer. Expire each point in time as the next one lands and you keep the same two pools, the same devices and the same command while holding exactly one state — the arrangement the next section takes apart.

The mirror did what it was told, on schedule, twice

Sync and replication fail the other question. A nightly rsync -a --delete mirror is the arrangement most often described as a backup, and its first night is indistinguishable from one. Then an operator deleted a file that was still needed, and 01:00 arrived.

Data-loss riskthe 01:00 run after an operator deleted src/invoice.txt
$ rsync -a --delete src/ mirror/
--- mirror now ---
mirror/orders.csv
mirror/reports/q3.txt
invoice.txt recoverable from the mirror? NO - the mirror deleted it too

Elapsed time between the mistake and the loss of the only other copy:
one scheduled interval. Nobody had to make a second mistake.

The next night the source was encrypted in place, every file renamed .locked, and the same command ran.

Data-loss riskthe 01:00 run after the source was encrypted in place
$ rsync -a --delete src/ mirror/
--- mirror after that run ---
mirror/orders.csv.locked
mirror/reports/q3.txt.locked

--- is the mirrored ledger still readable business data? ---
0000000   S   a   l   t   e   d   _   _   < 370 351 357 256   N   J   t

The capture’s closing count is plaintext copies of the ledger remaining anywhere: 0. Nothing malfunctioned. --delete exists to make the destination match the source, and the mirror did exactly that, on schedule, with exit code 0. A mirror is independent — it survives the loss of the primary device — and has no history at all, so every logical injury reaches it within one interval. Continuous replication is the same trade with a shorter interval, which makes the physical protection better and the logical protection worse.

The seven words, and what each one can hand back

Against those two questions the vocabulary stops being a matter of taste.

MechanismProtects againstDoes not protect againstHistory
Copyloss of the original, if it sits elsewhereeverything after the moment it was takenone point, until overwritten
Syncloss of the device on one sidedeletion, corruption and encryption, which it propagatesnone
Replicationloss of the primary node, array or sitelogical damage, reproduced within the lagnone beyond the lag
Snapshotlogical damage after it was taken, while its storage livesloss of that storage, and exhaustion of its own spaceyes, inside one failure domain
Versioningoverwrite and deletion of an object in the storedestruction of the store, or credentialled access to ityes, per object, under lifecycle rules
Archivenothing about the live system; it usually is the only copyeverything — it is a workload that itself needs backing uphistory of records, not of system state
Backupphysical loss and logical damage within its retentionanything older than its oldest restore pointyes, and held independently

Two rows deserve their own sentence. Archive is the word most often used as a synonym for backup and is closest to its opposite: archiving usually moves data out of the primary system, so the archived copy is frequently the only instance in existence — a single point of failure holding the records with the longest legal life. Versioning depends on who can reach the store: history an attacker’s credential can delete is history only until the attacker arrives.

Only the last row answers both, and it earns that not from a product but by being independent of the primary and retaining more than one point in time.

What to take from this

  • The LVM origin and its snapshot were extents in ONE volume group on ONE PV; after that PV was destroyed, vgs and lvs both printed nothing. A snapshot has history and no independence.
  • Rewriting the LVM origin cost the snapshot almost byte for byte: 10.12 percent used at 10 MiB, 50.32 at 50 MiB, 100.00 at 100 MiB, then swi-I-s--- with Invalidating snapshot: Unable to allocate exception. in the ring buffer.
  • After the ZFS production device was overwritten, zpool import reported no pools available to import and the snapshot answered cannot open 'rbdrprod': dataset does not exist.
  • The independent pool held both rbdrbkp/ledger@0900 and rbdrbkp/ledger@0930, and returned the 09:00 ledger byte-identical at md5 9eb4e2ad8e08e1dcaaf87ababab964b0: same mechanism, different failure domain, opposite outcome.
  • rbdrprod/ledger@0900 reported 0B used when taken and 47.0M after the live dataset moved on, with nothing written to it. Snapshot cost is driven by change on the origin.
  • The rsync -a --delete mirror removed the deleted file within one scheduled interval, then propagated the encrypted .locked files, ending at plaintext copies of the ledger remaining anywhere: 0.

Cross-course references

  • Linux for Production Sysadmins — Part XVI (LVM) builds the physical volume, volume group and logical volume layering that this lesson’s snapshot test destroyed; it is where you see why an origin and its snapshot are extents in one allocation pool, the mechanical reason vgs printed nothing.
  • Ceph & Distributed Storage for Production Sysadmins — Part XXIII (Replication) explains what replication guarantees, namely durability against device and host loss; that is exactly the guarantee the rsync capture here satisfies while handing the deletion and the encryption straight to the second copy.
  • Proxmox VE for Production Operators — Part VI (ZFS) covers snapshots on the hypervisor’s storage layer, and this lesson supplies the reason a guest snapshot is not a guest backup until a send stream has placed that point in time on a pool able to survive the first one.

Quiz

Knowledge check · 5 questions

  1. Q1. An hourly LVM snapshot of `data` is taken as `data_snap` in volume group `rbdrvg`, which has one physical volume. That volume fails permanently. What remains recoverable?

  2. Q2. A file server is replicated continuously to a second site with sub-minute lag, and the team calls the replica their backup. An operator deletes a directory at 14:02. What does the second site hold at 14:05?

  3. Q3. A file is deleted by mistake in the morning, and that afternoon the array holding the live data is permanently lost. Which arrangements can still return the file? Select all that apply.

  4. Q4. Nothing was written to the ZFS snapshot `rbdrprod/ledger@0900`, yet its reported USED grew from 0B to 47.0M.

  5. Q5. A colleague argues that because the ZFS snapshot and the received dataset came from the same mechanism, the snapshot is a backup too. State what the capture showed and why that does not follow.

Passing score: 75%. Answers are checked in this browser.