The snapshot test: why hourly snapshots may protect nothing

Send and receive earned their place in the previous lesson by moving snapshot content onto storage that fails separately, and that difference — not the snapshot mechanism, which is identical on both sides — is the whole subject here. This lesson takes the single most common protection claim in a virtualised estate, puts one question to it, and answers that question with three measurements on three unrelated implementations rather than with an opinion. The question is short enough to ask in a design review and blunt enough that nobody can answer it with a retention policy.

The question, phrased so a retention policy cannot answer it

Picture an estate that is entirely ordinary. A hypervisor cluster runs the production virtual machines. Every VM disk lives on one storage array, because that is what the array was bought for. A scheduler takes a snapshot of every VM every hour and keeps a week of them, so at any moment there are one hundred and sixty-eight recovery points per machine. The job has never failed. The runbook says, correctly, that any VM can be rolled back to any hour in the last seven days.

Now destroy the array. Not a failed disk, which the array handles, and not a failed controller, which it also handles — the array itself, permanently and completely. A firmware defect that writes garbage through both controllers. A fire in the row. An operator who reinitialises the wrong shelf. An intruder who reached the array’s own management interface and issued the factory reset from there. The failure model does not matter; what matters is that the storage that held the VM disks no longer holds anything.

How many of those one hundred and sixty-eight recovery points remain?

That is the snapshot test, and it has exactly one honest form: name the storage each snapshot physically occupies, then remove that storage and count what is left. It cannot be answered by pointing at the schedule, because the schedule describes when snapshots are created and how long they are kept, and neither of those facts is about where the bytes are. It cannot be answered by pointing at the array’s own redundancy, because redundancy inside a failure domain protects against the failures the domain was designed to absorb and says nothing about losing the domain itself.

The rest of this lesson measures the answer three times. The three implementations share no code, use different vocabulary and account for space in different ways, which is precisely why running the same test on all three is worth the effort: the result is a property of the architecture, not a quirk of one tool.

LVM: after the physical volume is overwritten, vgs and lvs print nothing

The LVM capture is the simplest of the three because its geometry is the most visible. A single loop device, /dev/loop5, is the only physical volume in a volume group named rbdrvg. Inside that group live two logical volumes: the 600 MiB origin data, carrying a small business ledger, and the 100 MiB snapshot data_snap taken against it. Both are extent allocations inside the same volume group, which means both resolve to the same physical volume.

The device is then destroyed in the way real storage is destroyed — the backing store is overwritten and re-attached at the same path, so the device node still exists and answers I/O. Nothing has been unplugged; the bytes are simply not the bytes LVM wrote.

$ pvs; vgs; lvs

Both origin and snapshot are extents in ONE volume group on ONE PV.
pvs before:
PV         VG     PSize   
/dev/loop5 rbdrvg 1020.00m

Simulating permanent loss of the underlying device:
re-attached the same (now destroyed) backing store as /dev/loop5

--- what survived? ---
vgs:
lvs:

Read the last three lines as the measurement they are. vgs printed no rows. lvs printed no rows. Not an error, not a degraded state, not a volume group listed as missing a device — nothing at all, because the LVM metadata that described the group lived on the physical volume that was overwritten. There is no origin to activate and no snapshot to mount.

This is the outcome the earlier LVM lesson set up without stating. A classic LVM snapshot is a table of copy-on-write exceptions plus a pointer to an origin. Both halves of that arrangement were allocated out of the same 1020.00m physical volume. The snapshot was never a second copy of the ledger; it was a partial record of what the ledger used to say, stored beside the ledger, and it was only ever readable while the origin’s extents were readable too.

The emptiness is a fact about discovery, not about damage. LVM keeps no registry of volume groups elsewhere on the host; it finds them by scanning the devices it may look at, reading the label near the start of each one and then the metadata area that label points to. Overwriting /dev/loop5 removed the label, so there was no physical volume, so there was no volume group, so there were no logical volumes. Printing nothing is the correct answer to a question whose subject does not exist, which is exactly why it reads at three in the morning like a broken command rather than a verdict.

The same capture shows the volume group failing a second way that needs no destroyed device at all. Earlier in the run, with /dev/loop5 healthy, 100 MiB of writes to the 600 MiB origin took the 100 MiB snapshot to 100.00%, the kernel logged Invalidating snapshot: Unable to allocate exception., lvs reported swi-I-s---, and the snapshot could no longer be mounted. One event destroyed the container and took the recovery point with it; the other left the container intact and destroyed the recovery point inside it. Neither is visible to a schedule that counts snapshots created.

ZFS: no pools available to import and a dataset that does not exist

ZFS gives the test a sharper edge, because the same capture holds both outcomes. Two pools exist on two devices: rbdrprod on /dev/loop5 carrying the live dataset rbdrprod/ledger, and rbdrbkp on /dev/loop6, which received the 09:00 snapshot by zfs send and the 09:30 state by an incremental send afterwards. Both pools are ONLINE and roughly the same size. From a capacity report they look interchangeable. They are not.

$ zpool list; zpool import; zfs list -t snapshot -r rbdrprod

--- before ---
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rbdrbkp    384M   100M   284M        -         -    12%    26%  1.00x    ONLINE  -
rbdrprod   384M   101M   283M        -         -    17%    26%  1.00x    ONLINE  -
production backing store fully overwritten, re-attached as /dev/loop5
--- zpool import (can ZFS find anything to import?) ---
no pools available to import

--- can the 09:00 snapshot be reached? ---
cannot open 'rbdrprod': dataset does not exist

Those two messages are worth separating. zpool import scans attached devices for ZFS labels and reports what it finds; no pools available to import means it found nothing to reconstruct rbdrprod from. The second message follows inevitably: cannot open 'rbdrprod': dataset does not exist is not a statement about the snapshot, it is a statement about the pool the snapshot lived inside. There is no faulted device to clear, no half-imported pool to force, no degraded vdev to replace. The namespace that contained rbdrprod/ledger@0900 is not present on this host any more.

The independent pool is untouched by all of it.

$ zfs list -r -t all rbdrbkp; zfs clone rbdrbkp/ledger@0900 rbdrbkp/restore0900; md5sum /rbdrbkp/restore0900/orders.csv

--- what the INDEPENDENT backup pool still holds ---
NAME                  USED  REFER
rbdrbkp               100M    24K
rbdrbkp/ledger        100M  50.1M
rbdrbkp/ledger@0900  50.0M  50.1M
rbdrbkp/ledger@0930     0B  50.1M

--- recovering the 09:00 ledger from the backup pool ---
$ zfs clone rbdrbkp/ledger@0900 rbdrbkp/restore0900
ORDER-1001,4500.00
ORDER-1002,1250.00
md5 recovered:     9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded 09:00: 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical to the 09:00 ledger

Both points in time are still there, and the 09:00 ledger comes back with the digest it had before production ever moved past it. Nothing about the snapshot mechanism changed between the two halves of this capture. rbdrbkp/ledger@0900 is a ZFS snapshot in exactly the same sense that rbdrprod/ledger@0900 was. The only difference between the one that vanished and the one that restored is which device its blocks were written to.

Btrfs: wrong fs type, bad option, bad superblock at exit code 32

The Btrfs capture runs the same experiment against a filesystem rather than a volume manager or a pool. Production is /dev/loop5 mounted at /mnt/prod, holding the subvolume ledger and two read-only snapshots of it, ledger-0900 and ledger-1000. Both snapshots were also sent to a separate filesystem on /dev/loop6 mounted at /mnt/bkp — the first in full, the second incrementally with -p. Then the production device is overwritten and re-attached, exactly as before.

$ mount /dev/loop5 /mnt/prod

  production device fully overwritten and re-attached as /dev/loop5

--- can the production filesystem be mounted? ---
mount: /mnt/prod: wrong fs type, bad option, bad superblock on /dev/loop5, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
>>> exit code: 32

That message is the one most likely to be misread at three in the morning. A bad superblock sounds like a filesystem that needs repairing, and the reflex is to reach for btrfs check or a backup superblock. There is nothing to repair. mount is reporting that the bytes on /dev/loop5 are not a Btrfs filesystem, because they are not the bytes Btrfs wrote. The two snapshots that lived on that filesystem were extent references inside it, and they are as absent as the subvolume they were taken from — the read-only flag that stopped writes to ledger-0900 was a property of the subvolume, never a property of the device.

$ btrfs subvolume list /mnt/bkp; cat /mnt/bkp/ledger-0900/orders.csv; md5sum /mnt/bkp/ledger-0900/orders.csv

--- what the independent backup filesystem still holds ---
ID 256 gen 13 top level 5 path ledger-0900
ID 257 gen 14 top level 5 path ledger-1000

--- recovering the 09:00 ledger ---
ORDER-1001,4500.00
ORDER-1002,1250.00
recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
09:00 md5     : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical to the 09:00 ledger

Three implementations, three different error strings, one result. The strings differ because of how each layer goes looking for its container, not because anything different happened to the three of them. LVM scans devices for a label and prints empty tables when it finds none. ZFS scans for pool labels and says no pools available to import, which reads like a search that came back empty because that is what it is. Btrfs expects a superblock at a known offset, finds bytes that are not one, and the mount helper renders the refusal as a list of everything that could explain it.

Each string then recruits a different and wrong instinct: re-run vgs with more privilege, force the import, run a filesystem check. All three assume the container is present and misbehaving, and in all three captures it was absent. No tool that works inside a container can rebuild the container it works inside, so the right response to all three messages is identical — stop working on the destroyed side and go to the independent copy, which is far easier when that copy was named in advance.

The snapshots that shared storage with production went with production. The copies that had been written to a second device came back, and came back byte-identical to what was recorded before the ledger was ever touched.

Retention answers one question; independence answers a different one

The general rule is now measurable rather than rhetorical: a snapshot inherits the failure domain of the storage that holds it. Everything else about snapshots — how cheap they are, how fast they are taken, how granular the schedule is — leaves that inheritance untouched.

Which means a snapshot schedule answers exactly one question well: how far back can I go? Hourly for seven days answers it with “any hour in the last week”, and that is a genuinely useful answer, because the overwhelming majority of data-loss events are not array destruction. They are a truncated table, a bad deployment, an operator who removed the wrong directory. Against those, a recent point-in-time view on fast local storage is the quickest recovery available, and the earlier lessons in this part measured it working.

What the schedule never answers is what happens when this storage is gone? That question is answered only by naming a second storage container and showing that a copy is in it. The two questions are independent, they need different mechanisms, and no amount of the first substitutes for the second. Doubling retention from seven days to thirty adds twenty-three days of rollback depth and adds nothing whatsoever to the array-loss answer.

Applying the test is mechanical once it is phrased this way. For each recovery mechanism, write down the storage container the bytes occupy — this array, this pool, this filesystem, this bucket, this account. Then, for each container, ask what single administrative or physical event removes the whole thing. Two mechanisms that name the same container do not give you two chances; they give you one, counted twice. That is the arithmetic that turns one hundred and sixty-eight recovery points into zero.

Run it against the estate from the opening and the column fills in fast. Hourly VM snapshots: container, the production array. The backup appliance whose repository sits on a LUN carved out of that same array because that was where the capacity was: the production array again, so its retention policy protects nothing the snapshots were not already protecting. Array replication to a second array in the other room: a genuinely different container, worth recording with its limits attached, because it survives the loss of the first array and does not survive a deletion that replication faithfully carries across. The weekly copy that leaves for object storage under separate credentials: a third container, and the only line whose failure has nothing in common with the first two. Four mechanisms, three containers, one still standing afterwards.

What to take from this

• After the LVM physical volume was overwritten, vgs and lvs both returned no rows at all. Origin and snapshot were extents in one volume group on one 1020.00m PV, so neither survived and there was nothing left to repair.
• After the ZFS production device was overwritten, zpool import reported no pools available to import and zfs list reported cannot open 'rbdrprod': dataset does not exist — a statement about the missing pool, not about the snapshot.
• The independent pool rbdrbkp still listed both ledger@0900 and ledger@0930 after that loss, and a clone of rbdrbkp/ledger@0900 returned md5 9eb4e2ad8e08e1dcaaf87ababab964b0, matching the digest recorded at 09:00.
• Mounting the destroyed Btrfs production filesystem failed with wrong fs type, bad option, bad superblock on /dev/loop5 at exit code 32, while the separate filesystem still listed ID 256 ... path ledger-0900 and ID 257 ... path ledger-1000 and restored byte-identical.
• The same mechanism produced opposite outcomes in the ZFS and Btrfs runs, and the only variable was which device the bytes were on. Independence, not the snapshot technology, decided what survived.
• A snapshot schedule answers “how far back can I go” and never “what if this storage is gone”; the second question is answered only by naming a second storage container and proving a copy is in it.

Cross-course references

• Proxmox VE for Production Operators — Part XIII (Proxmox Backup Server) covers writing guest data to a separate backup host, which is the mechanism that changes the answer to this lesson’s test: a Proxmox VM snapshot is stored on the same storage as the VM’s disk and fails with it, whereas a PBS backup occupies a container the hypervisor storage does not define.
• Ceph & Distributed Storage for Production Sysadmins — Part IV (Failure Domains) develops the vocabulary this lesson is applying informally; the snapshot test is a failure-domain question in disguise, and Ceph’s CRUSH model makes explicit the boundary that an array’s internal redundancy hides.
• Linux for Production Sysadmins — Part XVI (LVM) explains why an origin and its snapshot are allocations inside a single volume group, which is exactly the structural fact that made vgs and lvs return nothing once the underlying physical volume was overwritten here.

Quiz