VM snapshot, VM backup, and the difference that matters

Preserving evidence closed the previous part with a discipline about what must not be destroyed while you recover. This part changes the object of recovery from files and repositories to whole machines, and it opens on the vocabulary problem that does more damage in a virtualised estate than any other. Two operations sit next to each other in every hypervisor’s interface. Both produce something you can go back to. The interface presents them as a quick option and a slow one, which invites the conclusion that the quick one is the same thing done faster. It is not the same thing at all, and the difference is visible in where the bytes are the moment each operation finishes.

One caveat belongs at the top rather than in a footnote. No hypervisor was available to this course, so nothing below quotes hypervisor command output. The behaviour of the operations is taken from the vendor documentation cited in the frontmatter, and every measured number is taken from block-storage captures made on LVM and ZFS. That substitution is honest because the accounting being measured is the same accounting a hypervisor snapshot uses, and the lesson says each time which layer produced a figure.

Two operations that share a verb and nothing else

A hypervisor snapshot preserves a point-in-time view of a virtual machine’s disks by arranging that the blocks the guest referenced at that instant are not overwritten. Proxmox VE documents the operation in its chapter on virtual machines, including the option to capture the guest’s RAM so the machine can be returned to a running state rather than to a powered-off disk. What the operation does not do is move any data. When it finishes, the VM’s disks are still exactly one set of bytes on exactly one storage system, and the snapshot is a bookkeeping arrangement inside that same system.

A VM backup reads the contents of those disks and writes an independent copy of them into a repository — a different filesystem, a different appliance, a different account, a different building. Proxmox VE documents this in a separate chapter with a separate tool, and the separation is not a quirk of how the manual is organised. The product of the operation is different in kind. A snapshot’s product is a view. A backup’s product is a copy.

The two are related in practice, which is exactly why the confusion survives review. Most backups of a running guest begin by taking a snapshot, because a snapshot is the cheapest way to obtain a stable surface to read from while the guest keeps writing; the Proxmox backup documentation describes a mode in which the guest continues to run for precisely this reason. An operator watching that sequence sees a snapshot appear at the start and reasonably concludes that the snapshot is the protective act. The distinction is what happens at the end. The backup releases the snapshot and keeps the copy. A snapshot schedule releases nothing and keeps no copy.

That gives a test blunt enough to use in a design review, and it needs no product knowledge to apply. After this operation completes, what new bytes exist, and which storage system holds them? For a snapshot the honest answer is that some old blocks were retained where they already were. For a backup the answer names a second storage system. If the answer to the second half of the question is the name of the storage the VM already lives on, no independence was created, however many recovery points the console is now listing.

The snapshot’s bill is paid by the running guest

A snapshot is free to take and not free to keep, and the shape of the ongoing cost is the part that gets designed around badly. It is not proportional to the size of the VM, and it is not proportional to how many snapshots exist. It is proportional to how much the guest overwrites while the snapshot is held open.

The LVM capture measures that relationship directly. A 600 MiB origin volume carries a small business ledger; a 100 MiB snapshot is taken against it; then the origin — not the snapshot — is rewritten in 10 MiB steps, with the snapshot’s fill read back after each one.

$ lvs -o lv_name,data_percent rbdrvg/data_snap

Nothing is written to the snapshot. Only the origin is rewritten.
after rewriting  10 MiB of the origin: snapshot used 10.12%
after rewriting  20 MiB of the origin: snapshot used 20.17%
after rewriting  30 MiB of the origin: snapshot used 30.22%
after rewriting 100 MiB of the origin: snapshot used 100.00%

Ten mebibytes of rewriting on the origin consumed 10.12% of the snapshot, and the progression stayed slightly worse than one for one all the way to 100.00%. Read the first line of that output again, because it is the whole point: no process wrote to the snapshot at any stage. The consumption is a side effect of the guest doing ordinary work.

Translate that to a virtual machine and the sizing conversation changes. A snapshot on an idle VM costs almost nothing for as long as the VM stays idle. A snapshot on a database VM that overwrites 40 GiB a day costs roughly 40 GiB a day, every day it is left in place, whether anyone looks at it or not. The usual estate mistake is to reason from the disk: “it is a 200 GiB VM, so the snapshot cannot cost more than 200 GiB.” The correct upper bound is the write volume over the snapshot’s lifetime, which on a busy guest passes the disk size without difficulty.

ZFS accounts for the same effect with different vocabulary, which makes it worth showing rather than asserting. In the ZFS capture the @0900 snapshot reported 0B USED at the moment it was taken. After the live dataset was rewritten, with nothing written to the snapshot:

$ zfs list -o name,used,refer -t all -r rbdrprod

--- snapshot space accounting after the origin was rewritten ---
NAME                   USED  REFER
rbdrprod              97.3M    24K
rbdrprod/ledger       97.1M  50.1M
rbdrprod/ledger@0900  47.0M  50.1M

The snapshot’s USED went from 0B to 47.0M without being written to, because it now holds the only reference to blocks the live dataset overwrote. Two unrelated implementations, one behaviour: the recovery point’s storage cost is charged to the writes made after it was taken.

There is a second cost that no capture in this course measured and that the mechanism nonetheless demands. Preserving old blocks means the guest’s write path acquires an extra step for as long as the snapshot exists — a copy of the old contents before an overwrite, or an allocation plus a metadata update instead of an overwrite in place. Either way the first write to each region is more expensive than it would otherwise be, which is why a long-lived snapshot on a busy VM shows up as a latency complaint before it shows up as a capacity one.

An unconsolidated chain is a dependency chain

The second failure mode is structural rather than dimensional. Take a snapshot, then take another without removing the first, and the disk the guest is writing to is no longer a single object. It is a base image plus a delta, plus a second delta, each layer meaningful only in terms of the layer beneath it. QEMU’s disk image documentation describes this arrangement directly: an image can be created with a backing file, so that blocks absent from the top layer are read from the image below it.

The practical consequence is that snapshot count and recovery-point count are not the same number. A chain of six snapshots on a VM is not six independent places to go back to; it is one disk assembled from seven pieces, all in the same storage, where losing or corrupting any piece invalidates every state above it. The chain shares the dependency property of an incremental backup chain, without the property that makes an incremental chain defensible — the backup chain lives in a repository that fails separately from production, and the snapshot chain does not.

There is a related trap in how these objects get named. A snapshot taken before a risky upgrade and left behind afterwards is usually still called before-upgrade, and the name keeps implying that it is a checkpoint someone chose to retain. Six weeks later it is not a checkpoint; it is the bottom of a chain that the running disk depends on. The estate inventory that matters is not a list of snapshots but a list of snapshot ages, and any age measured in weeks is a finding.

What a backup produces that a snapshot cannot

Set against all that, the backup’s product is unexciting and structurally different: bytes in a second place. The ZFS capture makes the difference concrete without needing a hypervisor, because the snapshot and the copy are produced from the same source by the same mechanism, and only their location differs.

$ zpool import; zfs list -t snapshot -r rbdrprod; zfs list -r -t all rbdrbkp

  production backing store fully overwritten, re-attached as /dev/loop5
--- zpool import (can ZFS find anything to import?) ---
no pools available to import

--- can the 09:00 snapshot be reached? ---
cannot open 'rbdrprod': dataset does not exist

--- what the INDEPENDENT backup pool still holds ---
NAME                  USED  REFER
rbdrbkp               100M    24K
rbdrbkp/ledger        100M  50.1M
rbdrbkp/ledger@0900  50.0M  50.1M
rbdrbkp/ledger@0930     0B  50.1M

rbdrprod/ledger@0900 and rbdrbkp/ledger@0900 were the same point in time, created by the same snapshot mechanism, and one of them answered cannot open 'rbdrprod': dataset does not exist while the other was still listed with 50.1M REFER. Nothing about the snapshot technology decided that. The only variable was which device the blocks had been written to.

The LVM capture put the same question to a volume manager and got the blunter version of the answer. Origin and snapshot were extents in one volume group on one physical volume; after the backing device was destroyed and re-attached, vgs and lvs printed no rows at all — not a degraded group, not a missing device, nothing, because the metadata describing the group had been on the volume that was overwritten.

A VM backup is the operation that changes that outcome, and it changes it for reasons that have nothing to do with speed. The copy can outlive the VM, the datastore and the cluster, because none of those three define where it lives. Its retention is set by the repository rather than by the space pressure on production storage, so it does not get pruned by an operator who needs capacity tonight. And it can be restored somewhere the original hypervisor is not, which is the property the rest of this part depends on.

Production discipline

1. Size a snapshot from the guest’s write rate, never from its disk size. The measured relationship is post-snapshot writes to snapshot consumption: 10 MiB of origin rewrite took a 100 MiB LVM snapshot to 10.12%, and a ZFS snapshot nothing was written to reached 47.0M USED. A 200 GiB VM can overrun any reasonable snapshot reservation in a day.
2. Give every snapshot an expiry at the moment it is created. A snapshot taken for an upgrade window is a tool, and a tool left in place for eleven weeks is a chain link the running disk depends on. Alert on snapshot age directly; it is the one number that predicts both the capacity incident and the expensive merge.
3. Count recovery points by storage system, not by console entry. Six snapshots on one datastore are one failure domain listed six times. Write down which storage each entry occupies before counting anything, because that is the column the count depends on.
4. Require every VM to name a repository that is not its datastore. The backup is the operation that produces bytes somewhere else; in the ZFS capture the identical snapshot survived on rbdrbkp and vanished on rbdrprod, and location was the only difference between them.
5. Treat a long-lived snapshot as a scheduled outage waiting to happen. Removal is a merge whose cost grows with everything the guest wrote, so plan it into a window with headroom rather than discovering it when the datastore fills and the deletion has to run immediately.

Cross-course references

• Proxmox VE for Production Operators — Part IX (Virtual Machines) covers taking and rolling back VM snapshots on a real hypervisor, which is the operation this lesson bounds: that course shows how to perform it, and this one states what it does and does not produce.
• Linux for Production Sysadmins — Part XVI (LVM) explains why an origin and its snapshot are allocations inside a single volume group, which is the structural fact behind both numbers quoted here — the 10.12% fill and the empty vgs and lvs output after the physical volume was overwritten.
• Ceph & Distributed Storage for Production Sysadmins — Part XXXVII (RBD Snapshots) applies the same reasoning to the block storage under many virtualisation clusters: an RBD snapshot lives in the pool that holds the image, so the shared-fate question in this lesson is answered there by naming the pool.

Quiz