Copy-on-write: the mechanism under every snapshot

Restoring a service onto a clean host ended the previous part with bytes arriving from a repository a backup tool had assembled. This part changes the source: the point-in-time view now comes from the storage layer itself, asked to remember what a volume looked like a moment ago. LVM, Btrfs and ZFS answer that request with three implementations of one idea, and the surprises each of them springs are the same three consequences in different vocabulary.

Nothing moves when the snapshot is taken

The word invites the wrong picture. A photographic snapshot captures light and produces a new object, so a storage snapshot sounds like a copy made quickly. It is not a copy at all. Taking one records that, from this instant, the blocks the live data occupies are referenced by a second view as well — an accounting entry, and one measurable at creation.

$ zfs list -o name,used,refer,avail -t all -r rbdrprod

NAME                   USED  REFER  AVAIL
rbdrprod              50.6M    24K   205M
rbdrprod/ledger       50.1M  50.1M   205M
rbdrprod/ledger@0900     0B  50.1M      -

Read the two columns against each other. REFER is what a name can see: 50.1M for the live dataset and 50.1M for the snapshot, because they see the same bytes. USED is what it is charged for, and the snapshot is charged 0B — a view of 50.1 MiB that costs nothing, because every block in it is one the live dataset is still using and still paying for.

If creating a snapshot writes a reference rather than moving data, the time and space it takes are independent of how much data is referenced: a 40 TB dataset and a 40 GB dataset cost the same entry. That is what makes snapshots the usual foundation for consistency work — the window in which an application must hold still is the time to record a reference.

The bill arrives on the next write to the origin

If nothing is copied at snapshot time, something has to happen later. The trigger is a write to a block both views reference: one wants the new contents, the other the old, a single location cannot serve both, and the old contents must be preserved before the new ones land.

The LVM capture measures that on a deliberately small arrangement: a 100 MiB snapshot, data_snap, of a 600.00m origin, reporting Data% of 0.01 at creation. That 100 MiB is the exception store, its size fixed by the lvcreate -s -L that made it. The origin was then rewritten in 10 MiB increments, and nothing was written to the snapshot.

$ lvs --noheadings -o data_percent rbdrvg/data_snap

Nothing is written to the snapshot. Only the origin is rewritten.
after rewriting  10 MiB of the origin: snapshot used 10.12%
after rewriting  20 MiB of the origin: snapshot used 20.17%
after rewriting  30 MiB of the origin: snapshot used 30.22%
after rewriting  40 MiB of the origin: snapshot used 40.27%
after rewriting  50 MiB of the origin: snapshot used 50.32%
after rewriting  60 MiB of the origin: snapshot used 60.37%
after rewriting  70 MiB of the origin: snapshot used 70.42%
after rewriting  80 MiB of the origin: snapshot used 80.47%
after rewriting  90 MiB of the origin: snapshot used 90.52%
after rewriting 100 MiB of the origin: snapshot used 100.00%

The relationship is linear and close to one for one, each reading a fraction of a point above the round number — the store’s own bookkeeping, accounted alongside the chunks it preserves. Only 100 MiB of the 600 MiB origin was rewritten, and that consumed the whole snapshot.

This is the sentence operators most often hold backwards. Snapshot space is consumed by writes to the origin, not by writes to the snapshot. A snapshot nobody has ever opened will fill on schedule if the origin under it is busy.

Redirect-on-write, and why the accounting still looks the same

Btrfs shows the sharing side plainly. A snapshot shares every extent with its source, and the filesystem usage read straight afterwards shows it.

$ btrfs subvolume snapshot -r /mnt/prod/ledger /mnt/prod/ledger-0900; btrfs filesystem df /mnt/prod

  Create readonly snapshot of '/mnt/prod/ledger' in '/mnt/prod/ledger-0900'
>>> exit code: 0

--- filesystem usage right after the snapshot ---
Data, single: total=56.00MiB, used=40.00MiB
Metadata, DUP: total=32.00MiB, used=208.00KiB

The -r is not cosmetic: creating a file inside the snapshot returned cannot create /mnt/prod/ledger-0900/orders.csv: Read-only file system with exit code 2. Nothing can write to this view, and it will still accumulate cost, because every extent the live subvolume rewrites is one this snapshot becomes the last reference to.

The operator-facing rule therefore survives the change of mechanism. Where the two designs diverge is in where that space comes from, and that decides the failure mode rather than the cost.

The cost is change rate multiplied by retention

The third consequence follows from the first two. In the same ZFS run the operator truncated the ledger, the dataset changed further, and the snapshot was listed again.

$ zfs list -o name,used,refer -t all -r rbdrprod

NAME                   USED  REFER
rbdrprod              97.3M    24K
rbdrprod/ledger       97.1M  50.1M
rbdrprod/ledger@0900  47.0M  50.1M

REFER has not moved: the snapshot still sees the 50.1M it saw at 09:00, which is the point of holding it. USED has gone from 0B to 47.0M, and no byte of that was written by anything touching the snapshot. It measures blocks the live dataset overwrote and the snapshot alone now references.

The sizing rule follows. Snapshot cost is the distinct blocks replaced while the snapshot is held — change rate multiplied by retention — and dataset size appears nowhere in it, which is why the floor is computed from measured bytes of change, not a percentage of the volume:

DAILY_CHANGE_MIB=48
RETAIN_DAYS=6
COW_MIB=$(( DAILY_CHANGE_MIB * RETAIN_DAYS ))
echo "provision at least ${COW_MIB} MiB of copy-on-write space"

The change-rate term decides everything and is the one usually guessed. Measure it over a full business cycle: the night a batch job rewrites tables in place can exceed a quiet week, and it is the peak that invalidates a snapshot, not the average. Retention is less simple than the multiplication suggests once several snapshots are held at once, because a block replaced after three were taken is preserved once and referenced by all three — which is why destroying one can free far less than its own accounting appeared to promise.

When the copy-on-write space runs out

A fixed-size store that fills has to do something, and what LVM does is worth rehearsing first. The capture kept rewriting the origin past the snapshot’s 100 MiB.

$ lvs -o lv_name,lv_size,origin,data_percent,lv_attr rbdrvg; dmesg | grep -i snapshot; mount -o ro /dev/rbdrvg/data_snap /mnt/snap

  LV        LSize   Origin Data%  Attr      
data      600.00m               owi-aos---
data_snap 100.00m data   100.00 swi-I-s---

--- kernel ring buffer (last lines mentioning snapshot) ---
[739220.717450] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.

--- attempt to mount the snapshot again ---
mount: /mnt/snap: can't read superblock on /dev/mapper/rbdrvg-data_snap.
     dmesg(1) may have more information after failed mount system call.
MOUNT FAILED - the snapshot is gone.

The fifth character of the attribute string moved from a to I: invalid. The kernel said why in one line — it could not allocate another exception — and the volume that had held the 09:00 ledger could no longer be mounted.

Two properties of that failure deserve attention. It is not gradual: the snapshot does not degrade into a partial view, it becomes unreadable. And it is silent where anyone is watching, because the origin was never affected — applications kept writing, the service stayed up, and the only thing that ended was the recovery point.

Where that space is the shared pool rather than a fixed store there is nothing separate to invalidate, and exhaustion instead becomes a pool-capacity problem affecting live data.

What to take from this

• Immediately after creation, rbdrprod/ledger@0900 reported USED 0B while REFER showed 50.1M on zfs-2.4.1-1ubuntu5: taking a snapshot costs an accounting entry, not a copy.
• On LVM 2.03.31(2), rewriting 10 MiB of a 600.00m origin consumed 10.12% of a 100.00m snapshot, 20 MiB consumed 20.17%, and 100 MiB consumed 100.00%, with no write issued to the snapshot.
• That same ZFS snapshot grew from 0B to 47.0M USED while its REFER stayed at 50.1M: the charge is for blocks the live data replaced.
• Cost is change rate times retention, and dataset size is absent from the expression — one sixth of the 600 MiB origin, rewritten, filled the whole snapshot.
• When the store filled, the kernel logged Invalidating snapshot: Unable to allocate exception, the attributes read swi-I-s---, and the mount failed with can't read superblock, while the origin carried on unaffected.

Cross-course references

• Linux for Production Sysadmins — Part XVI (LVM) builds the volume group, physical volume and extent model the exception store is allocated from, and is where its size is chosen; the 100.00% and swi-I-s--- above are what that choice looks like when it was made from dataset size rather than change rate.
• Ceph & Distributed Storage for Production Sysadmins — Part XXXVII (RBD Snapshots) applies this same copy-on-write reasoning to a distributed block device, where pre-images are preserved across placement groups rather than in one volume group; the prediction made here — cost follows origin writes and retention — carries over unchanged.
• Proxmox VE for Production Operators — Part VI (ZFS) covers the pool and dataset administration the USED and REFER columns come from, and is where this mechanism turns operational: the 0B to 47.0M growth measured here is how a guest snapshot kept “just in case” becomes a capacity incident.

Quiz