Btrfs subvolumes and snapshots

An LVM snapshot turned out to be a set of copy-on-write exceptions living in the same volume group as its origin, which is why the previous lesson ended with both of them gone at the same moment. Btrfs reaches the same capability from a different direction, and the difference is not cosmetic. On LVM the thing you snapshot is a logical volume you had to size in advance. On Btrfs the thing you snapshot is a subvolume, it costs almost nothing to create, and — this is the part that matters for recovery — it can be serialised and shipped to a completely different filesystem. That last property is what turns a snapshot into something a backup architecture can build on, and it is available only under one condition.

A subvolume is a snapshottable tree, not a directory

A subvolume is a separate file tree with its own root inside a single Btrfs filesystem. It is not a partition: it has no fixed size, and it draws from the same pool of free space as every other subvolume on the device. What separates it from an ordinary directory is that it is addressable as a unit — it carries an ID and a generation number, and it is the only kind of path that btrfs subvolume snapshot will accept.

$ btrfs subvolume create /mnt/prod/ledger && btrfs subvolume list /mnt/prod

Create subvolume '/mnt/prod/ledger'
ID 256 gen 9 top level 5 path ledger

ID 256 names the subvolume, gen 9 is the filesystem generation at which it last changed, and top level 5 is the ID of the tree it hangs under — 5 being the top-level subvolume that every Btrfs filesystem has. The path is where it happens to be mounted into the namespace, and that is the only part of the line that resembles a directory.

Read those two numbers as local rather than portable. When this same subvolume is reconstructed on the backup filesystem further down, it arrives there as ID 256 as well — not because the number travelled with the data, but because that filesystem happened to be handing out 256 next. An ID names an object within one device and says nothing about its content, so a catalogue keyed on a subvolume ID rather than on a path and a snapshot name eventually points at the wrong tree with nothing raising an error.

The consequence for backup design is direct and unforgiving: the set of things you are able to capture at a point in time is exactly the set of subvolumes that exist. If application data sits in a plain directory inside the root subvolume, then the smallest unit containing that data is the whole root — the operating system, the package database, the logs, the caches and whatever a package manager left half-written. Every snapshot carries all of it, every send stream transmits all of it, and every retention decision applies to all of it at once. You cannot draw the boundary later without moving the data across it.

Nesting behaves as a barrier in both directions, and the upstream documentation is explicit about it: snapshotting is not recursive, so a nested subvolume is not captured when its parent is snapshotted. What the snapshot holds in its place is an empty directory. That is useful when you deliberately want caches or scratch space excluded from a root snapshot, and it is a quiet disaster when somebody made /var/lib/postgresql a subvolume and the team believes a root-subvolume snapshot contains the database.

Measured: the 09:00 ledger, held by a read-only snapshot

The capture below runs the same scenario as the LVM lesson so the two can be compared directly: a ledger subvolume with two orders in it, snapshotted at 09:00, and an operator who truncates the live copy at 09:30.

$ btrfs subvolume snapshot -r /mnt/prod/ledger /mnt/prod/ledger-0900

Create readonly snapshot of '/mnt/prod/ledger' in '/mnt/prod/ledger-0900'

>>> exit code: 0

The snapshot is created immediately and it is created cheaply, because nothing is copied. The new root points at exactly the extents the source root points at, and the two only begin to diverge when one of them is written to.

$ btrfs filesystem df /mnt/prod

Data, single: total=56.00MiB, used=40.00MiB
Metadata, DUP: total=32.00MiB, used=208.00KiB

Forty mebibytes of data are in use, and taking a full point-in-time copy of the ledger did not move that figure. Then the live copy is truncated and the two paths are compared.

$ cat /mnt/prod/ledger/orders.csv /mnt/prod/ledger-0900/orders.csv

live     : ORDER-9999,0.00
snapshot : ORDER-1001,4500.00
ORDER-1002,1250.00
MATCH - the snapshot still holds the 09:00 ledger

The live subvolume now contains one worthless row; the snapshot still contains both orders, and the comparison against the checksum recorded at 09:00 reported MATCH. That is the one thing a snapshot genuinely provides, and it is what LVM provided too: a readable point-in-time view while the source moves on. What Btrfs adds is that the view is enforced, not merely conventional.

$ sh -c 'echo tampered > /mnt/prod/ledger-0900/orders.csv'

sh: 1: cannot create /mnt/prod/ledger-0900/orders.csv: Read-only file system

>>> exit code: 2

Read-only file system is the shell reporting EROFS, and the refusal came from the subvolume being read-only rather than from any permission on the file. Root cannot write to it, a compromised application cannot write to it, and a backup agent that mounts it to read files cannot accidentally modify it. That is a materially different guarantee from a writable snapshot everybody has agreed not to touch: an agreement is only as good as the least careful process holding the path, and this refusal does not depend on who is asking.

Why btrfs send refuses a writable source

btrfs send does not copy a directory tree. It serialises a subvolume into a stream of instructions — create this file, set these attributes, write these bytes at these offsets, clone this range from a range that already exists — that a receiver replays to reconstruct the same tree. For that stream to be meaningful, the tree it describes has to have existed, in that exact state, at a single instant. A writable source cannot offer that. The encoder would be walking a tree that the application is still changing, and the resulting stream would describe a state that was never true of anything.

So btrfs send requires its source subvolume to be read-only, and the command’s documentation states that as a requirement rather than as a recommendation. The read-only flag is not a safety suggestion bolted onto the feature; it is the precondition that makes the output well-defined.

One corollary is easy to miss: the snapshot has to stay on the source for the whole duration of the send, because the encoder is reading from it. Where snapshots also bound capacity, retention and replication compete for it.

send and receive across a filesystem boundary

In the capture, /mnt/prod is /dev/loop5 and /mnt/bkp is /dev/loop6 — a separate filesystem on a separate device. That distinction is the entire point of the exercise, and the one the LVM snapshot could not offer.

$ btrfs send /mnt/prod/ledger-0900 | btrfs receive /mnt/bkp

At subvol ledger-0900

>>> exit code: 0

$ btrfs subvolume list /mnt/bkp
ID 256 gen 10 top level 5 path ledger-0900

The receiving filesystem now holds a subvolume of its own — its own ID, its own generation — reconstructed from the stream. It is not a reference to anything on /mnt/prod. In production this is where a wrapper script belongs, because the pipeline has two commands that can each fail:

set -euo pipefail

SRC=/mnt/prod/ledger
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
SNAP="/mnt/prod/.snapshots/ledger-${STAMP}"

btrfs subvolume snapshot -r "$SRC" "$SNAP"
btrfs send "$SNAP" | btrfs receive /mnt/bkp

Without pipefail the shell reports only the exit status of the last command in the pipeline, so a btrfs send that dies part way through is invisible to the scheduler for as long as btrfs receive returns 0 — the job is recorded as a success on the strength of the wrong process’s status. set -e on its own does not close that gap, because the pipeline as a whole did not fail. The second and every subsequent run should send only the difference:

PARENT=/mnt/prod/.snapshots/ledger-0900
CHILD=/mnt/prod/.snapshots/ledger-1000

btrfs send -p "$PARENT" "$CHILD" | btrfs receive /mnt/bkp

$ btrfs send -p /mnt/prod/ledger-0900 /mnt/prod/ledger-1000 | btrfs receive /mnt/bkp

At snapshot ledger-1000

>>> exit code: 0

$ btrfs subvolume list /mnt/bkp
ID 256 gen 13 top level 5 path ledger-0900
ID 257 gen 14 top level 5 path ledger-1000

Two subvolumes now exist on the backup filesystem, and the second was transmitted as a difference against the first. The operational obligation this creates is the same one every incremental chain creates: the parent has to be kept on both sides. Delete ledger-0900 from the source and the next incremental against it has no basis to compute from; delete it from the destination and the assertion -p makes is no longer true.

One detail across those two listings deserves care, because monitoring gets built on it. Immediately after the full send, ledger-0900 on the backup filesystem is listed at gen 10; once the incremental has been received the same subvolume is listed at gen 13. The 09:00 ledger inside it did not change. Generations are counters belonging to the filesystem holding the subvolume — the source’s ledger-0900 carried neither number — so a freshness check that compares a generation on one filesystem against one on another is comparing unrelated counters. The snapshot name and a digest taken on both sides are what answer that question.

The destruction test: which copies were still there

The previous lesson ended by destroying the storage the origin lived on. The same test is applied here; it is the only one that distinguishes a snapshot from a backup.

$ mount /dev/loop5 /mnt/prod

mount: /mnt/prod: wrong fs type, bad option, bad superblock on /dev/loop5, missing codepage or helper program, or other error.
     dmesg(1) may have more information after failed mount system call.

>>> exit code: 32

Both snapshots that lived on /mnt/prod — ledger-0900 and ledger-1000 — went with the filesystem, and so did the live ledger subvolume they were taken from. They were roots in the same tree, on the same device, and nothing about their read-only flag made them independent of it. The two subvolumes that had been sent were somewhere else:

$ cat /mnt/bkp/ledger-0900/orders.csv

ORDER-1001,4500.00
ORDER-1002,1250.00
recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
09:00 md5     : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical to the 09:00 ledger

Same mechanism, different independence. The snapshot and the sent copy contain the same bytes and came from the same command family; only one was still reachable after a single device was lost.

Space a snapshot will not give back

The cheapness of a Btrfs snapshot is a claim about creation, not about retention. Because the snapshot’s root references the same extents as the source, those extents cannot be freed while the snapshot exists. Delete a large file from the live subvolume and the free-space figure need not move at all, because the snapshot taken this morning still points at every extent that file occupied. The space is released when the last reference to it is dropped — in practice when the last snapshot holding it is deleted, not when the file is unlinked.

The pathological case is a workload that rewrites the same file repeatedly: each rewrite allocates new extents while every snapshot in the retention window pins the previous ones, so the file has one name and one size and the filesystem is holding as many versions of it as there are snapshots.

This is also where the reporting tools stop agreeing. btrfs filesystem df reports how space has been allocated to block group types and profiles — which is exactly what the Data, single and Metadata, DUP lines above are — so it answers a question about allocation, not about which tree holds which extents. Ordinary du sums the blocks of the files it walks with no notion of extents shared between subvolumes, so adding a subvolume and its snapshots together counts every shared extent once per tree and overstates what removing any of them would return. The command documented for the question operations actually asks is btrfs filesystem du, which separates total bytes from exclusive bytes — those referenced by nothing else — and reports a shared figure for the set it was pointed at. Exclusive bytes are the only ones a deletion releases.

The operational consequence is that a Btrfs estate can run out of space in a way that looks unreasonable from the outside: a filesystem whose visible files account for a fraction of its capacity, whose administrators have deleted plenty, and which is nonetheless pinned at full by a retention policy nobody re-read. Snapshot retention on Btrfs is a capacity decision as much as a recovery decision, and the two are set by the same number — which is the reason to alert on the exclusive figure of the snapshot set rather than on free space alone. Free space tells you that you are in trouble; the exclusive figure tells you which snapshot to remove to get out of it, and how much that would buy.

What to take from this

• The unit of snapshotting is the subvolume, not the directory. The capture’s production data was ID 256 gen 9 top level 5 path ledger, and only a path like that can be given to btrfs subvolume snapshot.
• btrfs subvolume snapshot -r created the 09:00 snapshot with exit code 0 and did not increase used=40.00MiB in Data, single, because a snapshot shares every extent with its source.
• After the live ledger was truncated to ORDER-9999,0.00, the snapshot still returned both original rows and the comparison reported MATCH.
• A write into the read-only snapshot was refused with Read-only file system at exit code 2, so the point-in-time view is enforced by the filesystem rather than by convention.
• btrfs send requires a read-only source because the stream must describe one tree that cannot change while it is walked; the full send and the -p incremental both exited 0, leaving ID 256 gen 13 ... ledger-0900 and ID 257 gen 14 ... ledger-1000 on the second filesystem.
• When the production device was overwritten, mount failed with exit code 32 and both on-filesystem snapshots were gone; the sent copies returned md5 9eb4e2ad8e08e1dcaaf87ababab964b0, identical to the 09:00 recording.

Cross-course references

• Linux for Production Sysadmins — Part XIV (Filesystems) covers how a filesystem is laid out on a device and mounted into the namespace, which is the layer at which the subvolume boundaries in this lesson have to be decided; Part XVI (LVM) supplies the volume-level snapshot model this lesson is contrasted against, where the unit is a pre-sized logical volume rather than a free-floating tree.
• Proxmox VE for Production Operators — Part VI (ZFS) presents the same snapshot-then-send architecture on a different copy-on-write filesystem, where the dataset plays the role the subvolume plays here and zfs send imposes the same requirement for a source that cannot change underneath it, so the design reasoning transfers directly while the commands do not.
• Ceph & Distributed Storage for Production Sysadmins — Part XXXVII (RBD Snapshots) applies the identical caution at the cluster layer: an RBD snapshot lives in the same pool as the image it captures, so it fails the destruction test in this lesson for exactly the same reason a Btrfs snapshot on /mnt/prod did.

Quiz