ZFS snapshots, clones and space accounting

Btrfs made the sharing visible but never named its price. btrfs filesystem usage reported Data, single: total=56.00MiB, used=40.00MiB for the whole filesystem — a true statement about the device that says nothing about which subvolume is responsible for which megabyte. ZFS reports the same physics with per-object attribution, and that attribution is why this lesson is on ZFS rather than on the other two. The accounting is not a reporting detail. Misreading it is the ordinary route to a pool that fills up while every dataset in it still looks small.

USED, REFER and AVAIL answer three different questions

These three columns are printed side by side and are routinely read as one number, which is the mistake. Each answers a question the others do not.

REFER is how much data the object references — everything reachable through it, whether or not other objects reference the same blocks. It is the closest thing ZFS prints to “how big is this thing”, and it is the column an administrator expects when asked how much data a dataset holds.

USED is not a size. It is the answer to a different question: how much space would be released if this object were destroyed. Blocks shared with something else are not counted, because destroying this object would not free them. For a dataset, USED also includes the space charged to its own snapshots, which is why a dataset’s USED can exceed its REFER by a wide margin.

AVAIL is how much space the object could still consume, drawn from the pool’s free space and therefore shared with every other dataset in that pool. Two datasets in one pool both report the same AVAIL and are both wrong the moment the other one writes.

The capture starts with a snapshot taken at the instant nothing has diverged.

$ zfs list -t all -o name,used,refer,avail -r rbdrprod

NAME                   USED  REFER  AVAIL
rbdrprod              50.6M    24K   205M
rbdrprod/ledger       50.1M  50.1M   205M
rbdrprod/ledger@0900     0B  50.1M      -

Read it row by row. The dataset rbdrprod/ledger references 50.1M and is charged 50.1M, because at this instant nothing else holds those blocks. The snapshot references the same 50.1M — it can produce every one of those bytes — and is charged 0B, because destroying it would release nothing. The pool root reports 50.6M USED, being the dataset plus a little metadata, and AVAIL is 205M for both objects that can grow, since both draw from the same free space.

The snapshot’s AVAIL is -. A snapshot cannot be written to, so there is no meaningful answer to “how much more could this consume”, and ZFS declines to invent one. That dash is the first hint that USED is going to behave strangely: the object charged nothing and unable to grow is about to become the most expensive thing in the pool.

Measured: 47.0M appeared under a snapshot nobody wrote to

Thirty minutes later an operator truncated the ledger and the dataset kept taking writes. The live data changed; the snapshot, by construction, did not.

$ zfs list -t all -o name,used,refer -r rbdrprod

--- snapshot space accounting after the origin was rewritten ---
NAME                   USED  REFER
rbdrprod              97.3M    24K
rbdrprod/ledger       97.1M  50.1M
rbdrprod/ledger@0900  47.0M  50.1M

Three numbers moved and one did not, and the one that did not is the most informative. rbdrprod/ledger still REFERs 50.1M: the live dataset presents about as much data as it did at 09:00, because a truncate-and-rewrite replaces content rather than adding to it. Its USED, though, went from 50.1M to 97.1M. The snapshot’s USED went from 0B to 47.0M. And 50.1 plus 47.0 is 97.1 — the dataset’s USED is now its own referenced data plus the space charged to its snapshot, exactly as zfsprops(7) describes the usedbysnapshots component.

Nothing was written to rbdrprod/ledger@0900. It cannot be written to. What changed is that the live dataset overwrote blocks the snapshot still references, and those blocks stopped being shared. The moment a block has exactly one holder, ZFS charges it to that holder — so the snapshot inherited 47.0M by doing nothing at all.

This is the sentence to carry out of the lesson: a snapshot is not billed when it is taken, it is billed when its origin changes. The cost of a snapshot is therefore a function of the write rate of the dataset it was taken from, over the interval it is retained. Two identically sized datasets on the same schedule can differ by an order of magnitude in snapshot cost, and nothing in the snapshot configuration explains the difference.

A clone is a writable fork of a snapshot, and sometimes the only way in

A snapshot can be read, but not by writing to it and not through a mount an operator creates for it. OpenZFS exposes snapshot contents through a .zfs/snapshot directory under the dataset root, and whether that directory is visible at all is governed by the snapdir property documented in zfsprops(7). That is a per-dataset setting, so the route can be present on one dataset and absent on the next one in the same pool, decided by whoever created each of them and not by the person who needs the file. It is a thin thing to plan a 03:00 recovery around, and the capture did not plan around it: every read-back in the transcript, on the production pool and later on the backup pool, went through a clone.

A clone depends on none of that. zfs clone creates a new, writable dataset whose starting content is a snapshot’s content, sharing every block with it. It is mounted like any other dataset and behaves like any other dataset, and because it shares rather than copies, it is charged almost nothing at creation and accrues cost only for what is written into it afterwards.

$ zfs clone rbdrprod/ledger@0900 rbdrprod/inspect0900

ORDER-1001,4500.00
ORDER-1002,1250.00
md5 from snapshot: 9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded 09:00: 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH

The clone returned the two original order lines at a checksum identical to the one recorded at 09:00, while the live dataset still held the truncated ledger. Both states were readable at the same time, from the same pool, with no restore step between them. That is the property that makes clones the right instrument for investigation: you compare the two states rather than replacing one with the other.

The full shape of the operation is short enough to write down, and worth writing down with the teardown attached rather than remembered separately.

SNAP=rbdrprod/ledger@0900
CLONE=rbdrprod/inspect0900

zfs clone "$SNAP" "$CLONE"
zfs list -o name,used,refer,origin "$CLONE"
zfs destroy "$CLONE"

The origin property in that listing is the part to look at. It names the snapshot the clone was forked from, and it is the record of a dependency that now exists in the pool.

The dependency runs backwards, and retention meets it first

A clone depends on its origin snapshot for every block it has not itself rewritten. ZFS therefore refuses to destroy a snapshot while a clone of it exists — the operation fails rather than silently orphaning the clone’s data. Short of removing the clone, two documented routes get past that refusal, and they are not interchangeable. zfs-destroy(8) documents -R, which recursively destroys all dependents including cloned filesystems outside the target hierarchy; read that sentence twice before typing it during a capacity incident, because the dependent it removes is a dataset somebody created on purpose and may still be reading from. The other route is zfs promote, documented in zfs-promote(8), which reverses the relationship so that the clone owns the shared blocks and the origin is re-parented beneath it — the move to reach for when the clone has quietly become the dataset you actually want to keep, rather than a thing to delete.

That refusal is correct behaviour and a genuine operational hazard, because of who encounters it. The person who created the clone was investigating something and has moved on. The thing that encounters the pinned snapshot is an automated retention job at 02:00, which tries to destroy snapshots past their age threshold, fails on the pinned one, and — depending entirely on how it was written — either stops there or logs a failure nobody reads.

Everything downstream follows from the first section. The pinned snapshot keeps accruing USED as the live dataset is rewritten. The snapshots taken after it accumulate beside it. The dataset’s USED climbs while its REFER stays flat, so every per-dataset size report an administrator is likely to look at continues to show a normal-sized dataset. AVAIL falls across every dataset in the pool at once, because AVAIL is a pool-wide figure. The alert that eventually fires is a pool capacity alert, several weeks after the cause, and the cause is one forgotten dataset that a person created deliberately and reasonably.

Everything above happened inside one pool

Every mechanism in this lesson — the sharing, the charging, the clone, the dependency — is resolved by metadata that lives in the pool. A snapshot is reachable because the pool can resolve its block references. A clone is readable because its origin’s blocks are still allocated in that pool. The accounting is precise, it is genuinely useful, and it describes a single allocation domain.

That is the boundary. Snapshot count says nothing about how many failure domains hold the data, because the answer is one, and it stays one however many snapshots are retained and however far back they reach. The capture made the point by destroying the production device: zpool import found nothing to import, and reaching for the snapshot returned cannot open 'rbdrprod': dataset does not exist. Forty-seven megabytes of carefully accounted history went with the device that held it.

What changes that is moving a snapshot’s contents into a different pool, on different devices, which is where the next lessons go. The mechanism is the same one taught here — the independence is the part that differs.

What to take from this

• At creation, rbdrprod/ledger@0900 reported 0B USED with 50.1M REFER. USED answers “what would destroying this free”, and the answer was nothing, because every block was still shared with the live dataset.
• After the origin was rewritten, the same snapshot reported 47.0M USED with its REFER unchanged at 50.1M, and nothing had been written to it. A snapshot is billed by its origin’s write rate, not by its own size.
• The dataset moved from 50.1M USED to 97.1M USED while REFER stayed at 50.1M. A dataset’s USED includes the space charged to its snapshots, so the two columns diverge exactly as much as history has diverged.
• AVAIL was 205M for both the pool root and the dataset, because AVAIL is drawn from pool free space and is shared with every other dataset. The snapshot reported -, since it cannot be written to.
• zfs clone rbdrprod/ledger@0900 rbdrprod/inspect0900 returned the 09:00 ledger at md5 9eb4e2ad8e08e1dcaaf87ababab964b0, matching the value recorded at 09:00, while the live dataset still held the truncated version. Every read-back in the capture went through a clone rather than through .zfs/snapshot.
• A clone pins its origin snapshot against destruction until it is removed or promoted, so one forgotten investigation clone stalls snapshot retention for that dataset and shows up weeks later as a pool capacity alert.

Cross-course references

• Linux for Production Sysadmins — Part XVI (LVM) covers the snapshot model this lesson is the counterpoint to: LVM makes you pre-size the copy-on-write exception store and invalidates the snapshot when it fills, whereas ZFS charges the same divergence after the fact against shared pool space, which is why the ZFS failure mode is a full pool rather than a dead snapshot.
• Proxmox VE for Production Operators — Part VI (ZFS) applies exactly this accounting to virtual machine disks, where guest volumes are zvols in a shared pool; the USED-versus-REFER divergence measured here is what makes a rewrite-heavy guest expensive to snapshot on a schedule that looks harmless.
• Ceph & Distributed Storage for Production Sysadmins — Part IV (Failure Domains) supplies the vocabulary for the closing section of this lesson: the ZFS pool is one failure domain, and no amount of snapshot retention inside it adds a second one.

Quiz