Replication with send/receive: when a snapshot becomes a copy

Everything in the previous lesson resolved inside one pool, and the destruction test removed the pool. Retention did not help, because retention decides how far back the history reaches and not how many devices hold it. What changes the outcome is a second set of devices and a way to put a snapshot’s contents onto them. zfs send and btrfs send are that way: each reads a point in time that cannot change and writes it to standard output, and a matching receive rebuilds it as a native object of the destination. The point in time is unchanged by the journey. Where it lives is not, and that is the whole distinction between a snapshot and a copy.

The stream is a serialisation, not a file transfer

Neither command copies a directory tree. zfs send creates, in the words of zfs-send(8), a stream representation of a snapshot written to standard output; btrfs send emits a sequence of operations — create this file, set these attributes, write these bytes here, clone this range from a range that already exists — which btrfs-receive(8) replays to reconstruct the tree. The first consequence is that the transport is whatever carries bytes: zfs-send(8) notes the output can be redirected to a file or to a different system using ssh(1).

The second consequence matters more. What lands on the far side is a first-class object of that filesystem, not a reference back to the source. zfs-recv(8) states that when a full stream is received a new file system is created; btrfs-receive(8) stores the received subvolumes at the given path and sets each read-only once the receive finishes. Nothing in the destination points at the origin, which is the property that lets the origin be destroyed without consequence.

$ zfs send rbdrprod/ledger@0900 | zfs receive rbdrbkp/ledger

exit=0
NAME             USED  REFER
rbdrbkp         50.2M    24K
rbdrbkp/ledger  50.1M  50.1M

Fifty megabytes now exist twice, on two pools, on two devices. The Btrfs capture does the same across a filesystem boundary — /mnt/prod on /dev/loop5, /mnt/bkp on /dev/loop6 — reporting At subvol ledger-0900 at exit code 0, after which /mnt/bkp holds a subvolume with its own ID and generation number.

There is a design decision hidden in that pipe. A stream can instead be redirected into a file and kept as an archive, and that file is then one opaque object worth exactly what a future receive can make of it. zfs-send(8) commits the stream format, promising streams remain receivable on future versions, and btrfs-receive(8) documents a --dump mode that validates a stream without touching the filesystem. Neither changes the shape of the risk: a stream in a file cannot be listed, partially read, or checksummed against the data it represents without being received first. Receiving on arrival is what keeps the destination inspectable.

An incremental is an assertion about the other side

A second full send would move the whole dataset again. The point is to move only what changed, which both tools do by naming a parent the receiving side is asserted to already hold.

$ zfs send -i @0900 rbdrprod/ledger@0930 | zfs receive rbdrbkp/ledger

exit=0
--- snapshots now held on the backup pool ---
NAME                  USED  REFER
rbdrbkp/ledger@0900  50.0M  50.1M
rbdrbkp/ledger@0930     0B  50.1M

The Btrfs equivalent spells the same argument -p and names two subvolumes rather than two snapshot suffixes.

$ btrfs send -p /mnt/prod/ledger-0900 /mnt/prod/ledger-1000 | btrfs receive /mnt/bkp

At snapshot ledger-1000

>>> exit code: 0

ID 256 gen 13 top level 5 path ledger-0900
ID 257 gen 14 top level 5 path ledger-1000

Read -i and -p as claims rather than hints. The sending side needs the parent because it computes the difference against it. The receiving side needs it because it has to apply that difference to something, and zfs-recv(8) is explicit: if an incremental stream is received, the destination file system must already exist and its most recent snapshot must match the incremental stream’s source. btrfs-receive(8) lists its refusals in the same spirit, failing when the receiving subvolume already exists, when a previously received subvolume has been changed since it was received, or when the filesystem was not mounted at the top-level subvolume.

So the chain has two ends and either can break it. Destroy @0900 on production and the next incremental has nothing to compute from — zfs-send(8) does accept a bookmark as an incremental source, which preserves the ability to send after the snapshot’s data is gone, but that helps only the sending side. Destroy @0900 on the backup pool and the assertion is false: the stream describes changes relative to a state the destination no longer holds, and the receive fails. Two retention policies written independently, one by the storage team and one by whoever set up the destination, is the ordinary route to a chain that breaks unwatched.

set -euo pipefail

SRC=rbdrprod/ledger
DEST=rbdrbkp/ledger
PARENT=$(zfs list -H -o name -t snapshot -s creation -r "$SRC" | tail -n 1)
SNAP="${SRC}@$(date -u +%Y%m%dT%H%M%SZ)"

zfs snapshot "$SNAP"
zfs send -i "$PARENT" "$SNAP" | zfs receive "$DEST"

PARENT is resolved before the new snapshot is taken, because the parent is the newest snapshot the destination already has. Without pipefail the shell reports the status of the last command in the pipeline, so a zfs send that dies partway leaves the exit code to zfs receive, and the schedule reports what the receiver made of a truncated stream.

Measured: two points in time that production no longer had

The capture then does the only test that separates a snapshot from a backup. The production backing store is fully overwritten and re-attached as /dev/loop5. zpool import answers no pools available to import, and an attempt to reach the 09:00 snapshot answers cannot open 'rbdrprod': dataset does not exist. Every snapshot that lived in that pool is gone, along with the accounting that made them look so well managed.

$ zfs list -t all -o name,used,refer -r rbdrbkp

--- what the INDEPENDENT backup pool still holds ---
NAME                  USED  REFER
rbdrbkp               100M    24K
rbdrbkp/ledger        100M  50.1M
rbdrbkp/ledger@0900  50.0M  50.1M
rbdrbkp/ledger@0930     0B  50.1M

Two points in time, on a pool that survived. Notice what production held at the moment it died: neither of them. The live ledger had been truncated to a single worthless row at 09:30 and written to again, so the 09:00 state existed nowhere on production even before the device was destroyed. The backup pool holds both because each was sent when it was current, and because nothing there destroyed them afterwards — a retention decision belonging to the destination, doing as much work here as the replication is.

Presence, though, is not readability.

$ zfs clone rbdrbkp/ledger@0900 rbdrbkp/restore0900

ORDER-1001,4500.00
ORDER-1002,1250.00
md5 recovered:     9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded 09:00: 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical to the 09:00 ledger

The clone is the instrument the previous lesson used for investigation, pointed at a different pool. It is also the only shape of verification that answers the third claim from the first lesson: the bytes were read back and compared against a value recorded before the incident.

What turns a schedule into a backup

The commands above are two lines. Everything that makes them a backup system is outside them, in three obligations.

The first is monitoring the outcome rather than the job. A pipeline reports success when its last command is content, which is not the same as the destination having gained anything. The signal worth alerting on lives on the receiving side: does the snapshot the schedule should have created exist there, and is its creation time inside the expected window? That catches a send that never ran, a truncated stream and a full destination, none of which a green job row distinguishes from success.

The second is retention set separately on each side. The value of the destination is that it holds points production has discarded, so copying production’s retention onto it produces a second copy of the history you already have. State the two as different sentences — production keeps hourly snapshots for two days; the backup pool keeps daily for thirty and monthly for a year — then check that the incremental parent survives on both sides for longer than the interval between transfers.

The third is verification that reads. A received Btrfs subvolume is read-only and can be read where it sits; on ZFS a clone gives the same access without disturbing the received dataset.

DEST=rbdrbkp/ledger
CHECK=rbdrbkp/verify
LATEST=$(zfs list -H -o name -t snapshot -s creation -r "$DEST" | tail -n 1)

zfs clone "$LATEST" "$CHECK"
( cd "/$CHECK" && md5sum -c /var/lib/recovery/ledger.md5 )
zfs destroy "$CHECK"

The destroy at the end is not tidiness: a clone pins its origin snapshot, so one left behind stalls retention on the backup pool exactly as the previous lesson measured on production.

Continuous replication answers a different question

Either capture could run every five minutes, and the destination would track production closely. That arrangement is valuable and not the same product. Replication bought for availability optimises recency: keep a standby close enough to current that a failover loses little, measured by how far behind it is. Replication bought for history optimises the spread of recoverable states, measured by which moments can be returned to.

The commands are identical; the retention is what differs. A destination keeping only the newest received snapshot has been configured to be a mirror — a place defined as whatever the source most recently was. That is the propagation problem from Part V, arriving through a copy-on-write filesystem instead of through rsync. The 09:30 truncation was a legitimate write, and a destination holding only the newest state would have received it, reported success, and held one worthless row.

Production discipline

1. Send only from a read-only point in time. btrfs send accepts nothing else, and btrfs receive sets each received subvolume read-only on success so it can serve as tomorrow’s -p parent. Keep the received ZFS dataset unwritten too, since zfs-recv(8) requires its most recent snapshot to match the next stream’s source.
2. Fail the whole pipeline, not just its last command. set -euo pipefail before any send | receive. The captures recorded exit=0 on the ZFS side and >>> exit code: 0 on the Btrfs side; without pipefail those figures describe the receiver alone.
3. Alert on the age of the newest snapshot at the destination. Query the receiving pool, not the sending host. The surviving evidence in the capture was a listing of rbdrbkp showing @0900 and @0930, taken when production could not be imported at all.
4. Write the two retention policies as two separate sentences. The backup pool held 09:00 and 09:30 while production held neither, and only because its retention is its own. Then confirm the incremental parent survives on both sides for longer than the interval between transfers.
5. Read something back on a schedule, and remove what you read it with. The capture cloned rbdrbkp/ledger@0900 and matched the md5 recorded at 09:00, 9eb4e2ad8e08e1dcaaf87ababab964b0, which is the only evidence that the copy is a copy. Destroy the verification clone: it pins its origin snapshot against retention.

Cross-course references

• Ceph & Distributed Storage for Production Sysadmins — Part CVI (RBD Backup) asks this lesson’s question of a cluster image: an RBD snapshot lives in the same pool as the image it captures, so the backup problem there is getting a point in time onto storage the cluster does not own — the same independence test with a larger failure domain. Part XXIII (Replication) is the contrast drawn in the closing section, since replication inside the cluster keeps the service available through a device or host loss and retains no history at all.
• Linux for Production Sysadmins — Part LXI (DRBD Concepts) covers block-level continuous replication, the clearest case of replication bought for availability: it propagates every write, the 09:30 truncation included, and so answers the device-loss question this lesson answers while answering none of the history question. Part XLVII (Backup Strategy) is where the number of independent copies is decided, which is the input a send destination implements.
• Proxmox VE for Production Operators — Part VI (ZFS) puts these datasets underneath virtual machine disks, where this lesson’s question — is the receiving side on storage that fails independently — decides whether a guest snapshot schedule protects against anything beyond a bad in-guest change. Part XIV (Disaster Recovery) then consumes whichever copies survived, which is where the retention decision made here is either sufficient or is discovered not to be.

Quiz