ZFS snapshots, clones and replication to an independent pool

Objective

A snapshot and a replication stream are made of the same blocks, produced by the same command family, and checked with the same hash. What separates them is where the blocks end up. A snapshot is not a backup while it lives on the pool it protects, because one lost device takes the origin and the snapshot with it.

By the end of this lab you will have destroyed a production pool’s backing store outright and recovered the ledger from the pool that was never touched. You will also watch a snapshot’s USED climb from 0B to 47.0M with no byte ever written to the snapshot, which is the accounting behaviour that quietly fills production pools.

Architecture

flowchart LR
    PIMG["rbdr-prod.img\n400 MiB file"] --> PDEV["loop device"]
    PDEV --> PPOOL["pool rbdrprod\n384M"]
    PPOOL --> LED["rbdrprod/ledger\norders.csv + 50 MiB"]
    LED --> S0900["@0900\n0B at creation"]
    LED --> S0930["@0930"]
    S0900 -- "zfs send" --> BLED["rbdrbkp/ledger"]
    S0930 -- "zfs send -i @0900" --> BLED
    S0900 -- "zfs clone" --> INSP["rbdrprod/inspect0900"]
    BIMG["rbdr-bkp.img\n400 MiB file"] --> BDEV["loop device"]
    BDEV --> BPOOL["pool rbdrbkp\n384M"]
    BPOOL --> BLED
    BLED --> RES["rbdrbkp/restore0900\nthe copy that answers"]
    PDEV -. "overwritten in Task 9" .-> GONE["no pools available to import"]

The two stacks share no device: rbdrprod and rbdrbkp fail independently, and Task 9 exercises that boundary.

Requirements

• Mode B-nested. The capture ran in a container on a host with the zfs module loaded; loop devices and pool creation need a real kernel.
• The userland must match the host’s kernel module. zfs --version prints both halves and they have to agree. The capture ran 2.4.1 on both sides.
• The capture used the ubuntu:26.04 image, because the Debian package of the matching userland did not install.
• Root privileges, loop device support, and roughly 1 GB free under /root.
• .zfs/snapshot did not automount in the container the capture ran in. Every read of a historical state below therefore goes through zfs clone.
• No out-of-band access requirement: nothing outside /root/rbdr-lab-07 and the two pools is touched.

$ zfs --version

zfs-2.4.1-1ubuntu5
zfs-kmod-2.4.1-1ubuntu5

Scenario

At 09:00 an orders ledger holds two settled orders and about 50 MiB of transaction detail. You snapshot it, as the runbook says, and replicate that snapshot to a second pool, which the runbook also says but nobody checks.

At 09:30 an operator truncates the ledger and a job rewrites the detail. Snapshot and replica now hold the only copies of the 09:00 state. Then production storage is lost outright, and exactly one of the two answers.

Tasks

Task 1 - Record the state Cleanup will be measured against

LAB=/root/rbdr-lab-07
mkdir -p "$LAB"

zpool list -H -o name 2>/dev/null | sort > "$LAB/state.pre-lab-pools"
losetup -a | sort > "$LAB/state.pre-lab-loops"

echo "pools before: $(wc -l < "$LAB/state.pre-lab-pools")"
echo "loops before: $(wc -l < "$LAB/state.pre-lab-loops")"

Both files are usually empty, and that is the point: Cleanup diffs against them, not your memory.

Task 2 - Create two pools on two independent devices

LAB=/root/rbdr-lab-07

truncate -s 400M "$LAB/rbdr-prod.img"
truncate -s 400M "$LAB/rbdr-bkp.img"

PROD_DEV=$(losetup --find --show "$LAB/rbdr-prod.img")
BKP_DEV=$(losetup --find --show "$LAB/rbdr-bkp.img")
echo "prod pool device:   $PROD_DEV"
echo "backup pool device: $BKP_DEV"

zpool create -o ashift=12 rbdrprod "$PROD_DEV"
zpool create -o ashift=12 rbdrbkp "$BKP_DEV"
zpool list

The pool names carry the rbdr prefix so a stray zpool destroy here cannot name anything of yours.

Task 3 - Write the 09:00 ledger and record its hash

LAB=/root/rbdr-lab-07

zfs create rbdrprod/ledger
printf 'ORDER-1001,4500.00\nORDER-1002,1250.00\n' > /rbdrprod/ledger/orders.csv
dd if=/dev/urandom of=/rbdrprod/ledger/detail.dat bs=1M count=50 status=none

md5sum < /rbdrprod/ledger/orders.csv | cut -d' ' -f1 > "$LAB/md5.0900"
cat /rbdrprod/ledger/orders.csv
cat "$LAB/md5.0900"

The hash goes outside the dataset: one recorded inside the thing you are about to lose proves nothing afterwards.

Task 4 - Snapshot, and see what it costs at the moment it is taken

$ zfs snapshot rbdrprod/ledger@0900
zfs list -r -t all -o name,used,refer,avail rbdrprod

NAME                   USED  REFER  AVAIL
rbdrprod              50.6M    24K   205M
rbdrprod/ledger       50.1M  50.1M   205M
rbdrprod/ledger@0900     0B  50.1M      -

USED is 0B while REFER is 50.1M: the snapshot refers to 50.1M of data and owns none of it, because every block it names is still in use by the live dataset. Per zfsprops(7), a snapshot’s used is the space that would be freed by destroying it - right now, nothing.

Task 5 - Replicate that snapshot to the independent pool

$ zfs send rbdrprod/ledger@0900 | zfs receive rbdrbkp/ledger; echo "exit=$?"

exit=0

$ zfs list -r -o name,used,refer rbdrbkp

NAME             USED  REFER
rbdrbkp         50.2M    24K
rbdrbkp/ledger  50.1M  50.1M

Two copies now sit on two pools backed by two devices. The second is the same blocks, moved.

Task 6 - 09:30, the ledger is truncated and rewritten

: > /rbdrprod/ledger/orders.csv
printf 'ORDER-9999,0.00\n' > /rbdrprod/ledger/orders.csv
dd if=/dev/urandom of=/rbdrprod/ledger/detail.dat bs=1M count=50 status=none
sync
cat /rbdrprod/ledger/orders.csv

$ zfs list -r -t all -o name,used,refer rbdrprod

NAME                   USED  REFER
rbdrprod              97.3M    24K
rbdrprod/ledger       97.1M  50.1M
rbdrprod/ledger@0900  47.0M  50.1M

Nothing was written to the snapshot and its USED went from 0B to 47.0M. It now holds the only reference to blocks the live dataset overwrote, so destroying it would free that much. The figure is unique referenced space, not a count of bytes rewritten: blocks still shared with the live dataset are not charged to it.

Task 7 - Read the 09:00 state back through a clone

zfs clone rbdrprod/ledger@0900 rbdrprod/inspect0900
ls -l /rbdrprod/inspect0900/

$ cat /rbdrprod/inspect0900/orders.csv
CLONE_MD5=$(md5sum < /rbdrprod/inspect0900/orders.csv | cut -d' ' -f1)
echo "md5 from snapshot: $CLONE_MD5"
echo "md5 recorded 09:00: $(cat /root/rbdr-lab-07/md5.0900)"
[ "$CLONE_MD5" = "$(cat /root/rbdr-lab-07/md5.0900)" ] && echo MATCH

ORDER-1001,4500.00
ORDER-1002,1250.00
md5 from snapshot: 9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded 09:00: 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH

A clone is a writable dataset sharing the snapshot’s blocks, so it mounts like any other dataset and costs nothing until written to. That is why it is the reliable route here: .zfs/snapshot did not automount in this container, and a recovery procedure depending on it would have stalled at the first step.

Task 8 - Send the 09:30 state incrementally

$ zfs snapshot rbdrprod/ledger@0930
zfs send -i @0900 rbdrprod/ledger@0930 | zfs receive rbdrbkp/ledger; echo "exit=$?"

exit=0

$ zfs list -r -t snapshot -o name,used,refer rbdrbkp

NAME                  USED  REFER
rbdrbkp/ledger@0900  50.0M  50.1M
rbdrbkp/ledger@0930     0B  50.1M

-i @0900 sends only blocks changed between the two snapshots; the receiver already holds @0900, which is what makes the stream applicable.

Task 9 - The failing case: production storage is permanently lost

LAB=/root/rbdr-lab-07
PROD_DEV=$(losetup -j "$LAB/rbdr-prod.img" | cut -d: -f1)
echo "about to overwrite: $PROD_DEV -> $LAB/rbdr-prod.img"

zpool export rbdrprod
losetup -d "$PROD_DEV"
dd if=/dev/zero of="$LAB/rbdr-prod.img" bs=1M count=400 conv=notrunc status=none
losetup --find --show "$LAB/rbdr-prod.img"

$ zpool import

no pools available to import

$ zfs list -r -t all rbdrprod

cannot open 'rbdrprod': dataset does not exist

The snapshot, the clone and the live dataset went together: all three were names for blocks on one device.

Task 10 - Restore from the pool that was never touched

$ zfs list -r -t all -o name,used,refer rbdrbkp

NAME                  USED  REFER
rbdrbkp               100M    24K
rbdrbkp/ledger        100M  50.1M
rbdrbkp/ledger@0900  50.0M  50.1M
rbdrbkp/ledger@0930     0B  50.1M

$ zfs clone rbdrbkp/ledger@0900 rbdrbkp/restore0900
cat /rbdrbkp/restore0900/orders.csv
REC=$(md5sum < /rbdrbkp/restore0900/orders.csv | cut -d' ' -f1)
echo "md5 recovered:     $REC"
echo "md5 recorded 09:00: $(cat /root/rbdr-lab-07/md5.0900)"
[ "$REC" = "$(cat /root/rbdr-lab-07/md5.0900)" ] && echo "RECOVERED - byte-identical to the 09:00 ledger"

ORDER-1001,4500.00
ORDER-1002,1250.00
md5 recovered:     9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded 09:00: 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical to the 09:00 ledger

LAB=/root/rbdr-lab-07

START=$(date +%s)
cp -a /rbdrbkp/restore0900/orders.csv "$LAB/orders.0900.restored.csv"
cp -a /rbdrbkp/restore0900/detail.dat "$LAB/detail.0900.restored.dat"
END=$(date +%s)

{
  echo "recovery route: rbdrbkp/ledger@0900 -> zfs clone -> rbdrbkp/restore0900"
  echo "recovered md5:  $(md5sum < "$LAB/orders.0900.restored.csv" | cut -d' ' -f1)"
  echo "recorded md5:   $(cat "$LAB/md5.0900")"
  echo "latest snapshot replicated before the loss: @0930"
  echo "restore seconds (copy out of the clone): $((END - START))"
} > "$LAB/restore-report.txt"
cat "$LAB/restore-report.txt"

Time the copy out of the clone, not the clone itself: a clone writes no data, so what matters is how long it takes to get the bytes where the application needs them.

Validation

• zfs --version prints zfs-2.4.1-1ubuntu5 and zfs-kmod-2.4.1-1ubuntu5; the two version numbers match. Exit code 0.
• Task 4: zfs list -H -o used rbdrprod/ledger@0900 prints 0B. Anything else means data was written between create and snapshot.
• Task 5: the send pipeline prints exit=0. A non-zero exit almost always means rbdrbkp/ledger already exists.
• Task 6: zfs list -H -o used rbdrprod/ledger@0900 prints 47.0M. A value near 0B means the rewrite landed outside the snapshotted dataset.
• Task 7: the final line of the comparison block is MATCH, and the clone’s hash is 9eb4e2ad8e08e1dcaaf87ababab964b0.
• Task 9 (the failing case): zpool import prints exactly no pools available to import, and zfs list -r -t all rbdrprod prints cannot open 'rbdrprod': dataset does not exist and exits 1. If either succeeds, the dd missed the device the pool was on and the test proved nothing.
• Task 10: zfs list -r -t all -o name,used,refer rbdrbkp lists both rbdrbkp/ledger@0900 and rbdrbkp/ledger@0930, and the recovery block’s last line is RECOVERED - byte-identical to the 09:00 ledger.
• restore-report.txt exists and its recovered and recorded hashes are the same string.

Expected Outcome

The production pool is unrecoverable and the 09:00 ledger is on disk anyway, restored from a pool sharing no device with it. Both hashes read 9eb4e2ad8e08e1dcaaf87ababab964b0.

The RPO here is set by how often you send, not by ZFS: sending hourly leaves an hour of exposure however many snapshots the source holds.

Troubleshooting

cannot open 'rbdrprod': dataset does not exist before Task 9. The pool was never created, or zpool create failed on a busy loop device. Re-run losetup --find --show and check that zpool list names both pools.

The receive in Task 5 is refused because the destination already exists. Task 5 was run twice. Destroy rbdrbkp/ledger and send again; the stream has nowhere to land.

The incremental receive in Task 8 is refused for a mismatched incremental source. The receiving dataset was modified after the full receive, so it no longer matches @0900. The incremental depends on the target being untouched since the previous snapshot.

ls /rbdrprod/ledger/.zfs/snapshot is empty or absent. Expected here: it did not automount in the container the capture ran in. Use zfs clone, as Tasks 7 and 10 do.

The snapshot’s USED stays near 0B after Task 6. The rewrite went to a different path. dd must target /rbdrprod/ledger/detail.dat, inside the snapshotted dataset.

zpool import lists rbdrprod after Task 9. The dd wrote to a detached image while another loop device still carried the old data. Re-check the device the pool was on.

Cleanup

LAB=/root/rbdr-lab-07

cp "$LAB/state.pre-lab-pools" /root/rbdr-lab-07-pools.baseline
cp "$LAB/state.pre-lab-loops" /root/rbdr-lab-07-loops.baseline

zpool destroy rbdrbkp

for IMG in rbdr-prod.img rbdr-bkp.img; do
  DEV=$(losetup -j "$LAB/$IMG" | cut -d: -f1)
  if [ -n "$DEV" ]; then losetup -d "$DEV"; fi
done

rm -rf "$LAB"

zpool list -H -o name 2>/dev/null | sort | diff /root/rbdr-lab-07-pools.baseline -
losetup -a | sort | diff /root/rbdr-lab-07-loops.baseline -
rm -f /root/rbdr-lab-07-pools.baseline /root/rbdr-lab-07-loops.baseline
ls -d /root/rbdr-lab-07 2>/dev/null || echo "lab directory removed"

Both diff invocations must print nothing and exit 0. That is the restoration assertion: the pool list and the loop device list are exactly what Task 1 recorded, so no pool is imported and no loop device attached. rbdrprod is already gone, which is why only rbdrbkp is destroyed here.

Production notes

• Replicate to a pool on different hardware, ideally in a different building: two pools in one chassis fail together for a power supply or a fire. This lab’s two pools sit on one host, which proves the mechanism and is not an air gap.
• Alert on the age of the newest snapshot on the receiving side. A send that stopped three weeks ago looks identical to one that ran last night if you only watch the source.
• Keep the recorded hash outside the dataset, as Task 3 does. Verification that lives with the data verifies nothing once the data is gone.
• Watch snapshot USED as a capacity signal. A pool filling because retained snapshots hold overwritten blocks is routine, and the tempting remedy - destroying recovery points to reclaim space - is the wrong move.

What You Learned

• A snapshot costs nothing at creation and grows as its origin changes. 0B became 47.0M with no write to the snapshot, because it became the sole reference to overwritten blocks.
• zfs send moves those same blocks into an independent failure domain. The mechanism is identical to a snapshot; the independence is the whole difference.
• zfs send -i extends a chain the receiver must already anchor. The target has to hold the previous snapshot and be unmodified since it.
• zfs clone is the dependable way to read a historical state. It does not rely on .zfs/snapshot being automounted.
• Losing the device removed the pool, the dataset, the clone and both snapshots at once. no pools available to import is what a same-pool recovery plan looks like on the day it is needed.