Btrfs subvolumes, read-only snapshots and send/receive

Objective

A Btrfs snapshot is cheap, immediate, and sitting on the same device as the data it came from. This lab separates the two properties that get conflated: a snapshot’s ability to hold an earlier version of a file, and a filesystem’s ability to survive losing a disk.

You will build two Btrfs filesystems on two devices, take a read-only snapshot on the first, prove that it holds the original bytes and refuses writes, replicate it to the second filesystem with btrfs send, add an incremental send, then destroy the first device outright. What comes back afterwards, and what does not, is the point.

Architecture

flowchart LR
    subgraph PROD["loop device A, btrfs, mounted at /mnt/rbdr-prod"]
      L["rbdr-ledger, read-write, live"]
      S1["rbdr-ledger-0900, read-only"]
      S2["rbdr-ledger-1000, read-only"]
    end
    subgraph BKP["loop device B, btrfs, mounted at /mnt/rbdr-bkp"]
      R1["rbdr-ledger-0900, received"]
      R2["rbdr-ledger-1000, received"]
    end
    L -->|"subvolume snapshot -r"| S1
    L -->|"subvolume snapshot -r"| S2
    S1 -->|"btrfs send, btrfs receive"| R1
    S2 -->|"btrfs send -p rbdr-ledger-0900"| R2
    D["dd over loop device A"] -->|"takes L, S1 and S2 with it"| PROD
    R1 -->|"restore orders.csv"| OUT["md5 matches the 09:00 ledger"]

Everything in the left box shares one superblock and one device. The edge crossing to the right box is the only one that changes what survives when that device is lost.

Requirements

• btrfs-progs. The capture reproduced here was taken on btrfs-progs v6.14 under Debian GNU/Linux 13 (trixie).
• Root or sudo, for losetup, mkfs.btrfs and mount.
• Two loop devices and about 1 GiB free. The lab creates two 512 MiB images and never touches a real disk.

Scenario

An orders ledger is written to all morning. At 09:00 a snapshot is taken. At 09:30 a deployment truncates the live file to one worthless row. At 10:00 a second snapshot is taken. Some time after that, the array under the production filesystem is lost.

Establish, by doing it, which of those recovery points is still readable afterwards.

Tasks

Task 1 — Record the pre-lab state

Cleanup is provable only against something. Take the inventory first.

LAB="$HOME/rbdr-lab-08"
mkdir -p "$LAB"
date -Is > "$LAB/t-start.txt"
{
  losetup --list --output NAME,BACK-FILE
  findmnt --raw --output TARGET,SOURCE,FSTYPE
  ls -d /mnt/rbdr-* 2>/dev/null || echo "no rbdr mount points"
} > "$LAB/pre-state.txt"
cat "$LAB/pre-state.txt"

If /mnt/rbdr-prod or /mnt/rbdr-bkp already exists, an earlier run is still attached and the cleanup comparison will not be clean.

Task 2 — Two filesystems on two devices

truncate -s 512M "$LAB/rbdr-prod.img"
truncate -s 512M "$LAB/rbdr-bkp.img"
sudo losetup --find --show "$LAB/rbdr-prod.img" > "$LAB/prod-dev"
sudo losetup --find --show "$LAB/rbdr-bkp.img"  > "$LAB/bkp-dev"
PROD_DEV="$(cat "$LAB/prod-dev")"
BKP_DEV="$(cat "$LAB/bkp-dev")"

sudo mkfs.btrfs -L rbdr-prod "$PROD_DEV"
sudo mkfs.btrfs -L rbdr-bkp  "$BKP_DEV"
sudo mkdir -p /mnt/rbdr-prod /mnt/rbdr-bkp
sudo mount "$PROD_DEV" /mnt/rbdr-prod
sudo mount "$BKP_DEV"  /mnt/rbdr-bkp
findmnt --raw --output TARGET,SOURCE,FSTYPE /mnt/rbdr-prod /mnt/rbdr-bkp

Two mkfs runs, two superblocks, two devices. Nothing depends on that separation until Task 7, when it becomes the only thing that does.

Task 3 — A subvolume, then a read-only snapshot of it

sudo btrfs subvolume create /mnt/rbdr-prod/rbdr-ledger
sudo dd if=/dev/urandom of=/mnt/rbdr-prod/rbdr-ledger/rbdr-filler.bin \
  bs=1M count=40 status=none
printf 'ORDER-1001,4500.00\nORDER-1002,1250.00\n' \
  | sudo tee /mnt/rbdr-prod/rbdr-ledger/orders.csv > /dev/null
sudo md5sum /mnt/rbdr-prod/rbdr-ledger/orders.csv | tee "$LAB/md5-0900.txt"
sudo btrfs subvolume list /mnt/rbdr-prod

$ btrfs subvolume create /mnt/prod/ledger && btrfs subvolume list /mnt/prod

Create subvolume '/mnt/prod/ledger'
orders.csv md5 at 09:00: 9eb4e2ad8e08e1dcaaf87ababab964b0
ID 256 gen 9 top level 5 path ledger

That digest is reproducible: the two order rows plus their trailing newline hash to 9eb4e2ad8e08e1dcaaf87ababab964b0 on any host, so md5-0900.txt should contain it exactly. Now the snapshot, and only the read-only kind.

date -Is > "$LAB/t-snapshot-0900.txt"
sudo btrfs subvolume snapshot -r \
  /mnt/rbdr-prod/rbdr-ledger /mnt/rbdr-prod/rbdr-ledger-0900
echo "snapshot exit: $?"
sudo btrfs filesystem df /mnt/rbdr-prod

$ btrfs subvolume snapshot -r /mnt/prod/ledger /mnt/prod/ledger-0900

Create readonly snapshot of '/mnt/prod/ledger' in '/mnt/prod/ledger-0900'

>>> exit code: 0

$ btrfs filesystem df /mnt/prod

Data, single: total=56.00MiB, used=40.00MiB
Metadata, DUP: total=32.00MiB, used=208.00KiB

A snapshot references the extents its source references. Nothing is copied, and nothing is charged, until the two diverge.

Task 4 — Truncate the live ledger and read the original back

printf 'ORDER-9999,0.00\n' \
  | sudo tee /mnt/rbdr-prod/rbdr-ledger/orders.csv > /dev/null
echo "live     : $(sudo cat /mnt/rbdr-prod/rbdr-ledger/orders.csv)"
sudo cat /mnt/rbdr-prod/rbdr-ledger-0900/orders.csv
sudo md5sum /mnt/rbdr-prod/rbdr-ledger-0900/orders.csv

$ cat /mnt/prod/ledger/orders.csv /mnt/prod/ledger-0900/orders.csv

live     : ORDER-9999,0.00
snapshot : ORDER-1001,4500.00
ORDER-1002,1250.00
MATCH - the snapshot still holds the 09:00 ledger

Now the failing case, and it must fail. If a snapshot can be written to, it is not a fixed point and it cannot be the source of a send.

sudo sh -c 'echo tampered > /mnt/rbdr-prod/rbdr-ledger-0900/orders.csv'
echo "write exit: $?"

$ sh -c 'echo tampered > /mnt/prod/ledger-0900/orders.csv'

sh: 1: cannot create /mnt/prod/ledger-0900/orders.csv: Read-only file system

>>> exit code: 2

Read-only file system is EROFS, from the subvolume flag rather than any permission on the file. Root is refused too.

Task 5 — Send the snapshot to the other filesystem

sudo btrfs send /mnt/rbdr-prod/rbdr-ledger-0900 | sudo btrfs receive /mnt/rbdr-bkp
echo "send/receive exit: $?"
date -Is > "$LAB/t-send-0900.txt"
sudo btrfs subvolume list /mnt/rbdr-bkp

$ btrfs send /mnt/prod/ledger-0900 | btrfs receive /mnt/bkp

At subvol ledger-0900

>>> exit code: 0

$ btrfs subvolume list /mnt/bkp
ID 256 gen 10 top level 5 path ledger-0900

Exit code 0 says the stream was produced and consumed. The listing on the destination says a subvolume now exists on the other device, which is the confirmation that matters.

Task 6 — Add a row, snapshot again, send incrementally

printf 'ORDER-1001,4500.00\nORDER-1002,1250.00\nORDER-2001,900.00\n' \
  | sudo tee /mnt/rbdr-prod/rbdr-ledger/orders.csv > /dev/null
sudo btrfs subvolume snapshot -r \
  /mnt/rbdr-prod/rbdr-ledger /mnt/rbdr-prod/rbdr-ledger-1000
sudo btrfs send -p /mnt/rbdr-prod/rbdr-ledger-0900 \
  /mnt/rbdr-prod/rbdr-ledger-1000 | sudo btrfs receive /mnt/rbdr-bkp
echo "incremental exit: $?"
date -Is > "$LAB/t-send-1000.txt"
sudo btrfs subvolume list /mnt/rbdr-bkp

$ btrfs send -p /mnt/prod/ledger-0900 /mnt/prod/ledger-1000 | btrfs receive /mnt/bkp

At snapshot ledger-1000

>>> exit code: 0

$ btrfs subvolume list /mnt/bkp
ID 256 gen 13 top level 5 path ledger-0900
ID 257 gen 14 top level 5 path ledger-1000

-p names a parent both ends already hold, so the stream carries only the difference. Without that parent on the destination the send fails rather than silently sending everything.

Task 7 — Destroy the production device and recover

PROD_DEV="$(cat "$LAB/prod-dev")"
sudo losetup --list --output NAME,BACK-FILE "$PROD_DEV"
date -Is > "$LAB/t-destroyed.txt"
sudo umount /mnt/rbdr-prod
sudo dd if=/dev/urandom of="$PROD_DEV" bs=1M count=64 status=none
sudo sync
sudo mount "$PROD_DEV" /mnt/rbdr-prod
echo "mount exit: $?"

$ mount /dev/loop5 /mnt/prod

mount: /mnt/prod: wrong fs type, bad option, bad superblock on /dev/loop5, missing codepage or helper program, or other error.
     dmesg(1) may have more information after failed mount system call.

>>> exit code: 32

Both read-only snapshots were roots in that tree, and are gone with it; the read-only flag changed nothing about that. The other filesystem was never touched:

RESTORE_START="$(date +%s)"
sudo btrfs subvolume list /mnt/rbdr-bkp
sudo mkdir -p /mnt/rbdr-restore
sudo cp -a /mnt/rbdr-bkp/rbdr-ledger-0900/orders.csv /mnt/rbdr-restore/orders.csv
sudo md5sum /mnt/rbdr-restore/orders.csv | tee "$LAB/md5-restored.txt"
RESTORE_END="$(date +%s)"
echo "Actual restore time: $((RESTORE_END - RESTORE_START))s" \
  | tee "$LAB/restore-time.txt"

LAST_SEND="$(date -d "$(cat "$LAB/t-send-1000.txt")" +%s)"
DESTROYED="$(date -d "$(cat "$LAB/t-destroyed.txt")" +%s)"
echo "Actual RPO observed: $((DESTROYED - LAST_SEND))s of ledger writes unprotected" \
  | tee "$LAB/rpo.txt"

$ btrfs subvolume list /mnt/bkp

ID 256 gen 13 top level 5 path ledger-0900
ID 257 gen 14 top level 5 path ledger-1000

$ cat /mnt/bkp/ledger-0900/orders.csv

ORDER-1001,4500.00
ORDER-1002,1250.00
recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
09:00 md5     : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical to the 09:00 ledger

Validation

diff "$LAB/md5-0900.txt" - <<<"9eb4e2ad8e08e1dcaaf87ababab964b0  /mnt/rbdr-prod/rbdr-ledger/orders.csv"
grep -c 9eb4e2ad8e08e1dcaaf87ababab964b0 "$LAB/md5-restored.txt"
sudo btrfs subvolume list /mnt/rbdr-bkp | grep -c rbdr-ledger
sudo mount "$(cat "$LAB/prod-dev")" /mnt/rbdr-prod; echo "exit=$?"

1. md5sum of the 09:00 ledger (Task 3) must print 9eb4e2ad8e08e1dcaaf87ababab964b0; diff exits 0.
2. btrfs subvolume snapshot -r prints Create readonly snapshot of and exits 0.
3. The write into the snapshot prints Read-only file system and exits 2. Any other exit code means the snapshot was not created with -r.
4. btrfs send | btrfs receive prints an At subvol line naming rbdr-ledger-0900 and exits 0; btrfs subvolume list /mnt/rbdr-bkp then lists rbdr-ledger-0900.
5. The incremental send prints an At snapshot line naming rbdr-ledger-1000 and exits 0; the listing then shows both subvolumes, so grep -c prints 2.
6. After the dd, mount prints wrong fs type, bad option, bad superblock and exits 32.
7. md5sum of the restored orders.csv prints 9eb4e2ad8e08e1dcaaf87ababab964b0; grep -c prints 1 and exits 0.

Expected Outcome

The 09:00 ledger comes back byte-identically from a filesystem that was never mounted on the destroyed device, and the two snapshots that lived beside the live subvolume do not come back at all.

• Actual restore time: the value in restore-time.txt. On a two-row file over loop devices this is under a second; record what your run produced, because the number you quote in an incident comes from the size of the real subvolume, not this one.
• Actual RPO observed: the value in rpo.txt — the interval between the last completed btrfs send and the overwrite. Every ledger write in that window existed only on the destroyed filesystem. It is a property of your send schedule, not of Btrfs.

Troubleshooting

Cleanup

LAB="$HOME/rbdr-lab-08"
sudo umount /mnt/rbdr-bkp
sudo umount /mnt/rbdr-prod 2>/dev/null || true
sudo losetup --detach "$(cat "$LAB/prod-dev")"
sudo losetup --detach "$(cat "$LAB/bkp-dev")"
sudo rmdir /mnt/rbdr-prod /mnt/rbdr-bkp
sudo rm -rf /mnt/rbdr-restore
rm -f "$LAB/rbdr-prod.img" "$LAB/rbdr-bkp.img"

{
  losetup --list --output NAME,BACK-FILE
  findmnt --raw --output TARGET,SOURCE,FSTYPE
  ls -d /mnt/rbdr-* 2>/dev/null || echo "no rbdr mount points"
} > "$LAB/post-state.txt"
diff "$LAB/pre-state.txt" "$LAB/post-state.txt" && echo "OK environment restored"

diff exiting 0 with OK environment restored is the assertion: the same loop devices, the same mounts, no /mnt/rbdr-* left behind. A difference in an unrelated mount is normal on a busy host; a difference naming rbdr- is not, and names exactly what to remove. The files under $LAB are the deliverables and are kept.

Production notes

• btrfs send requires a read-only snapshot, and the reason is arithmetic, not policy. The stream describes a tree — create this, write these bytes, rename that — and is produced by walking a tree that must not change during the walk; for -p it is the difference between two such trees. A writable subvolume can be modified halfway through, leaving the receiver holding a state that never existed on the sender. The read-only flag makes the source a fixed point, and is the same flag that lets -p name a parent both ends hold identically.
• Snapshot and send are two schedules. Snapshots are cheap; sends cost network and destination capacity. Only the recovery points that completed a send survive the array, so report that interval as your loss window.
• The pipeline’s exit code is not the confirmation. Confirm arrival with btrfs subvolume list on the destination, and periodically by reading a file out of a received subvolume and comparing its digest.
• Retention on the destination is coupled to the send chain. Delete the parent there and the next incremental has nothing to build on.

What You Learned

• A read-only snapshot holds the earlier bytes and refuses writes, at exit code 2, from root included.
• btrfs send accepts only a read-only source, because the stream describes a tree that must not move under it.
• -p sends the difference against a parent both ends hold, and fails rather than silently sending everything when they do not.
• btrfs subvolume list on the destination is the confirmation; the pipeline’s exit code is not.
• Losing the device takes every subvolume on it, live and snapshot alike. Only the copies that crossed to the other filesystem returned the ledger with a matching md5.
• The loss window is set by the send schedule, which is why you measured it.