ZFS send/receive: replication, off-site backups, and migration

zfs send and zfs receive are ZFS’s killer feature for data movement. They stream block-level deltas between snapshots, are resumable, and respect compression and encryption. Used well, they let you replicate a multi-terabyte dataset over a slow link by sending only what’s changed.

This lesson covers the operational patterns: full replication, incremental replication, off-site backup, and live VM migration.

Mental model

A ZFS snapshot is a reference to a point-in-time block tree. zfs send serialises that tree into a stream of write operations. zfs receive applies that stream to a destination dataset, recreating the snapshot there.

There are two send modes:

• Full send: the entire snapshot’s data, from the snapshot’s perspective.
• Incremental send: the delta between two snapshots — only the blocks that changed.

A full send is essentially a backup format. An incremental send is what you want for ongoing replication.

Full replication (initial)

# On the source host, create a snapshot
zfs snapshot tank/vm-storage@replicate-2024-01-15

# Send it to the destination
zfs send tank/vm-storage@replicate-2024-01-15 | \
  ssh backup-host "zfs receive backup-pool/vm-storage"

The destination dataset must not exist before the receive. Use zfs receive -F to force-overwrite an existing dataset (DANGEROUS: it deletes the destination first).

Incremental replication

For ongoing replication, take regular snapshots and send only the delta:

# Initial full send (one time)
zfs snapshot tank/vm-storage@replicate-day-1
zfs send tank/vm-storage@replicate-day-1 | \
  ssh backup-host "zfs receive backup-pool/vm-storage"

# Subsequent incremental sends
zfs snapshot tank/vm-storage@replicate-day-2
zfs send -i tank/vm-storage@replicate-day-1 tank/vm-storage@replicate-day-2 | \
  ssh backup-host "zfs receive backup-pool/vm-storage"

The -i flag tells zfs send to compute the delta from @replicate-day-1 to @replicate-day-2. The destination receives only the changed blocks.

The receive side automatically creates the corresponding snapshot on the destination dataset.

Resumable replication

A long send over a flaky link will fail mid-stream. Use zfs send --resume to resume from the last successfully sent byte:

# The first send fails halfway through
zfs send -i src@day-1 src@day-2 | \
  ssh backup "zfs recv -s backup/dst"  # -s enables resumable receive
# Send stream ends with "cannot receive: out of space" or similar

# The resume picks up where it left off
zfs send -t \
  -i src@day-1 src@day-2 | \
  ssh backup "zfs recv -s backup/dst"

The -t flag on the send side says “use the resume token from the destination’s last partial receive”. Without zfs send --resume support, you have to fall back to a full re-send on failure.

PVE 9.x has built-in ZFS replication jobs (Datacenter → Replication) that handle the snapshot / send / receive / cleanup lifecycle for you.

Compressed and encrypted streams

ZFS streams respect the source’s compression and encryption settings. For wire transfer, you can also add transport compression:

# Use mbuffer to smooth the bandwidth profile and add compression
zfs send src@day-2 | mbuffer -s 1k -m 1G | \
  ssh backup-host "mbuffer -s 1k -m 1G | zfs receive backup/dst"

# Or pipe through gzip / zstd
zfs send src@day-2 | zstd -T0 | \
  ssh backup-host "zstd -d | zfs receive backup/dst"

For VM replication over a slow link, the mbuffer pattern prevents the receiver from blocking when the network drops. The zstd pattern is best for limited-bandwidth off-site backup.

Off-site backup with rotation

A common pattern is: keep 7 days of hourly snapshots on-site, and replicate to an off-site PBS or ZFS pool once a day.

#!/bin/bash
# /usr/local/bin/zfs-replicate.sh
set -euo pipefail

SRC=tank/vm-storage
DST_HOST=backup.example.com
DST_POOL=backup-pool/vm-storage
SNAP="${SRC}@replicate-$(date +%Y-%m-%d)"
LAST_SNAP="${SRC}@replicate-$(date -d 'yesterday' +%Y-%m-%d)"

# Create today's snapshot
zfs snapshot "$SNAP"

# Send the delta from yesterday to today
if zfs list -H -t snapshot "$LAST_SNAP" >/dev/null 2>&1; then
  zfs send -i "$LAST_SNAP" "$SNAP" | \
    ssh "$DST_HOST" "zfs receive -F '$DST_POOL'"
else
  # No yesterday snapshot — full send
  zfs send "$SNAP" | \
    ssh "$DST_HOST" "zfs receive -F '$DST_POOL'"
fi

# Prune old snapshots on the destination (keep 14 days)
ssh "$DST_HOST" "zfs list -H -t snapshot -o name '$DST_POOL' | \
  grep '@replicate-' | \
  while read snap; do
    date=\$(echo \$snap | grep -oP '\d{4}-\d{2}-\d{2}')
    age=\$((\$(date +%s) - \$(date -d \"\$date\" +%s)))
    if [ \$age -gt 1209600 ]; then  # 14 days in seconds
      zfs destroy \$snap
    fi
  done"

Run this daily via cron:

# (heredoc replaced)
echo "#!/bin/bash" >> /etc/cron.daily/zfs-replicate
echo "/usr/local/bin/zfs-replicate.sh" >> /etc/cron.daily/zfs-replicatechmod +x /etc/cron.daily/zfs-replicate

Live VM migration with ZFS send/receive

For an offline migration of a VM between hosts:

# On source host
qm shutdown 100
zfs snapshot tank/vm-storage/vm-100-disk-0@migrate
zfs send tank/vm-storage/vm-100-disk-0@migrate | \
  ssh dest "zfs receive dest-pool/vm-storage/vm-100-disk-0"
# Configure the destination to recognise the new VM
# (qm importdisk or recreate VM with the imported disk)

For a live migration with minimal downtime, use the snapshot approach while the VM is running:

# 1. Take a snapshot while the VM is running (instant)
zfs snapshot tank/vm-storage/vm-100-disk-0@pre-migrate

# 2. Send the snapshot to the destination
zfs send tank/vm-storage/vm-100-disk-0@pre-migrate | \
  ssh dest "zfs receive dest-pool/vm-storage/vm-100-disk-0"

# 3. Take a second snapshot
zfs snapshot tank/vm-storage/vm-100-disk-0@post-migrate

# 4. Send the incremental delta (only the writes from the first)
zfs send -i tank/vm-storage/vm-100-disk-0@pre-migrate \
          tank/vm-storage/vm-100-disk-0@post-migrate | \
  ssh dest "zfs receive dest-pool/vm-storage/vm-100-disk-0"

# 5. Stop the source VM, take a final snapshot
qm shutdown 100
zfs snapshot tank/vm-storage/vm-100-disk-0@final
zfs send -i tank/vm-storage/vm-100-disk-0@post-migrate \
          tank/vm-storage/vm-100-disk-0@final | \
  ssh dest "zfs receive dest-pool/vm-storage/vm-100-disk-0"

# 6. Start the destination VM
ssh dest "qm importdisk 100 dest-pool/vm-storage/vm-100-disk-0"

Total downtime: the time for the final delta send, typically seconds to a few minutes depending on write rate during the sync.

Encrypting ZFS streams in transit

The default ZFS send stream is unencrypted. For off-site transfer over an untrusted network:

# mbuffer + ssh with key-based auth is encrypted by default
zfs send src@day-2 | ssh backup-host "zfs receive backup/dst"

# For extra paranoia, layer mTLS or wrap in a WireGuard tunnel

For an untrusted destination, also enable ZFS native encryption on the destination pool:

zpool create -O encryption=on -O keylocation=prompt -O keyformat=raw \
  backup-pool /dev/disk/by-id/...

Common mistakes

• Not testing the receive side. A failed receive leaves a partial dataset. Always run zfs list -t snapshot after a receive to confirm the snapshot exists.
• Forgetting -i flag. Sending a full snapshot instead of the delta wastes bandwidth.
• Backups with no restore drill. A ZFS replication pipeline that has never been tested for restore is not a backup — it’s a copy. Test every quarter.
• Encrypting only the wire, not the disk. Wire encryption protects in transit. Disk encryption protects at rest. For an off-site backup, you want both.

Production considerations

• Bandwidth planning. A typical busy VM generates 5–50 GB/day of writes. At 100 Mbps off-site link, that’s 7–70 minutes per incremental send. Schedule after-hours.
• Snapshot retention on both ends. The source keeps recent snapshots for fast local recovery. The destination keeps older snapshots for off-site history. Different retention, both managed.
• Verify, don’t trust. A ZFS replication that has never been tested for restore is a copy, not a backup. Run a quarterly restore drill.
• Throttling for noisy neighbours. zfs send will saturate the network link if you let it. Use mbuffer or trickle to limit bandwidth on shared connections.

Key takeaways

• Full send for initial replication, incremental send for ongoing.
• zfs send --resume for long transfers over flaky links.
• Compress with zstd for limited-bandwidth off-site.
• Test the receive side after every replication. Test restore quarterly.

Knowledge check