Choosing a replication strategy: ZFS send, Ceph RBD mirror, PBS sync · Proxmox VE

“We replicate to the DR site” is a sentence that can mean three completely different things, with RPOs ranging from seconds to a day and recovery behaviours that differ in kind, not just in degree.

This lesson puts all three on the same axes so the sentence has to become specific.

The three mechanisms

	ZFS replication (`pvesr`)	Ceph RBD mirroring	PBS sync
What moves	ZFS snapshot deltas	RBD journal or snapshot deltas	Deduplicated chunks
Typical RPO	1 - 60 min (scheduled)	Seconds to minutes	1 - 24 h (scheduled)
RTO at the far end	Minutes - start the guest	Minutes - promote and start	Hours - restore the guest
Point-in-time history	Only retained snapshots	Effectively none	Full retention policy
Deduplication	None across guests	None	Global across the datastore
Survives a deletion?	Only within snapshot retention	No	Yes
In Proxmox’s UI	Yes, fully	No	Yes, fully
Storage cost at far end	Full copy	Full copy	Deduplicated, compressed
Guest downtime to recover	Start on the far node	Promote, then start	Full restore transfer

Read down the “survives a deletion” row. It is the row that decides whether a mechanism is a backup, and two of the three answer no.

Replication is not backup, and the reason is one sentence

Replication propagates state. Backup preserves history.

If someone deletes a database table, encrypts your filesystems, or drops a directory, replication carries that faithfully to the far site - promptly, and in the case of RBD mirroring, within seconds. The DR copy is now as damaged as the primary, and the mechanism worked perfectly.

A backup with retention has the state from before. That is the entire difference, and it is why the combinations at the end of this lesson always include PBS, regardless of what else is in them.

The tempting error is quantitative reasoning: “we replicate every 5 minutes, so our RPO is 5 minutes, which is better than nightly backups.” Both halves are true and the conclusion is wrong, because they are RPOs against different failures. Replication’s RPO is against hardware loss. It has no RPO at all against corruption, because it has no earlier state to return to.

ZFS replication with `pvesr`

The mechanism Proxmox integrates for local-storage clusters. A scheduled incremental zfs send of the guest’s dataset to another node.

Configuration changecreate and inspect a replication job— Creates a scheduled replication. The first run transfers the whole dataset; subsequent runs send only the delta since the last common snapshot.

set -euo pipefail
VMID=100
TARGET=pve-02

pvesr create-local-job "$VMID-0" "$TARGET" --schedule '*/15'

pvesr list
pvesr status

Read-only / Safereplication status, and the field that matters

# pvesr status

JobID    Enabled    Target           LastSync             NextSync             Duration  FailCount State
100-0    Yes        local/pve-02     2026-08-12_09:15:02  2026-08-12_09:30:00      12.44          0 OK
101-0    Yes        local/pve-02     2026-08-12_09:15:04  2026-08-12_09:30:00      31.09          0 OK
104-0    Yes        local/pve-03     2026-08-11_22:45:11  2026-08-12_09:30:00       0.00          7 command 'zfs send' failed

Illustrative output

Job 104-0 is the one to notice. FailCount 7 and a LastSync eleven hours old means that guest’s DR copy is eleven hours behind while every dashboard still shows a replication job configured for it.

Strengths. No shared storage required, so it works on the cheapest possible cluster. Fully integrated - the HA stack knows about it and can recover a guest onto a replication target. The transfer is a block-level delta, so a guest that changed little sends little.

Limits worth stating plainly.

Pairwise and directional. A guest replicated to pve-02 cannot be recovered on pve-03. Your HA target set for that guest is one node.
Recovery is a rollback. The guest resumes at the last replicated snapshot; everything after it is gone and the application does not know.
The schedule is the RPO. A 15-minute schedule is a 15-minute RPO, and that number belongs in whatever document promises one.
Failures are quiet. A failing job increments a counter. Nothing pages.

Read-only / Safethe check that catches a silently stale replica— Read-only. Run it from monitoring, not by hand - the failure this catches is one nobody notices for days.

set -euo pipefail
MAX_AGE_MIN=45
BAD=0

# Columns: JobID Enabled Target LastSync NextSync Duration FailCount State
# Process substitution, not a pipe: a pipe would run the loop in a subshell
# and BAD would never survive back to the exit.
while read -r JOB _EN TARGET LAST _NEXT _DUR FAILS REST; do
# LastSync is YYYY-MM-DD_HH:MM:SS; convert for date(1).
WHEN=${LAST//_/ }
AGE=$(( ( $(date +%s) - $(date -d "$WHEN" +%s) ) / 60 ))
if [ "$FAILS" -gt 0 ] || [ "$AGE" -gt "$MAX_AGE_MIN" ]; then
  printf 'STALE %-8s target=%-16s age=%dmin fails=%s state=%s\n' \
    "$JOB" "$TARGET" "$AGE" "$FAILS" "$REST"
  BAD=1
fi
done < <(pvesr status | tail -n +2)

exit "$BAD"

Ceph RBD mirroring

The honest position first, because it is the reason this section is short.

RBD mirroring is a Ceph feature, not a Proxmox one

Ceph implements cross-cluster RBD mirroring through the rbd-mirror daemon, and it works. Proxmox VE does not manage it: there is no interface for it in the web UI, no pveceph subcommand for it, and the PVE documentation does not cover it. You configure it with Ceph’s own tooling, against a second Ceph cluster, and you own the result.

That is not a reason to avoid it. It is a reason to be clear-eyed about what adopting it means:

Two Ceph clusters, not one stretched cluster. Mirroring is between clusters, and each needs its own monitors, managers and OSDs.
A daemon Proxmox does not know about. rbd-mirror runs somewhere, and its health is not in any PVE dashboard. If it stops, nothing in Proxmox notices.
Promotion is a manual Ceph operation, and the resulting image has to be attached to a guest that you also have to create on the far side, because PVE’s guest configuration does not mirror with the image.
Support conversations get harder. When a restore goes wrong, the boundary between “Proxmox storage layer” and “your Ceph replication” is yours to reason about.

For estates with Ceph expertise and a hard seconds-scale RPO requirement it is the right tool and it is the only one of the three that reaches that RPO. For everyone else, the operational surface is larger than the benefit, and the combination of Ceph locally with PBS sync offsite is the answer.

Two modes exist, and the choice between them is the usual latency-versus-cost trade:

Mode	Mechanism	RPO	Cost
Journal-based	Every write is journalled, then replayed at the peer	Near-continuous	A write amplification on every I/O at the primary
Snapshot-based	Periodic image snapshots, deltas shipped	The snapshot interval	No steady-state write penalty

Journal-based mirroring makes every write on the primary cluster more expensive, permanently, in exchange for a very low RPO. That cost is paid by production, not by DR, and it is the factor most often left out of the decision.

PBS sync

Covered mechanically in DR for the backup server and offsite replication. Its place on these axes:

The slowest RPO and the only real history. An hourly sync means up to an hour of loss for a hardware event - worse than either alternative - and it is the only one of the three that can recover you from something that happened three weeks ago.

The cheapest bytes by a wide margin. Deduplication is global across the datastore, so a hundred guests built from the same template store the common chunks once. Compare a hundred full ZFS copies.

The slowest recovery. The guest must be restored, which means transferring its data before it can start. That is the RTO arithmetic from live restore, and live restore is what narrows the gap.

The only one that survives a compromise, when configured pull-direction with remove-vanished off, because the DR copy is not reachable from the primary.

Why the three have the RPOs they do

The RPO of each mechanism is a direct consequence of what it has to move and when it can know what to move.

ZFS replication sends the delta between two snapshots. Producing that delta requires taking a snapshot, and a snapshot is a discrete event. So the RPO can never be better than the snapshot interval, and the interval cannot be made arbitrarily small because each run has fixed costs - enumerate datasets, establish a session, negotiate the common snapshot. Every minute or two is the practical floor, and 15 minutes is where most estates land.

Journal-based RBD mirroring does not wait for an event, because every write is already an event. The write is journalled at the primary, the peer’s rbd-mirror daemon replays the journal continuously, and lag is bounded by throughput rather than by a schedule. That is why it reaches seconds - and the cost is structural: every write is now two writes at the primary, forever.

PBS sync moves chunks, and the chunks only exist once a backup has run. So PBS’s RPO is not the sync schedule at all - it is the backup schedule. A backup taken nightly and synced hourly still has a 24-hour RPO, because the hourly sync spends 23 of those hours copying nothing new. This is a genuinely common misconfiguration: people tune the sync frequency to improve the RPO and change nothing, because the constraint is upstream.

The general shape: RPO is set by the cadence at which the mechanism learns what changed. Snapshot-based mechanisms learn on a schedule; journal-based mechanisms learn continuously; backup-based mechanisms learn when a backup runs.

What each does with a deletion

The single most useful way to compare them, because it is the failure people plan for least.

At 14:00, someone deletes the contents of a production filesystem.

Mechanism	At 14:01	At 20:00	Recovery
ZFS replication, 15 min	Far copy still intact	Deletion replicated; earlier ZFS snapshots may remain	Roll back to a retained snapshot, if the retention covers it
RBD mirroring, journal	Deletion already at the far site	Same	None from this mechanism
PBS, nightly + hourly sync	Last night’s snapshot intact	Still intact	Restore, losing the day’s other work

RBD mirroring’s row is not a defect. It is a correct description of a mechanism designed to keep two sites identical, applied to a failure where identical is the problem.

What happens if you run ZFS replication and PBS backups against the same guest?

They coexist, and there is one interaction worth knowing about that produces confusing behaviour.

Both mechanisms take ZFS snapshots of the same dataset. pvesr creates snapshots named for its jobs and prunes them once the delta has been sent; vzdump in snapshot mode takes its own and removes it when the backup completes. Neither deletes the other’s - the naming keeps them apart - so the normal case is fine.

The abnormal case is a stuck job. A replication job that fails partway leaves its snapshot behind, and it will be retained until the job succeeds or is removed. A backup that was interrupted leaves one too. Each retained snapshot pins the blocks that existed when it was taken, so on a busy dataset the pool grows for reasons that du inside the guest cannot explain.

The symptom is a ZFS pool filling with no corresponding growth in guest data:

zfs list -t snapshot -o name,used,creation -s creation | tail -20
zfs list -o name,used,usedbysnapshots,available

A snapshot from three weeks ago holding a large used figure is a stuck job, not a mystery. Find the job, fix or delete it, and the space returns.

The ordering point is smaller but worth having: schedule replication and backup so they do not overlap. Both read heavily from the same dataset, and running them together makes each slower without making either safer.

Combinations that actually work

The design rule: every named failure gets exactly one recovery path, and every mechanism is there because a specific failure names it.

Small estate, no shared storage

Local ZFS + pvesr to a second node, 15 minutes - covers node loss.
PBS nightly to a local datastore - covers deletion and corruption.
PBS sync to an offsite datastore, pull direction - covers site loss and ransomware.

RPO: 15 minutes for hardware, 24 hours for everything else. RTO: minutes for node loss, hours for anything requiring a restore. This covers most small estates and is the cheapest complete answer.

Ceph cluster, single site

Ceph replication across failure domains - covers disk, host and rack.
PBS to a local datastore - covers deletion and corruption.
PBS sync offsite, pull direction, remove-vanished off - covers site loss.

RPO: zero for hardware inside the site, the backup interval for everything else. Note what is not here: no cross-site synchronous anything, because for almost every workload the honest requirement is “survive the site with a few hours of loss” and this meets it without a second Ceph cluster.

Two sites, seconds-scale RPO genuinely required

Ceph at both sites, RBD mirroring between them - covers site loss with a seconds RPO.
PBS at both sites, syncing - covers deletion, corruption and ransomware, which mirroring cannot.
A documented promotion runbook, because promotion is a manual Ceph operation and the guest configurations do not mirror with the images.

This is a materially larger estate to run. Adopt it when a specific workload’s requirement names it, not as a general upgrade.

Write the mapping down, in this shape

Failure	Mechanism	RPO	RTO	Last tested
Disk failure	Ceph / ZFS redundancy	0	0	2026-07-14
Node failure	HA + Ceph	0	~5 min	2026-06-02
Guest corruption	PBS local restore	24 h	~40 min	2026-08-01
Accidental deletion	PBS file-level restore	24 h	~10 min	2026-08-01
Site loss	PBS offsite restore	25 h	~8 h	2026-05-19
Ransomware	PBS offsite, pull, immutable	25 h	~8 h	never

The value is in the last two columns. A row whose “last tested” is empty is a belief, and the row that is always empty is the one that matters most.

Knowledge check

Why this matters

The word “replication” does more damage in DR conversations than any other, because it sounds like a single capability and is really three, with the property people most care about - surviving a mistake - present in only one of them.

The failure this produces is specific and common. An estate invests in low-latency replication, achieves a genuinely impressive RPO against hardware loss, and reasons that its backup requirements are now softer. Then someone deletes something, the deletion is at the DR site within seconds, and the recovery depends entirely on the backup regime that was quietly deprioritised.

The discipline that prevents it is the table at the end of this lesson: enumerate the failures, name one mechanism per failure, and record when each was last tested. It takes an hour, it fits on a page, and it makes two things visible that no architecture diagram does - which failures have no owner, and which recovery paths nobody has ever exercised.