RBD mirroring for cross-site replication · Proxmox VE

A size 3 pool with a host failure domain survives a node. A CRUSH rule with a rack failure domain survives a rack. Neither survives the site, because every copy is still inside one RADOS cluster and one cluster has one fate.

RBD mirroring is the answer to that: asynchronous replication of block images between two independent Ceph clusters. It is the piece the disaster-recovery part needs from Ceph, and it is entirely unmanaged by Proxmox — no GUI, no pveceph subcommand, no storage plugin. You configure it with rbd on both clusters and you operate it yourself.

That last point is the one to internalise before anything else in this lesson. Everything here is a Ceph procedure that Proxmox will neither help with nor warn you about.

Two mechanisms, and only one of them is practical for VM disks

	Journal-based	Snapshot-based
Required image features	`exclusive-lock`, `journaling`	`exclusive-lock`
Write path cost	Every write is written twice	None on the write path
RPO	Near-continuous	The snapshot interval
Granularity	Every write, in order	Delta between two mirror snapshots
Enabled	Per pool or per image	Per image

The write-path row decides it. The upstream documentation is blunt: “Since each write to the RBD image will result in two writes to the Ceph cluster, expect write latencies to nearly double while using the RBD journaling image feature.”

Doubling write latency on the pool that holds your VM disks is not a trade most estates will accept for an RPO measured in seconds rather than minutes. Snapshot-based mirroring costs nothing on the write path; it takes a point-in-time mirror snapshot on a schedule and ships the delta between it and the last one.

Pool mirror mode interacts with this. “When configured in pool mode, all images in the pool with the journaling feature enabled are mirrored.” Since your images will not have that feature, pool mode mirrors nothing. Snapshot mirroring is enabled per image, so in practice you set the pool to image mode and opt images in explicitly. That is also the safer arrangement: a new VM disk is not silently replicated across a WAN link because somebody created it in the wrong pool.

Building it

Two clusters. Call them site-a (primary) and site-b (secondary). Both need the rbd-mirror package; the daemon itself runs on the cluster that receives data.

1. Enable mirroring on the pool, on both sides

Configuration changeon site-a and again on site-b— Enables the mirroring subsystem for the pool and records this cluster's site name. The site name is how each side identifies itself in peer relationships and in status output; pick something that will still make sense in an incident.

POOL=pve-vms

# on site-a
rbd mirror pool enable --site-name site-a "$POOL" image

# on site-b
rbd mirror pool enable --site-name site-b "$POOL" image

The pool must exist on both clusters with the same name, and the secondary’s pool should have the same size and min_size as the primary. A DR copy on a size 2 pool is a DR copy you cannot trust.

2. Bootstrap the peer relationship

The bootstrap token carries the connection details and a cephx credential, so it is a secret. Move it the way you would move a private key.

Configuration changeon site-a: create the token— Creates a token file containing monitor addresses and a generated cephx key for the peer. Anyone holding this file can read and write the pool on this cluster.

POOL=pve-vms
rbd mirror pool peer bootstrap create \
--site-name site-a "$POOL" > /root/site-a-token

Configuration changeon site-b: import it— Establishes the peering. rx-only means site-b pulls from site-a and does not push back; rx-tx sets up two-way mirroring where each site can hold the primary copy of different images.

POOL=pve-vms
rbd mirror pool peer bootstrap import \
--site-name site-b \
--direction rx-only \
"$POOL" /root/site-a-token

shred -u /root/site-a-token

Pick the direction deliberately. rx-only is right when site-b exists solely to receive; it means site-b cannot accidentally push a stale image back. rx-tx is for an active/active estate where some workloads are primary at each site, and it is meaningfully harder to reason about during an incident.

3. Run the daemon on the secondary

Configuration changeon site-b: create the daemon user and start it— The daemon is what actually moves data. Without it, mirroring is configured and nothing replicates - a state that produces no error anywhere until you look at mirror status.

apt install -y rbd-mirror

ID=site-b
ceph auth get-or-create "client.rbd-mirror.$ID" \
mon 'profile rbd-mirror' \
osd 'profile rbd' \
-o "/etc/ceph/ceph.client.rbd-mirror.$ID.keyring"

systemctl enable --now "ceph-rbd-mirror@rbd-mirror.$ID"

Run it on more than one node at the secondary site. The daemons elect among themselves and share the image set, so a second one is both capacity and redundancy — and a mirroring pipeline that stops because one host rebooted is a pipeline that will be behind when you need it.

4. Enable and schedule per image

Configuration changeon site-a: opt an image in— Enables snapshot-based mirroring for one image and sets how often a mirror snapshot is taken. The interval suffix is d, h or m for days, hours or minutes.

POOL=pve-vms
IMAGE=vm-101-disk-0

rbd mirror image enable "$POOL/$IMAGE" snapshot

rbd mirror snapshot schedule add \
--pool "$POOL" --image "$IMAGE" 15m

A schedule set at pool level (--pool without --image) applies to every mirrored image in it, which is usually what you want once you have more than a handful.

Reading the status, which is the whole job

Configuration is a one-off. Watching replication is the ongoing work, and mirroring fails in the specific way that it keeps reporting success while falling further behind.

Read-only / Safethe daily check— Read-only. Run it on the secondary. The health line and the per-image state are what tell you whether your RPO is what you think it is.

POOL=pve-vms
rbd mirror pool status --verbose "$POOL"
rbd mirror pool info "$POOL"

Read-only / Safe— Illustrative capture. The two fields that matter are state and the gap between local_snapshot_timestamp and remote_snapshot_timestamp.

$ rbd mirror pool status --verbose pve-vms

health: OK
daemon health: OK
image health: OK
images: 12 total
  12 replaying

DAEMONS
service 4141:
instance_id: 42133
client_id: site-b
hostname: pve-dr-01
version: 19.2.3
leader: true
health: OK

IMAGES
vm-101-disk-0:
global_id:   b7c1e1a4-0000-0000-0000-000000000000
state:       up+replaying
description: replaying, {"bytes_per_second":18432000.0,"last_snapshot_bytes":8123456789,"last_snapshot_sync_seconds":412,"local_snapshot_timestamp":1786000000,"remote_snapshot_timestamp":1786000900}
last_update: 2026-08-12 09:41:02

Illustrative output

Three fields carry all the meaning.

state. up+replaying is healthy: the daemon is running and the image is being synchronised. up+stopped on the secondary means the image is primary here, which after a failover is correct and before one is alarming. down+* means no daemon is servicing the image.

last_snapshot_sync_seconds against the schedule interval. If a 15-minute schedule takes 412 seconds to ship, you have headroom. If it takes 900, you are exactly at the limit and the next busy hour puts you behind.

The gap between the two timestamps. That difference, in seconds, is your actual RPO for that image right now. It is the number to alert on, and it is the only honest answer to “how much would we lose”.

Mirroring reports OK while the RPO quietly becomes eight hours

The failure mode is not an error. It is a backlog.

A change rate that exceeds what the inter-site link can carry means each scheduled snapshot ships more than the interval allows, so the next one starts late, and the lag grows monotonically. health: OK stays OK throughout — every snapshot that completes, completes successfully. state stays up+replaying, because it genuinely is replaying.

Nothing in Ceph raises a warning for “the RPO you documented is no longer true”. The only signal is arithmetic on the timestamps, and it is the reason to alert on the timestamp delta rather than on health.

The usual trigger is not steady growth. It is one event: a large migration onto the pool, a database reindex, a Windows patch cycle across two hundred guests. The backlog appears in an afternoon and takes a week to drain, and nobody notices until the DR test.

Failover

There are two procedures and the difference between them is whether the primary site is reachable. Running the wrong one is how estates lose data.

Planned: the primary is alive

Service impact possibleon site-a: demote, then let the last delta ship— Demoting makes the image non-primary and writable by nobody. The VM must be shut down first; demoting an image a running VM has open is a good way to corrupt a filesystem.

POOL=pve-vms
IMAGE=vm-101-disk-0
VMID=101

qm shutdown "$VMID"
rbd mirror image demote "$POOL/$IMAGE"

# take a final mirror snapshot so the delta since the last scheduled one ships
rbd mirror image snapshot "$POOL/$IMAGE"

Wait for the secondary to report the image as up+unknown or otherwise show that it has caught up, then promote there:

Service impact possibleon site-b: promote— Makes the local copy primary and writable. No --force is needed because site-a propagated its demotion, so there is exactly one primary at all times.

POOL=pve-vms
IMAGE=vm-101-disk-0

rbd mirror image status "$POOL/$IMAGE"
rbd mirror image promote "$POOL/$IMAGE"

Zero data loss, because the demotion was propagated and the final snapshot shipped.

Unplanned: the primary is gone

Data-loss riskon site-b: forced promotion— Use only when site-a is unreachable and you have decided to accept the loss of everything written since the last mirror snapshot. This creates a split-brain that is resolved later by discarding one side.

POOL=pve-vms
IMAGE=vm-101-disk-0

rbd mirror image status "$POOL/$IMAGE"
rbd mirror image promote --force "$POOL/$IMAGE"

The documentation states what you are doing: “Forced promotion is needed when the demotion cannot be propagated to the peer Ceph cluster… This will result in a split-brain scenario between the two peers and the image will no longer be in-sync until a force resync command is issued.”

You lose everything written to the primary since the last mirror snapshot completed — which is why the timestamp delta you were monitoring is the number that tells you, before you press it, how much that is.

Failback

When site-a returns, both clusters believe their copy is primary. Ceph will not merge them. You choose which one survives, and the other is overwritten:

Data-loss riskon the site whose copy you are discarding— Demote the losing copy, then request a resync. The image is re-synchronised from the surviving primary and everything written to the losing copy during the split is gone. Confirm you have identified the right side before running this.

POOL=pve-vms
IMAGE=vm-101-disk-0

rbd mirror image demote "$POOL/$IMAGE"
rbd mirror image resync "$POOL/$IMAGE"

Under the hood

Why the primary flag lives in the image, and what “split-brain” means here mechanically.

Mirroring adds two pieces of state to an RBD image. The first is a global ID: a UUID that identifies the image as the same logical image across both clusters, independent of pool or name. The second is a primary flag stored in the image metadata, which says whether this cluster’s copy is the writable one.

Exactly one copy should carry the primary flag. The rbd-mirror daemon refuses to write to a primary image, which is what prevents the secondary from being overwritten while it is in charge — and equally what prevents replication into a copy you promoted.

A clean demotion works because it is replicated. rbd mirror image demote clears the flag locally and the change propagates to the peer, so at no point do two copies claim it. A forced promotion sets the flag locally with no way to tell the peer. When the peer comes back, both copies have the flag set and the same global ID, and Ceph has no way to decide which one is correct — because there genuinely is no correct answer. Both received real writes.

That is what split-brain is here: not corruption, but two divergent lineages of the same image. rbd mirror image resync resolves it by declaring one lineage authoritative and discarding the other entirely, re-synchronising from scratch or from the last common mirror snapshot.

Snapshot-based mirroring layers on top of this. Each mirror snapshot is a normal RBD snapshot with mirroring metadata, and the daemon ships the delta between the last snapshot the secondary holds and the newest one on the primary. This is rbd export-diff semantics, which is why the transfer size tracks changed data rather than image size, and why the first synchronisation of a large image is enormous while subsequent ones are small.

It also explains the deletion behaviour that surprises people: the secondary retains the mirror snapshots it needs to stay consistent, so freeing space at the DR site by deleting snapshots there breaks the delta chain and forces a full resynchronisation.

What happens if you fail over and the VM will not start at the DR site?

Almost certainly because the disk arrived and the VM did not.

RBD mirroring replicates images. It does not replicate /etc/pve/qemu-server/101.conf, and it has no idea that file exists. On the DR cluster there is a perfectly good vm-101-disk-0 in the pool and nothing that knows it is a VM: not its memory size, not its NIC model, not its boot order, not its machine version, not the CPU type it was running under.

This is the single most common way a Ceph DR plan fails its first test, and it fails at the worst moment, because the images look correct and everyone assumed the hard part was done.

What you do about it, in the order that they matter:

Replicate the guest configurations separately. They are small text files. A cron job that copies /etc/pve/qemu-server/ and /etc/pve/lxc/ to the DR site every few minutes costs nothing and is the entire fix. Store them somewhere that survives the primary site, not on a node in it.

Check the machine version specifically. A VM pinned to a machine version, or one that was running an implicit one, may not boot the same way under a different QEMU. Recreating the VM by hand at 3 AM with defaults produces a guest that boots to a different virtual chipset than the one its Windows installation was activated against.

Pre-create the VMs at the DR site, stopped. Same VMID, same configuration, disks pointing at the mirrored pool. They cannot start — the images are non-primary and unwritable — but the definitions are already correct, and failover becomes promote-then-start rather than promote-then-reconstruct.

Test it. Promote one image at the DR site, start its VM, confirm it boots, then demote and resync. That exercise takes an hour and is the only thing that converts a DR design into a DR capability.

Mirroring is not backup, and the difference is not academic

Both make a second copy. They protect against different things, and substituting one for the other leaves a real hole.

	RBD mirroring	Proxmox Backup Server
Protects against	Loss of a site or a cluster	Loss of data, at a point in time
RPO	Minutes	Hours to a day
Recovery point choice	The current state only	Any retained snapshot
Ransomware in a guest	Replicated within minutes	Recoverable from before it
Accidental `rm -rf`	Replicated within minutes	Recoverable from before it
Guest configuration	Not included	Included in the backup
Verification	Manual failover test	`proxmox-backup-client` verify jobs

Mirroring is a faithful copy, which means it faithfully copies your mistakes. An encrypted filesystem at the primary site is an encrypted filesystem at the DR site fifteen minutes later, and the DR copy is now the only copy of the encrypted data.

The estates that get this right run both: mirroring for the site-loss scenario with a short RPO, PBS for everything else with retention and verification. They are complementary and neither is optional if you have promised both an RTO and a recovery-point choice.

Common mistakes

Turning on journal-based mirroring for VM disks. Write latency nearly doubles, permanently, on the pool that hosts everything.
Configuring mirroring and not starting rbd-mirror. Everything reports as configured and nothing replicates.
Alerting on health: OK. It stays OK while the RPO grows. Alert on the timestamp delta.
Forced promotion when the primary was merely unreachable. A network partition that heals leaves you with two divergent copies and a choice about which day of work to delete.
Mirroring the images and not the guest configurations. The disks arrive; the VMs do not exist.
A DR pool at size 2 to save money. The copy you fail over to has less redundancy than the one you lost.
Deleting mirror snapshots at the DR site to free space. It breaks the delta chain and forces a full resynchronisation across the WAN.
Treating it as backup. It replicates deletion and encryption as faithfully as it replicates anything else.

Key takeaways

Snapshot-based mirroring is the practical choice for Proxmox VM images; journal-based mirroring nearly doubles write latency.
Pool mode only mirrors images with the journaling feature, so use image mode and opt in per image.
The rbd-mirror daemon runs at the receiving site; run more than one.
Your real RPO is the gap between local_snapshot_timestamp and remote_snapshot_timestamp, not the schedule you configured.
Planned failover is demote-then-promote and loses nothing. Forced promotion loses everything since the last mirror snapshot and creates a split-brain resolved by resync discarding one side.
Mirroring replicates images, not guest configurations. Replicate /etc/pve/qemu-server/ separately or pre-create the VMs.
Mirroring and PBS solve different problems. Run both.

Knowledge check

Why this matters

Ceph is so good at surviving hardware failure that it is easy to stop thinking about failure altogether. Three copies, self-healing, automatic rebalancing — the cluster handles a dead disk without anyone noticing and a dead node with a health warning. After a couple of years of that, “what happens if we lose the site” stops being asked, because nothing has gone wrong in a long time.

Mirroring is where that reasoning has to stop. It is the first Ceph feature that does not heal itself, does not warn you when it is degraded, and has a failure mode — forced promotion — whose correct resolution is deliberately deleting data. Every safety property you have come to rely on from Ceph is absent here.

The specific things to carry away are the three that are easy to get wrong quietly:

Configured is not running. A pool with mirroring enabled, a peer registered, and no rbd-mirror daemon reports healthy and replicates nothing.

Healthy is not current. health: OK and a two-hour backlog are the same string.

A disk is not a VM. The most thorough mirroring setup in the world fails its first real test if /etc/pve/qemu-server/ never crossed the WAN.

All three are found by the same exercise: promote an image at the DR site, boot the VM, look at it, demote and resync. Do it once a quarter and none of the three can survive undetected. Skip it and you will discover all three on the same night.