Designing a multi-site DR strategy with PBS and ZFS replication

DR is not a single product — it’s a designed combination of PBS sync jobs, ZFS replication, off-site storage, and documented procedures. Get any one of these wrong and your RTO/RPO promise breaks.

This lesson walks through a complete DR design for a hypothetical two-site cluster and shows how to run the kind of exercise that catches design flaws before a real disaster does.

The target: 2 sites, 30-min RTO, 15-min RPO

The textbook target for SMB DR:

• RPO (Recovery Point Objective): 15 minutes — at most 15 minutes of writes lost in a disaster.
• RTO (Recovery Time Objective): 30 minutes — services restored within 30 minutes of declaring a disaster.

Both numbers drive every architectural choice. Tighter RPO/RTO means more synchronous replication, more sites, more cost.

Site topology

Site A (Primary)               Site B (DR)
─────────────────              ─────────────────
3× PVE nodes                   1× PVE node (DR)
1× PBS primary                 1× PBS secondary
ZFS mirror pool                ZFS mirror pool
100 GbE interconnect           100 GbE interconnect
                              
PBS sync: Site A → Site B, every 15 minutes
ZFS async replication: Site A → Site B, every 15 minutes

The asymmetric sizing (3 nodes primary, 1 node DR) is common: DR capacity is for emergency operation, not normal load. The DR node should be sized to run the most critical VMs at minimum capacity, not the full primary workload.

Replication layers

Three replication paths, each with different characteristics:

1. PBS sync jobs (chunk-level, dedup-aware)

# On PBS at Site B, add the Site A PBS as a remote
proxmox-backup-manager remote create site-a-pbs \
  --host pbs-a.site-a.example.com \
  --user sync@pbs \
  --password <password> \
  --fingerprint <ssl-fingerprint>

# Create a sync job
proxmox-backup-manager sync-job create site-a-to-b \
  --remote site-a-pbs \
  --store-name main \
  --remote-store-name main \
  --schedule '*/15' \
  --max-depth 5 \
  --remove-vanished true

PBS sync streams only new chunks from Site A to Site B. With dedup, a 1 TB VM with 200 GB of unique blocks sends only ~200 GB the first time, and only the changed chunks on subsequent syncs.

Properties:

• Bandwidth-efficient (dedup + incremental)
• Survives WAN link (uses HTTPS)
• Restore time: minutes to hours depending on dataset size
• RPO: tied to sync interval (15 min in our target)
• Application-aware: stores VMs as VMs, not as blocks; PVE-side restore directly

2. ZFS send/receive (block-level)

For the cluster filesystem itself (config, ISO templates, custom images):

# On a primary node, snapshot and send to DR
zfs snapshot tank/vm-storage@dr-$(date +%Y-%m-%d-%H%M)
zfs send tank/vm-storage@dr-$(date +%Y-%m-%d-%H%M) | \
  ssh pve-dr "zfs receive dr-pool/vm-storage"

# Or use PVE's built-in replication:
# Datacenter → Replication → Add → schedule every 15 minutes

Properties:

• Bandwidth-efficient (incremental)
• Block-level: the DR site can attach the VM disk immediately, no restore step
• Faster recovery than PBS — start the VM from the local ZFS replica
• RPO: tied to replication interval

3. Cold storage backup (immutable, off-WAN)

For the deepest layer — backups that survive even a ransomware attack on the DR site:

# Weekly PBS export to immutable storage (S3 with Object Lock,
# tape, or detached disk)
# On Site B PBS:
proxmox-backup-manager sync-job create site-b-to-s3 \
  --remote s3-cold \
  --store-name main \
  --schedule 'weekly' \
  --keep-last 12

The cold-storage backup is your last-line defence. It should be disconnected from any network that the cluster can reach.

Failover procedure

When Site A is unavailable:

# 1. Declare disaster
# 2. Verify Site A is truly down (not just network split)
#    - Ping, IPMI, physical check, etc.
# 3. Promote Site B PBS to primary
proxmox-backup-manager datastore promote main --remote site-a-pbs

# 4. On the DR PVE node, attach the replicated ZFS pool
zpool import -f dr-pool
pvesm add zfspool vm-storage --pool dr-pool/vm-storage --content images,rootdir

# 5. Start critical VMs
qm start 100  # web
qm start 200  # db
# ... etc.

# 6. Update DNS to point to DR site IPs
# 7. Notify users that we're on DR

# 8. Document timeline in incident log

Total time: 20–40 minutes for a well-prepared DR. The longest part is usually DNS propagation and user communication.

Failback procedure

When Site A is recovered:

# 1. Verify Site A hardware is functional
# 2. Re-install PVE on Site A nodes (if needed) and rejoin cluster
# 3. Reverse-replicate from Site B to Site A
#    - PBS: re-add Site B as a remote and pull
proxmox-backup-manager sync-job create site-b-to-a \
  --remote site-b-pbs \
  --store-name main \
  --schedule 'hourly'

#    - ZFS: send the latest snapshots back
zfs send -i dr-pool/vm-storage@latest dr-pool/vm-storage@latest2 | \
  ssh pve-a "zfs receive tank/vm-storage"

# 4. Migrate VMs back to Site A
qm migrate 100 pve-a-01 --online
qm migrate 200 pve-a-02 --online

# 5. Update DNS back to Site A
# 6. Re-establish normal replication (Site A → Site B)

# 7. Update runbooks with any lessons learned

Failback is more complex than failover because you have to reverse-replicate the data accumulated on Site B during the disaster. Plan for the failback window to be longer than the failover window.

Designing the runbook

A real DR runbook has:

• Trigger conditions — what events declare a disaster? Power loss at Site A for >30 minutes? Ransomware detection? Provider outage?
• Decision authority — who calls DR? The on-call engineer, the team lead, the CTO?
• Communication plan — what gets announced, to whom, in what order? Internal users first, then customers, then vendors.
• Time-boxed steps — each step has a target time. If a step exceeds its time, escalate.
• Rollback conditions — under what conditions do we abort the failover and try again later?
• Validation checklist — how do we know the failover is successful? Specific checks: VMs respond to ping, application health checks pass, database replicas are caught up.

The DR exercise

Twice a year, run a full DR exercise. The exercise should be:

• Scheduled — not during production incidents
• Realistic — actually fail over, not just walk through the runbook
• Measured — time every step; compare against target
• Followed by failback — to test the harder direction

# Exercise day timeline:
# 09:00 - announce exercise window
# 09:15 - simulate Site A power loss (IPMI off)
# 09:30 - declare disaster; begin failover
# 10:00 - failover complete (target: 30 min from declaration)
# 10:00 - 12:00 - validate DR site (applications, monitoring)
# 12:00 - announce failback window
# 12:30 - reverse-replicate
# 14:00 - failback complete (target: 90 min)
# 14:00 - 16:00 - validate primary site
# 16:00 - exercise debrief

Capture every step’s actual time and any deviation from the runbook. Use the exercise to update the runbook.

Common mistakes

• No tested DR. A DR plan that has never been executed is a wish, not a plan. Test quarterly at minimum.
• DR site too small. A DR site sized for the full primary workload is expensive. Sized for “the most critical VMs at reduced capacity” is the right trade-off.
• Replication direction wrong. ZFS replication and PBS sync both push from primary to DR. Failback requires explicit reverse-pull.
• DNS not automated. Manual DNS changes during a disaster are slow and error-prone. Use a TTL-tuned setup or dynamic DNS.
• No failback plan. Failover is the visible half of DR. Failback is the half that takes longer and is more likely to fail.

Production considerations

• Replication lag monitoring. Alert when PBS sync is more than 1 hour behind, or ZFS replication hasn’t run in 2 hours.
• Capacity planning. DR site needs CPU, RAM, and storage for the critical subset of VMs. Estimate at 60–70% of primary capacity to leave headroom.
• Network cost. Cross-site replication on a 100 Mbps WAN bottleneck takes 10× longer than on 10 Gbps LAN. Budget for replication bandwidth separately from production traffic.
• Immutability. The cold storage layer must be immutable (S3 Object Lock, tape, or write-once media). Otherwise ransomware encrypts the backups too.

Key takeaways

• Combine PBS sync + ZFS replication + cold storage for layered DR.
• Failover is 20–40 minutes. Failback is longer.
• Test quarterly with a full exercise including failback.
• The DR plan that has never been tested is not a plan.

Knowledge check