Site failure design and DR site architecture

Why this matters in production

Most DR architectures fail when activated, not when designed. People forget passwords. DNS doesn’t update. Networks are misconfigured. This lesson teaches the design that survives real disasters.

Site architecture options

flowchart LR
  subgraph PRIMARY[Primary site]
    P1[Cluster A]
    PBSP[Primary PBS]
  end
  subgraph DR[DR site]
    P2[Cluster B standby]
    PBSR[Remote PBS]
  end
  PBSP -->|sync hourly| PBSR
  P1 -. replication .-> P2

Choosing the right option

For most businesses:

• PBS off-site only covers ransomware and accidental deletion.
• PBS + cold DR cluster covers site failure with RTO 1-4 h.
• PBS + warm DR cluster covers site failure with RTO < 30 min.
• Active/active is the rarest; reserved for the most critical workloads.

DR site components

A useful DR site needs:

• Compute: at least enough capacity to run critical VMs.
• Storage: PBS datastores with synced backups.
• Network: IPs, VLANs, firewall rules matching production.
• DNS: ability to redirect external traffic.
• Identity: restored or sync’d from primary.
• Credentials: vault of administrative credentials.
• Documentation: offline access to runbooks and IPs.

The components people forget, and what each costs when missing

Compute and storage are on everyone’s list. These are the ones that turn a designed DR site into a working one, and each has a specific failure when absent.

The pattern: every dependency that the primary site quietly supplied is a dependency the DR site must supply itself. The ones that hurt are invisible in normal operation precisely because the primary always provided them.

set -euo pipefail
PBS_HOST=pbs-dr.example.com
NODES='dr-01 dr-02 dr-03'

echo '--- can we resolve, without the primary site ---'
getent hosts "$PBS_HOST" || echo "FAIL cannot resolve $PBS_HOST"
grep -E '^nameserver' /etc/resolv.conf

echo '--- is the clock right, from a source that survives ---'
timedatectl show -p NTPSynchronized -p TimeUSec
grep -rhE '^(NTP|FallbackNTP)=' /etc/systemd/timesyncd.conf* 2>/dev/null || true

echo '--- is the backup server reachable and is its fingerprint known ---'
pvesm status --storage pbs-dr || echo 'FAIL pbs-dr storage not usable'

echo '--- do the bridges production expects exist here ---'
for BR in vmbr0 vmbr1 vmbr2; do
ip link show "$BR" >/dev/null 2>&1 \
  && printf '%s present\n' "$BR" \
  || printf '%s MISSING\n' "$BR"
done

echo '--- can we reach every DR node out of band ---'
for N in $NODES; do
printf '%-8s ' "$N"
ping -c1 -W2 "${N}-ipmi" >/dev/null 2>&1 && echo 'IPMI OK' || echo 'IPMI UNREACHABLE'
done

Cold vs warm DR cluster

A warm DR cluster can run a subset of services continuously. A cold DR cluster spins up on demand.

Network and DNS cutover

When activating DR:

1. Update DNS records to point to the DR site’s public IPs (if applicable).
2. Activate firewall rules allowing external traffic to DR.
3. Bring up DR cluster nodes and PBS.
4. Restore VMs in dependency order.
5. Verify each tier before moving to the next.

Production considerations

Common mistakes

• DR site without documented credentials.
• Long DNS TTLs that delay cutover.
• DR cluster hardware that hasn’t been booted in months.
• DR exercises that “work on paper” but fail in practice.

Key takeaways

• PBS off-site is the minimum.
• Warm DR for tight RTO; cold DR for moderate RTO.
• DNS TTL is a recovery-time floor.

Knowledge check