Reference architectures: small, medium, hyperconverged, SAN-backed, DR

Why this matters in production

Reference architectures save you from reinventing every decision. They also constrain your options to designs that have been proven to work — which is most of what you want when the business is depending on the platform.

Architecture 1: Small business — 3-node cluster

flowchart TB
  subgraph CL[3-node cluster]
    N1[pve-01] --- N2[pve-02]
    N2 --- N3[pve-03]
    N1 --- N3
  end
  PBS[PBS backup server]
  N1 -->|nightly| PBS
  N2 -->|nightly| PBS
  N3 -->|nightly| PBS

Characteristics

• Each node has local storage (ZFS mirror or RAID).
• VMs use local storage; cross-node migration moves the disk to the destination first.
• PBS (separate physical box, ideally off-site or on different failure domain) holds backups.

Failure modes

• Lose a node → HA restarts VMs on survivors if shared storage is in use. With local storage and replication, manual recovery is required.
• Lose two nodes → quorum lost; cluster freezes.

When to use

• 5–25 VMs, single site, modest RPO/RTO. Backup-driven recovery is acceptable.

Architecture 2: Medium environment — 5–8 compute nodes

flowchart TB
  subgraph C[Compute cluster: 5-8 nodes]
    N1[pve-01] --- N2[pve-02]
    N2 --- N3[pve-03]
    N3 --- N4[pve-04]
    N4 --- N5[pve-05]
    N1 --- N5
  end
  subgraph S[Shared storage]
    SAN[NFS / iSCSI SAN]
  end
  PBS[PBS]
  N1 --> SAN
  N2 --> SAN
  N3 --> SAN
  SAN --> PBS

Characteristics

• Dedicated shared storage (NFS, iSCSI, FC). Each compute node sees the same volumes.
• VMs can live-migrate freely.
• Backup target is PBS, separated from the SAN.

Failure modes

• Compute node failure → VMs restart on survivors via HA. SAN keeps disks online.
• SAN failure → all VMs freeze simultaneously. This is the dominant availability risk.
• Network failure → quorum may be lost.

When to use

• 25–200 VMs, single site, requires live migration. The team has SAN operational competence.

Architecture 3: Hyperconverged — Proxmox + Ceph

flowchart TB
  subgraph HC[Compute + storage on the same nodes]
    N1[pve-01 + Ceph OSDs]
    N2[pve-02 + Ceph OSDs]
    N3[pve-03 + Ceph OSDs]
    N4[pve-04 + Ceph OSDs]
  end
  N1 --> N2
  N2 --> N3
  N3 --> N4
  N4 --> N1
  PBS[PBS off-site]
  N1 -->|nightly| PBS

Characteristics

• Each node contributes disks to a Ceph cluster.
• VMs use RBD or CephFS; live migration just moves RAM.
• No separate storage tier.

Failure modes

• Lose a node → Ceph recovers from the surviving OSDs; compute VMs restart on survivors via HA.
• Lose a disk → Ceph backfills from replicas.
• Lose two nodes simultaneously → some PGs may go incomplete. Recovery depends on replication factor and CRUSH placement.

When to use

• 50+ VMs, need distributed storage, want to consolidate the storage and compute teams.

Architecture 4: External shared storage (SAN/NFS)

Same as Architecture 2, but the SAN is its own appliance — NetApp, Pure, Dell PowerStore, HPE Nimble, or a third-party Ceph/S3 cluster. Proxmox connects via NFS, iSCSI, or FC.

When to use

• You already have a competent storage team.
• Compliance requires separation of duties between storage and compute admins.
• The workload is large enough that storage and compute scale on different curves.

Architecture 5: DR architecture

flowchart LR
  subgraph PRIMARY[Primary site]
    CL1[Proxmox cluster A]
    PBS1[PBS-A]
  end
  subgraph DR[DR site]
    CL2[Proxmox cluster B]
    PBS2[PBS-B]
  end
  PBS1 -->|sync every 15 min| PBS2
  CL1 -. failover .-> CL2
  PBS2 --> CL2

Characteristics

• Two clusters, geographically separated.
• PBS sync jobs push backups from primary to DR.
• In a disaster, the DR cluster restores VMs from PBS.

When to use

• RTO ≤ 4 h, RPO ≤ 1 h, regulatory requirements for geographic separation.

Decision matrix

Common mistakes

• Buying hyperconverged Ceph for 10 VMs because “Ceph is good.” The operational overhead exceeds the benefit at small scale.
• Treating DR architecture as “the same cluster, but at a different site.” It is its own design with its own failure modes.
• Skipping capacity headroom for HA. A 3-node cluster running at 80 % utilisation cannot host the VMs of a failed node.

Key takeaways

• Five architectures cover most production use cases.
• Each has specific failure modes; pick the one whose failure modes you can absorb.
• Capacity headroom for HA is mandatory. A cluster that fits all VMs on N−1 nodes is the minimum.

Knowledge check