SAN vs distributed vs replicated — the decision framework

Why this matters in production

The wrong storage backend multiplies every other operational cost. Migrating between backends later is expensive (data movement, downtime). Choosing the right backend at the start saves months of pain.

The three approaches

flowchart TB
  subgraph SAN[Shared SAN]
    N1a[pve-01] --> L[Shared LUN]
    N2a[pve-02] --> L
    N3a[pve-03] --> L
  end
  subgraph DIST[Distributed: Ceph]
    N1b[pve-01 + OSDs] --> R[RADOS]
    N2b[pve-02 + OSDs] --> R
    N3b[pve-03 + OSDs] --> R
  end
  subgraph REP[Replicated: ZFS replication]
    N1c[pve-01 zvol] -->|zfs send| N2c[pve-02 zvol]
  end

Decision framework

Hybrid storage in one cluster

A single Proxmox cluster can use multiple storage backends simultaneously. A common pattern:

• Ceph RBD for VM disks on the same nodes (distributed, replicated).
• ZFS local for fast scratch / non-critical VMs.
• NFS for ISO/template storage.
• PBS for backup target.

flowchart TB
  subgraph CL[Cluster]
    VM1["VM 100 (Postgres)"] --> CEPH[Ceph RBD]
    VM2["VM 101 (Dev)"] --> ZFS[Local ZFS]
    ISO[ISO library] --> NFS[NFS]
  end
  CEPH --> BACKUP[PBS]

Migrating between backends

Common migrations:

qm move-disk copies the disk while the VM runs. Once complete, switch the active disk. For non-shared source storage, the VM must be stopped.

Production considerations

Common mistakes

• Choosing Ceph for 10 VMs because “Ceph is distributed.” The complexity outweighs the benefit at that scale.
• Choosing local-only storage for a cluster that needs live migration.
• Ignoring the operational complexity of the chosen backend until it is in production.
• Migrating between backends without testing the rollback.

Key takeaways

• SAN, distributed, and replicated each have a place.
• Pick based on workload requirements and team competence.
• One cluster can use multiple backends deliberately.

Knowledge check