Shared storage for live migration: LVM-Thin, Ceph, and NVMe-oF

Live migration requires that a VM’s disk be accessible from every node that might run it. Three patterns cover most production deployments:

• LVM-Thin on iSCSI — file-on-block, simple, works for small to medium clusters
• Ceph RBD — distributed block storage, scales to large clusters, more complex
• NVMe-oF/TCP — the modern replacement for iSCSI, line-rate on 100 GbE

This lesson compares the three and shows when each is the right choice.

What “shared storage” actually means in PVE

PVE’s live migration (offline and online) moves the VM’s memory and CPU state from one node to another. The disk must already be accessible from both nodes — that’s “shared”. PVE supports three forms of shared storage:

1. Cluster filesystem — same directory mounted on every node (NFS, CIFS, GlusterFS). The VM disk is a file. Live migration works without copying.
2. Shared LVM — same logical volume visible on every node (iSCSI-backed). The VM disk is a block device. PVE handles the sharedness via CLVM.
3. Shared Ceph RBD — the Ceph cluster provides the disk; every node has access through librbd. The VM disk is a Ceph RADOS object.

Local-only storage (ZFS pool, ext4 directory) cannot host a VM that migrates between nodes — you’d have to copy the disk image, which defeats the purpose of live migration. PVE’s “storage” types clearly mark which support shared:

pvesm status
# Shared: cephfs, ceph, nfs, cifs, glusterfs, iscsi, lvm
# Local:  dir, zfspool, btrfs, lvm-thin (without shared), etc.

LVM-Thin on iSCSI: the simple shared backend

The classic PVE pattern: an iSCSI LUN, an LVM physical volume on top, an LVM volume group, and an LVM-Thin pool. Every node sees the same LV via iSCSI multipath.

# On one node (after iSCSI multipath is configured)
pvs /dev/mapper/mpath0
vgcreate vg-shared /dev/mapper/mpath0
lvcreate -L 1.95T -n lv-shared vg-shared
lvconvert --type thin-pool vg-shared/shared-pool --poolmetadatasize 1g

# Add to cluster as shared LVM-Thin
pvesm add lvmthin shared-lvm --vgname vg-shared --poolname shared-pool --thinpool --shared yes

The --shared yes flag is the magic: PVE registers the storage as shared, and every node can see it because every node mounts the same iSCSI LUN.

Pros:

• Simple. LVM-Thin is well-understood.
• Snapshots are cheap (LVM-Thin CoW).
• Works with any iSCSI backend (TrueNAS, NetApp, generic Linux target).

Cons:

• Single LUN = single point of failure unless you use multipath (which is also mandatory for production).
• Scales to ~10 nodes before the iSCSI target becomes a bottleneck.
• No replication across sites without manual intervention.

Right for: small to medium clusters (3–8 nodes), modest performance requirements, single-site deployments.

Ceph RBD: the distributed backend

Ceph is the cluster-native shared storage for PVE. Every node runs an OSD daemon and serves parts of the data; RBD images are replicated across nodes according to the pool’s CRUSH rules.

# On the cluster (assuming Ceph is already configured)
pvesm add rbd shared-ceph --pool vm-storage --content images,rootdir

The pool is accessible from every node that has the Ceph client keys installed (/etc/pve/priv/ceph). No extra configuration needed — Ceph’s CRUSH map handles placement automatically.

Pros:

• Scales horizontally: add nodes, get more capacity AND throughput.
• Self-healing: a failed disk triggers automatic recovery from replicas.
• Snapshots and clones are nearly free.
• Replication across sites via CRUSH rules (e.g., 2 replicas on site A, 1 replica on site B).

Cons:

• More complex to operate. Ceph tuning is a full-time job at scale.
• Higher memory overhead (OSD daemons, MDS, MON).
• Requires dedicated storage disks (don’t share with the OS or VM data on the same disks).
• 3× replication overhead minimum (raw capacity / 3 = usable capacity).

Right for: medium to large clusters (5+ nodes), multi-site deployments, environments where storage capacity and throughput need to scale together.

NVMe-oF/TCP: the modern high-performance backend

NVMe-oF is the networked version of NVMe (the protocol SSDs use internally). Compared to iSCSI, NVMe-oF/TCP offers:

• Lower latency (no SCSI translation layer)
• Higher IOPS (multiple queues per session)
• Modern features: zoned namespaces, end-to-end data protection

PVE 9.x supports NVMe-oF on both the target side (Linux kernel NVMe-OF target) and the initiator side (kernel nvme_tcp driver). For shared storage, NVMe-oF provides block devices visible to every node that connects.

Target setup

# On the storage host
apt install -y nvme-cli

# Create a loopback NVMe device for testing (real deployments use physical NVMe)
modprobe nvmet
modprobe nvmet_tcp

# Create a subsystem and namespace
mkdir /sys/kernel/config/nvmet/subsystems/shared-nvme
cd /sys/kernel/config/nvmet/subsystems/shared-nvme
echo 1 > attr_allow_any_host
mkdir namespaces/1
ln -s /dev/nvme0n1 namespaces/1/device

# Create a port (listener)
mkdir /sys/kernel/config/nvmet/ports/1
cd /sys/kernel/config/nvmet/ports/1
echo 10.0.4.10 > addr_traddr
echo 4420 > addr_trsvcid
echo ipv4 > addr_adrfam
echo tcp > addr_trtype
ln -s /sys/kernel/config/nvmet/subsystems/shared-nvme \
   /sys/kernel/config/nvmet/ports/1/subsystems/shared-nvme

Initiator setup

On each PVE node:

modprobe nvme_tcp
nvme connect -t tcp -n shared-nvme -a 10.0.4.10 -s 4420

# Verify
nvme list
# Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
# /dev/nvme0n1     ...                  Linux NVMe-OF Target                    1          500.00 GB / 500.00 GB   512 B + 0 B    5.10.0

# Use it as a PVE storage backend (treat as block device)
pvesm add lvm nvme-shared --vgname vg-nvme /dev/nvme0n1 --shared yes

NVMe-oF performance

On a 100 GbE network with NVMe-oF/TCP, you can expect:

• Sequential read: 9–11 GB/s
• Random 4K read: 700k–1M IOPS
• Latency: 50–100 µs

That’s competitive with local NVMe and far beyond iSCSI’s ~200k IOPS at the same latency.

Pros:

• Lowest-latency networked block storage available.
• Line-rate on 100 GbE.
• Modern, well-supported by PVE 9.x.
• Multipath via multiple NVMe-oF sessions.

Cons:

• Requires 25 GbE or faster for full benefit. On 10 GbE the advantage over iSCSI is modest.
• NVMe-oF target setup is more complex than iSCSI.
• Less mature tooling — fewer appliances support it natively.

Right for: high-performance clusters (databases, VDI), 25/100 GbE networks, environments that need block storage close to local NVMe performance over the network.

Other options worth mentioning

• GlusterFS — distributed filesystem, mostly deprecated in PVE 9.x. Don’t start new deployments.
• CIFS / SMB — works for VM disks, but limited concurrent writers. Suitable for small NAS-backed deployments.
• DRBD — block-level replication between two nodes. Cheap HA but not scalable beyond 2 nodes. Mostly superseded by Ceph.
• CephFS — distributed filesystem on top of Ceph. Useful for ISO/template storage, less so for VM disks (RBD is faster).

Picking the right backend

For a brand-new cluster, Ceph RBD is the default. LVM-Thin on iSCSI is the right choice when you already have a NAS or SAN and don’t want to operate Ceph. NVMe-oF is the right choice when performance is the primary constraint and you have the network budget.

Production considerations

• Don’t mix shared and local for the same VM. A VM on shared storage doesn’t migrate to local. Pick one per VM.
• Monitor shared storage separately. A failing shared-storage backend is a cluster-wide outage, not a per-node problem. Alert on degraded pools (Ceph), path failures (iSCSI multipath), and capacity at 70%.
• Test live migration before going live. Pull a node out of the cluster and confirm the VMs migrate to the survivors. With Ceph this is automatic; with iSCSI it depends on multipath being correctly configured.
• Plan for storage maintenance. Moving a Ceph OSD, replacing an iSCSI controller, or migrating a Ceph pool all cause brief performance impact. Schedule during off-hours.

Common mistakes

• Using local-only storage and expecting migration. PVE will refuse to migrate; you’ll waste hours before realising.
• Single-path iSCSI for shared storage. One cable failure = cluster outage.
• Mixing Ceph and local storage without explicit per-VM rules. A VM that lives on local storage can’t migrate; one on shared can. Be explicit about the choice.
• Skipping capacity planning. Shared storage fills up faster than you expect, especially with snapshots. 70% used is the warning threshold; 80% is the emergency.

Key takeaways

• LVM-Thin on iSCSI for small clusters, Ceph for medium+, NVMe-oF for performance.
• Every node must see the same disk — that’s the whole point.
• Test live migration before going live.
• Alert on shared-storage health; it’s cluster-wide.

Knowledge check