Choosing a storage backend: the capability matrix · Proxmox VE

Storage arguments in virtualisation teams are usually arguments about technology. They are more usefully arguments about capabilities, because a capability is something you either have on a Tuesday afternoon or do not.

Proxmox VE publishes the matrix. This lesson reads it, explains what each column costs, and turns it into four questions.

The documented matrix

Storage type	Plugin	Level	Shared	Snapshots
ZFS (local)	`zfspool`	both — VM disks are zvols	no	yes
Directory	`dir`	file	no	qcow2 only
BTRFS	`btrfs`	file	no	yes — technology preview
NFS	`nfs`	file	yes	qcow2 only
CIFS	`cifs`	file	yes	qcow2 only
Proxmox Backup	`pbs`	both	yes	n/a
CephFS	`cephfs`	file	yes	yes
LVM	`lvm`	block	no — but see below	volume chains, PVE 9+
LVM-thin	`lvmthin`	block	no	yes
iSCSI (kernel)	`iscsi`	block	yes	volume chains, PVE 9+
iSCSI (libiscsi)	`iscsidirect`	block	yes	volume chains, PVE 9+
FC / SAS	handled natively	block	yes	volume chains, PVE 9+
Ceph / RBD	`rbd`	block	yes	yes
ZFS over iSCSI	`zfs`	block	yes	yes

Three footnotes in that table carry most of the operational meaning.

Shared, for LVM over a SAN. LVM’s own “shared” column is no, but the documentation qualifies it: with iSCSI, Fibre Channel or SAS block storage as shared storage in a cluster, LVM splits the LUN into virtual disks. That is the shared-LVM-over-SAN arrangement, and it is genuinely shared — the plugin is not what shares, the LUN is.

Snapshots on file-level backends need qcow2. The documentation is explicit: on file-based storages, snapshots are possible with the qcow2 format, using either the internal snapshot function or snapshots as volume chains — and creating or deleting internal qcow2 snapshots blocks a running VM and is not an efficient operation. A raw file on NFS has no snapshot capability at all.

LVM gained snapshots in PVE 9. Since Proxmox VE 9, snapshots as a volume chain are available for VMs: the snapshot data lives in separate volumes which are layered. This is a real change to a long-standing limitation, and it is the single most likely thing to be out of date in an older runbook or in an engineer’s memory.

Snapshots and thin provisioning are the same answer

One sentence in the storage documentation removes a whole category of uncertainty: all storage types which have the Snapshots feature also support thin provisioning.

So the matrix has one fewer column than it looks. If a backend can snapshot, it can thin-provision, and both are consequences of the same underlying property — that the backend allocates on write rather than up front, and can reference blocks from more than one point in time.

The same property is what makes linked clones possible, since a linked clone is a writable reference to a shared base image. The practical rule: snapshot capability, thin provisioning and linked clones travel together.

And the same warning attaches to all three. Thin provisioning means the sum of allocated sizes can exceed the physical capacity, and the documentation is direct about the risk: over-provisioning risks data corruption if storage becomes full. A backend that can thin-provision is a backend that needs a capacity alarm.

What each capability actually buys

Shared

Shared means every node reads and writes the same data. It buys exactly two things, and they are the two biggest operational features in the product:

Live migration without copying disks. Memory moves; disks stay. Seconds rather than hours.
HA that can restart a guest anywhere. The HA stack needs the guest’s disks to be reachable from whichever node it recovers onto.

Without shared storage neither is impossible, but both change character. Live migration becomes live migration with local disks, which copies. HA becomes HA constrained to nodes that hold a replica.

Replication is not shared storage, and it has an RPO

ZFS replication with pvesr is the standard answer for clusters without shared storage, and it is a good one. It is not equivalent.

Replication sends periodic snapshots to another node. Between one run and the next, the replica is behind — and how far behind is your recovery point objective, chosen by the schedule. A 15-minute replication schedule means that an unplanned failover loses up to 15 minutes of writes, silently, with the guest starting happily on stale data.

Shared storage has no such window because there is only one copy of the data.

Both designs are legitimate. What is not legitimate is a design document that says “we have HA” without stating which of the two it means and, if it is replication, what the RPO is and who agreed to it. Part XIV is where that number gets negotiated.

Snapshots

A snapshot is a rollback point taken in seconds. What it buys is the confidence to change things — an upgrade you can undo, a configuration change you can reverse, a test you can run on the real system.

Two things it is not. It is not a backup: it lives on the same storage as the data, so it does not survive the storage failing, and on most backends it does not survive the guest being deleted. And on file-level backends it is not free: internal qcow2 snapshot creation and deletion block a running VM.

Thin provisioning

Buys density: allocate 100 guests with 100 GB disks on 4 TB of real storage, and pay only for what is written. It is genuinely useful and it is how most estates run.

The cost is that capacity becomes a thing you have to watch rather than a thing you allocated. A thin backend hitting 100% does not politely refuse new allocations — it fails writes to guests that already exist, which is a far worse failure than a provisioning refusal.

Clones

Full clones copy; linked clones reference a base image. Linked clones make template-based provisioning almost instant and almost free, at the price of a dependency: the base image cannot be deleted while linked clones exist, and the clones’ read performance depends on the base image’s.

The four questions

1. Do more than three nodes need to run this guest, and does it need HA? If yes, you need shared storage or replication. Shared: Ceph/RBD, CephFS, NFS, CIFS, or LVM over a SAN LUN. Replication: ZFS with pvesr, with an RPO you have written down.

2. Do you need to roll back? If yes, you need snapshots — which rules out raw on a directory or NFS store, and rules in ZFS, LVM-thin, RBD, CephFS, and qcow2 on file-level storage.

3. Do you have a storage team and a SAN, or do you have servers with disks? This is an organisational question and it decides more designs than the technical ones. A SAN estate leads to LVM over FC/iSCSI. A commodity-server estate leads to Ceph or ZFS. Fighting the organisation’s shape produces a correct design nobody can operate.

4. What happens when it is full, and who finds out? Every thin backend needs this answered before it is deployed. Which alert, at what threshold, to whom.

The bundles, as they actually appear

Backend	The bundle you get	Where it fits
Ceph / RBD	shared, snapshots, thin, clones, scales out	Three or more nodes, commodity hardware, want everything and can afford the operational learning
ZFS local + `pvesr`	snapshots, thin, clones, checksums, compression; not shared — replication with an RPO	Two or three nodes, no SAN, an RPO of minutes is acceptable
LVM over SAN	shared, thin on the array, snapshots via volume chains since PVE 9	You already have a SAN and a storage team
LVM-thin (local)	snapshots, thin, clones; not shared	Single node, or local fast tier alongside something shared
NFS / CIFS	shared, easy, snapshots only with qcow2	ISOs, templates, backups; guest disks where simplicity beats performance
Directory	whatever the filesystem gives you; snapshots only with qcow2	Local scratch, ISO staging, small deployments
CephFS	shared file storage: ISOs, templates, snippets, backups	The file-level companion to an RBD cluster you already run
PBS	backup content only, deduplicated and incremental	Always. It is not an alternative to the above, it is the other half

Read-only / Safewhat capabilities does this cluster actually have today— Read-only. The first call lists configured storages with their type and shared flag; the second shows capacity and activation. Read them together — a shared storage that is inactive on half the nodes is not providing what its flag claims.

pvesh get /storage --output-format json \
| jq -r '.[] | [.storage, .type, (.shared // 0), (.content // "-"), (.nodes // "all")] | @tsv'

pvesm status

Read-only / Safe

$ pvesh get /storage --output-format json | jq -r '.[] | [.storage, .type, (.shared // 0), (.content // "-"), (.nodes // "all")] | @tsv'

local	dir	0	iso,vztmpl,backup	all
local-zfs	zfspool	0	images,rootdir	all
shared-rbd	rbd	1	images,rootdir	all
cephfs-shared	cephfs	1	iso,vztmpl,snippets,backup	all
pbs-primary	pbs	1	backup	all

Under the hood

Why does snapshot capability determine so much?

Because a snapshot and a thin allocation are the same mechanism seen from two angles. Both require the storage to maintain a mapping from logical addresses to physical blocks, rather than a fixed one-to-one layout, so that a block can be allocated at first write and referenced from more than one point in time.

Once a backend has that mapping:

Thin provisioning falls out: allocate nothing until written.
Snapshots fall out: keep the old mapping, write new blocks elsewhere.
Linked clones fall out: a new mapping that starts as a copy of the base image’s and diverges on write.

A backend without that indirection — a raw file on a plain filesystem, a classic LVM logical volume before PVE 9 — has a fixed layout and can offer none of the three. That is why the matrix’s Snapshots column predicts the others, and why the documentation can state in one line that everything with snapshots also does thin provisioning.

It is also why PVE 9’s LVM change is structural rather than cosmetic. “Snapshots as a volume chain” adds exactly that indirection: snapshot data lives in separate volumes which are layered, so a read walks the chain. That has a cost which is worth knowing — a long chain means more layers to traverse, so chains want pruning rather than accumulating — but it converts LVM from a backend with no rollback into one with rollback, and it is the reason to re-check any advice about LVM written before PVE 9.

The 4 TB pool with 40 TB of guests

Thin provisioning worked so well that nobody thought about it. Eighty guests, each provisioned with 500 GB because that was the template, on a 4 TB LVM-thin pool. Actual usage sat around 55% for a year.

Then a logging change inside a dozen guests started writing steadily. The pool crossed 100% on a Sunday.

What happens next is worse than a failed provisioning request. A thin pool with no free extents cannot satisfy writes from guests that already exist, so guests begin failing writes mid-operation. Filesystems inside them remount read-only or corrupt. LVM marks the pool as needing a check. And the space is not recoverable by deleting a guest, because deleting a guest requires writes.

The first instinct — delete snapshots — is right but often insufficient, and on a pool this far gone the safe move is to add physical extents to the volume group and extend the pool, which is a hardware conversation on a Sunday.

The controls that would have prevented it are unglamorous: an alarm on the thin pool’s data percentage at 75% and again at 85%, a policy that the sum of provisioned sizes is reviewed against physical capacity every quarter, and templates that provision what a guest needs rather than a round number.

What happens if you pick the wrong backend and only find out in year two?

You migrate, and it is more survivable than the decision felt at the time.

Guest disks can be moved between storages while the guest runs, one disk at a time, with format conversion handled for you. The next part covers the mechanics (qm move-disk and its container equivalent), and the honest summary is: moving 100 guests between backends is a scheduling problem, not an engineering one.

What does not migrate easily:

Snapshots. Moving a disk to a backend with a different snapshot mechanism does not carry a snapshot tree across. Decide in advance which history you keep and where.
The operational model. Moving from ZFS-plus-replication to Ceph does not just change storage; it changes what an outage looks like, what maintenance requires, and what the team needs to know. Budget for the learning, not just the copy.
Assumptions baked into runbooks. Every procedure that says “take a snapshot first” needs re-checking against the new backend’s snapshot cost.

The practical advice: choose for question 1 and question 2 — shared access and rollback — because those are the ones that are painful to change later. Density and performance can be tuned; the shape of migration and HA cannot.

And run the four questions again whenever the estate’s shape changes: adding a third node, acquiring a SAN, or upgrading to a release that moves a line in the matrix, as PVE 9 did for LVM.

Key takeaways

The documented matrix is the starting point, and three footnotes carry the meaning: LVM is shared when the LUN underneath it is; file-level snapshots require qcow2 and internal snapshot operations block a running VM; and PVE 9 added snapshots as volume chains for LVM.
Every backend with snapshots also supports thin provisioning, and linked clones follow the same property. Snapshot capability predicts the bundle.
Shared storage buys live migration without copying and HA that can recover anywhere. Replication buys an approximation of both, with an RPO equal to the schedule — write the number down.
A snapshot is a rollback point, not a backup: same storage, and often the same lifetime as the guest.
Thin provisioning converts capacity from an allocation into a thing you monitor. A full thin pool fails writes for existing guests, which is worse than refusing new ones.
Choose with four questions: does it need shared access and HA; does it need rollback; what shape is the organisation and its hardware; and what happens when it fills, monitored by whom.
Get shared access and rollback right first. Both are hard to change later; density and performance are not.

Knowledge check

Why this matters

Storage decisions have the longest half-life of anything in a virtualisation platform. A network can be re-cabled in a weekend, a hypervisor version is a rolling upgrade, and a monitoring stack can be replaced by someone with a free afternoon. Storage is measured in years, because moving it means moving every byte.

That would be manageable if the decision were mostly about performance, which is measurable and tunable. It is not. The decision is mostly about capabilities, and capabilities determine what routine operations feel like: whether migration is seconds or hours, whether an upgrade is reversible, whether HA can use the whole cluster or half of it, and whether “the storage is full” is a Tuesday ticket or a Sunday incident.

Read the matrix before you read the benchmarks. A slightly slower backend with the right bundle of capabilities will make the team faster than a fast one that cannot roll back.