The PVE storage model: storage.cfg, types and content · Proxmox VE

Every storage lesson that follows — ZFS, Ceph, NFS, iSCSI — is about a technology. This one is about the layer Proxmox VE puts in front of all of them, and it is the layer you actually operate.

When a backup fails with “storage does not support content type”, when a migration refuses to start, when a disk lands somewhere you did not expect, the answer is almost never in the storage technology. It is in the eight lines of storage.cfg that describe it.

The file

Storage configuration lives in /etc/pve/storage.cfg and is therefore cluster configuration: written once, present on every node, and covered by the same quorum rules as everything else in /etc/pve. Its format is deliberately plain:

<type>: <STORAGE_ID>
        <property> <value>
        <property> <value>

A realistic file:

dir: local
        path /var/lib/vz
        content iso,vztmpl,backup
        prune-backups keep-last=3

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

rbd: shared-rbd
        content images,rootdir
        krbd 0
        pool vm-pool

pbs: pbs-primary
        datastore main
        server 192.0.2.20
        content backup
        fingerprint aa:bb:cc:dd:ee:ff:00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd:ee:ff:00:11:22:33:44:55:66:77:88:99
        prune-backups keep-daily=14,keep-weekly=8,keep-monthly=6

nfs: iso-share
        export /export/isos
        server 192.0.2.30
        content iso,vztmpl
        nodes pve-a1,pve-a2,pve-a3
        options vers=4.2

Five entries, five different backends, one grammar. That uniformity is the point of the abstraction: qm, pct, vzdump, the GUI and the API all speak to a storage through the same interface regardless of what is underneath.

Type: what is underneath

The type is the plugin. It determines the storage’s capabilities, and the documentation divides the plugins into two classes that predict most of the behaviour you will meet.

File-level — ZFS (as a filesystem), Directory, BTRFS, NFS, CIFS, CephFS. POSIX filesystems, so they can hold any content type, including ISO images, container templates, backups and snippets. Disk images on them are files: qcow2, raw or vmdk.

Block-level — LVM, LVM-thin, iSCSI, Ceph/RBD, ZFS zvols. They hold raw volumes and nothing else, so a block storage cannot store an ISO or a backup file.

Type	Plugin	Level	Shared	Snapshots
ZFS	`zfspool`	both	no	yes
Directory	`dir`	file	no	qcow2 only
NFS	`nfs`	file	yes	qcow2 only
CIFS	`cifs`	file	yes	qcow2 only
LVM	`lvm`	block	possible	volume chains (PVE 9+)
LVM-thin	`lvmthin`	block	no	yes
Ceph/RBD	`rbd`	block	yes	yes

pvesm add accepts: btrfs, cephfs, cifs, dir, esxi, iscsi, iscsidirect, lvm, lvmthin, nfs, pbs, rbd, zfs, zfspool.

Content: what it is allowed to hold

The content property is a comma-separated list, and it is enforced. The seven content types:

Content type	Holds
`images`	QEMU/KVM virtual machine disks
`rootdir`	Container root filesystems
`vztmpl`	Container templates
`iso`	ISO images
`backup`	`vzdump` backup files
`snippets`	Guest hook scripts and Cloud-Init snippets
`import`	OVAs and disk images staged for import

The "does not support content type" error is a configuration message

This is the error that sends people looking for a storage fault, and it almost never is one. It means one of two things:

The backend genuinely cannot hold it. An LVM-thin storage will not accept iso or backup, because it holds raw block volumes and an ISO is a file. No configuration change fixes this; you need a file-level storage.
The backend could, but the content list does not include it. This is the common case, and it is one pvesm set away.

One trap in the fix: pvesm set <id> --content <list> replaces the list, it does not append to it. Setting --content backup on a storage that was holding images,rootdir,backup removes the first two — and existing guest disks on it become unusable while the configuration says the storage holds only backups. Read the current value first and write the whole intended list.

Read-only / Safewhat each storage is, holds and has left— Read-only. pvesm status is the single most useful storage command: it shows type, status, total, used and available for every configured storage, with the active/enabled state that explains a storage the GUI is showing in grey.

pvesm status

pvesm status --content images

pvesm list local-lvm

Read-only / Safe

$ pvesm status

Name              Type     Status           Total            Used       Available        %
iso-share          nfs     active       524288000        31457280       492830720    6.00%
local              dir     active       100663296         8388608        92274688    8.33%
local-lvm      lvmthin     active      1932735283       772294113      1160441170   39.96%
pbs-primary        pbs     active     10737418240      4294967296      6442450944   40.00%
shared-rbd         rbd     active     32212254720      9663676416     22548578304   30.00%

Node restriction, and the `shared` flag

Two properties that look similar and mean very different things.

nodes restricts which cluster nodes may access this storage. It is an access-control statement about topology: an NFS export reachable only from the management VLAN, or a local storage that exists on three nodes and not the fourth. Guests using a node-restricted storage cannot be migrated to a node outside the list.

shared tells PVE that every node sees the same data on this storage. It is the flag that makes live migration cheap — with shared storage, migrating a running VM moves memory and leaves the disks in place.

The shared flag is an assertion, not an implementation

Setting shared 1 does not make a storage shared. It tells Proxmox VE that the storage already is, and PVE then acts on that promise without verifying it.

The classic failure: a directory storage on each node, backed by local disks, with the same path. It looks shared — the path exists everywhere, pvesm status is green on every node — so somebody sets shared 1 to enable live migration.

Migration then succeeds instantly and does not copy anything, because PVE believes the disk is already at the destination. The VM starts on the new node against a different file at the same path: an older copy, an empty file, or nothing at all. The original data is still on the source node, untouched and now diverged from what the guest is writing.

Nothing errors. The task log is green. The failure surfaces later as data loss that is very difficult to reconstruct, because two versions of the disk exist and neither is complete.

The rule is simple and absolute: set shared 1 only when the storage layer itself provides shared access — NFS, CIFS, CephFS, RBD, or LVM on a genuinely shared SAN LUN. Never for dir, never for local ZFS or LVM-thin.

The volume identifier

Every guest disk is named by a volume ID, storage:volume, and this is what appears in a guest configuration:

scsi0: shared-rbd:vm-141-disk-0,size=64G,discard=on,ssd=1
scsi1: local:141/vm-141-disk-1.qcow2,size=500G
rootfs: local-lvm:subvol-205-disk-0,size=8G

Read the second one carefully: on a file-level storage the volume part includes the VMID directory and the file extension, because it is a path relative to the storage. On a block-level storage it is just a volume name, because there is no directory structure to reference.

The storage ID is the first field, which has a consequence worth internalising: a guest’s disk is bound to a storage ID, not to a device. Rename a storage and every guest referencing it breaks. Point a storage ID at different hardware and every guest referencing it silently follows.

Read-only / Saferesolve a volume ID to a path or device— Read-only. pvesm path turns a volume ID into the filesystem path or block device the node will actually open. This is the fastest way to answer 'where is this disk really' and to prove that two nodes resolve the same volume differently.

pvesm path local:141/vm-141-disk-1.qcow2

pvesm path local-lvm:vm-141-disk-0

pvesm list shared-rbd --vmid 141

Read-only / Safe

$ pvesm list shared-rbd --vmid 141

Volid                              Format  Type       Size VMID
shared-rbd:vm-141-disk-0           raw     images 68719476736 141
shared-rbd:vm-141-disk-1           raw     images 536870912000 141

Under the hood

What does PVE actually do when a guest starts?

The guest configuration holds a volume ID, not a path. At start time, the storage layer resolves it in three steps:

Look up the storage ID in storage.cfg to find the plugin and its properties.
Ask the plugin to activate the volume. This is the step that has side effects and varies most between backends: mounting an NFS export, logging in to an iSCSI target, activating an LVM logical volume with lvchange -ay, mapping an RBD image, or doing nothing at all for a plain directory.
Return a path or device, which is what ends up on the QEMU command line.

You can watch the last step directly. qm showcmd <vmid> --pretty prints the QEMU invocation PVE builds, with the resolved paths in it — which settles arguments about whether a configuration option reached the hypervisor far faster than reading configuration files does.

Two things follow from the activation step. First, a storage being “up” is not a boolean: it can be configured, enabled, and still fail to activate on one node because a mount is stale or an iSCSI session is down. That is why pvesm status distinguishes active from inactive and why it should be read on the node in question, not on whichever node your SSH session landed on.

Second, this is the mechanism the shared flag bypasses reasoning about. Activation is per node, so nothing in the process compares the data one node resolves against the data another resolves — it cannot, because for a genuinely shared backend that comparison is unnecessary and for everything else PVE was told not to worry.

Adding a storage properly

Configuration changeadd an NFS storage restricted to three nodes— Changes cluster configuration. pvesm add validates the type and property names and writes through pmxcfs, so the entry appears on every node atomically. The storage is created enabled; add --disable 1 to stage it.

pvesm add nfs backup-nfs \
--server 192.0.2.30 \
--export /export/backups \
--content backup \
--nodes pve-a1,pve-a2,pve-a3 \
--options vers=4.2 \
--prune-backups keep-daily=14,keep-weekly=8

pvesm status --storage backup-nfs

Configuration changechange a content list safely— Changes cluster configuration. --content REPLACES the existing list. Read the current value first, then write the full intended list. Removing a content type that existing volumes rely on makes those volumes unusable without deleting them.

pvesm status --storage local | cat

grep -A6 '^dir: local$' /etc/pve/storage.cfg

pvesm set local --content iso,vztmpl,backup,snippets

Properties worth setting deliberately

Property	Why it earns a decision
`content`	The whole point of the entry. Set the full list; remember it is replaced, not merged
`nodes`	Omitting it means “every node”, which for a local-only backend is a promise you cannot keep
`shared`	Only where the storage layer genuinely shares. See the callout — this one is not recoverable
`disable`	Stages an entry, or takes a failing storage out of scheduling without deleting its definition
`format`	`raw`, `qcow2` or `vmdk` on file-level backends. `qcow2` buys snapshots on a directory store; `raw` buys performance
`preallocation`	`off`, `metadata`, `falloc`, `full`. The difference between a thin file and a guaranteed one
`prune-backups`	Retention. Absent means nothing is ever pruned, which is how backup storages fill

And one habit: after any storage change, run pvesm status on more than one node. A storage entry is cluster-wide; its activation is not.

What happens if you remove a storage that still has volumes on it?

pvesm remove <id> deletes the configuration entry. It does not touch the data.

What breaks immediately. Any guest whose configuration references <id>:volume can no longer resolve it. A running guest keeps running — its disks are already open — but it will not start again, cannot be migrated, and cannot be backed up. The error names the storage, so it reads like a storage outage rather than a deliberate removal, especially to whoever is on call.

What is left behind. Every volume, exactly where it was: logical volumes in the volume group, RBD images in the pool, qcow2 files in the directory. They consume space and nothing in PVE lists them any more, because listing volumes is a function of a configured storage.

Recovery is usually easy, if you are quick. Re-add the storage with the same ID and the same properties and everything resolves again. This is why keeping storage.cfg in version control, or at least in the baseline document, is worth the five minutes: the fix is to restore an entry you can no longer read.

The genuinely destructive version is different: removing the volumes. pvesm free <volid> deletes a volume, and deleting a guest deletes its disks. Those are the operations to be careful with. Removing a storage entry is recoverable; freeing a volume is not.

Before removing a storage, prove nothing uses it:

pvesm list <id> shows every volume on it, and grepping the guest configuration tree for the storage ID shows every reference. If both are empty, the entry is genuinely unused.

Key takeaways

/etc/pve/storage.cfg is cluster configuration: <type>: <ID> followed by indented properties, identical grammar for every backend.
The type is the plugin and it determines capability. File-level backends (dir, nfs, cifs, cephfs, btrfs, zfspool as a filesystem) hold any content type; block-level backends (lvm, lvmthin, iscsi, rbd) hold raw volumes only.
content is enforced. “Does not support content type” means the backend cannot hold it, or the list does not include it. pvesm set --content replaces the list.
nodes restricts where a storage may be used and constrains migration. shared asserts that every node sees the same data.
shared 1 on a storage that is not genuinely shared causes silent data divergence during migration. Set it only where the backend shares: NFS, CIFS, CephFS, RBD, or LVM on a shared LUN.
A volume ID is storage:volume, and a guest is bound to the storage ID. File-level volumes carry the VMID directory and an extension; block-level volumes do not.
pvesm status for capacity and activation, pvesm list for volumes, pvesm path to resolve a volume to a device, pvesm alloc/free to create and destroy. Check status on more than one node.
Removing a storage entry breaks references but keeps data and is recoverable by re-adding it. Freeing a volume is not.

Knowledge check

Why this matters

The Proxmox storage abstraction is a small piece of software with an unusual amount of leverage. It is why the same qm command works whether the disk is a file on NFS or an RBD image in Ceph, why a backup job does not care what it is writing to, and why you can move a guest between backends without touching the guest.

The price of that leverage is that the abstraction can be told things that are not true. shared 1 on a storage that is not shared. A content type that promises what the backend cannot deliver. A storage ID re-pointed at different hardware. In each case PVE does exactly what it was told, correctly and efficiently, and the result is wrong.

That is why this lesson comes before the technology chapters rather than after them. Knowing ZFS deeply will not save you from a storage entry that lies, and the entry is eight lines you can read in ten seconds — if you know what the lines mean.