LVM and LVM-thin as Proxmox storage · Proxmox VE

Two storage plugins share a name and almost nothing else.

lvmthin is what a default Proxmox installation gives you: local, thin, snapshot-capable, and unable to be shared. lvm is the plugin you point at a SAN LUN: shared across the cluster, thick, and — since PVE 9 — snapshot-capable in a completely different way.

Confusing them produces designs that cannot do what the design document says they do, which is why this lesson keeps them apart from the first paragraph.

The two plugins side by side

	`lvmthin`	`lvm`
Typical use	Local storage on each node	A shared iSCSI/FC/SAS LUN
Shared	no	possible — when the underlying LUN is shared
Thin provisioning	yes, in the pool	no, at the LVM layer — the array may do it
Image formats	`raw`	`raw`, and `qcow2` for volume-chain snapshots
Snapshots	yes, native thin snapshots	yes, as volume chains, since PVE 9
Linked clones	yes	no
Content types	`images`, `rootdir`	`images`, `rootdir`
Live migration without disk copy	no	yes, when shared

Their configuration entries look almost identical, which is part of the problem:

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

lvm: san-lun-a
        vgname san-vg-a
        content rootdir,images
        shared 1
        base iscsi-portal:0.0.0.scsi-360000000000000000000000000000001

The shared 1 line is the whole difference in behaviour, and the previous part established what it means: an assertion that every node sees the same data, which PVE acts on without verifying.

LVM over a shared LUN, and the lock

The arrangement is straightforward. A SAN presents one large LUN to every node in the cluster. Each node sees the same block device. LVM on top of it splits the LUN into logical volumes, one or more per guest, and PVE names them vm-<VMID>-<NAME>.

The obvious question is what stops two nodes writing LVM metadata at the same time and corrupting the volume group. The documentation answers it directly: the backend implements proper cluster-wide locking if the storage is marked as shared in the configuration.

No cLVM, no lock manager — pmxcfs does it

Historically, shared LVM on Linux required a clustered LVM daemon and a distributed lock manager, which were their own operational burden.

Proxmox VE does not use one. It already has a quorate, replicated, cluster-wide filesystem — pmxcfs — and it takes its locks there. Metadata-changing operations on a shared LVM storage (creating a volume, removing one, resizing one) acquire a cluster lock first, so they serialise across nodes.

Two consequences worth holding onto:

The lock depends on quorum. A node without quorum has a read-only /etc/pve and therefore cannot take the lock, so it cannot allocate or remove volumes on the shared LUN. That is a feature: it is the same mechanism that stops a partitioned node making conflicting changes anywhere else.

The lock only covers PVE. It serialises Proxmox VE’s own operations against each other. It knows nothing about anything else attached to that LUN.

The LUN belongs to Proxmox VE alone

The single rule that makes shared LVM safe is that nothing outside this Proxmox VE cluster may write to that LUN. Not another cluster, not a physical server that was given the same LUN “read-only for a backup”, not an administrator running vgchange from a rescue system.

LVM metadata sits at the head of the volume group and is rewritten on every change. Two writers with no shared lock produce a metadata state that reflects one of them, and the volumes described by the other simply cease to be described. What you get is not a corrupted file; it is logical volumes that no longer exist as far as LVM is concerned, and the guest data inside them unreachable without metadata archaeology.

The controls are at the array, not in PVE: LUN masking that presents the LUN to exactly the cluster’s initiators, and a change process for the SAN that treats adding an initiator to that mask as a production change. PVE cannot enforce this and does not try.

Configuration changeadd a shared LVM storage over an existing SAN volume group— Changes cluster configuration. Assumes the volume group already exists on a LUN presented to every node. Marking it shared is what enables cluster-wide locking and live migration without disk copy — and it must be true before you set it.

pvesm add lvm san-lun-a \
--vgname san-vg-a \
--content images,rootdir \
--shared 1

pvesm status --storage san-lun-a

Read-only / Safeconfirm every node really does see the volume group— Read-only. Run on each node. If a node does not list the volume group, the shared flag is a promise that node cannot keep, and any guest migrated there will fail to start — or worse, if the flag is wrong in the other direction, succeed against the wrong data.

vgs -o vg_name,vg_size,vg_free,pv_count

pvs -o pv_name,vg_name,pv_size

pvesm status --storage san-lun-a

LVM-thin: the local default

A default Proxmox VE installation creates a pve volume group with a thin pool called data, and registers it as the local-lvm storage. Building one by hand is two commands:

Destructivecreate a thin pool in an existing volume group— DESTRUCTIVE if the volume group or its space is in use. lvcreate consumes 100 GiB of the volume group; lvconvert changes that logical volume into a thin pool, after which its contents are managed by the thin target and it cannot be used as an ordinary LV. Confirm free space with vgs first.

VG=pve
POOL=data

vgs "$VG"

lvcreate -L 100G -n "$POOL" "$VG"
lvconvert --type thin-pool "$VG/$POOL"

lvs -o lv_name,lv_size,data_percent,metadata_percent "$VG"

Configuration changeregister the thin pool as a PVE storage— Changes cluster configuration. Note the absence of --shared: a thin pool is local to the node whose disks back it, and marking it shared is the failure described in the storage model lesson.

pvesm add lvmthin local-lvm \
--vgname pve \
--thinpool data \
--content images,rootdir

pvesm status --storage local-lvm

The thin pool has two ways to fill

This is the part that costs people a weekend, and it is not the one they monitor.

A thin pool has two allocated areas: the data area, which holds guest blocks, and a separate metadata area, which holds the mapping from logical blocks to physical ones. Both can fill. Only the first is what pvesm status and most dashboards show.

Read-only / Safethe only thin-pool health command that matters— Read-only. data_percent is what everyone watches; metadata_percent is what takes the pool down while data sits at 40%. Snapshots and many small volumes consume metadata far faster than they consume data.

lvs -o lv_name,lv_size,data_percent,metadata_percent,lv_metadata_size,snap_percent

lvs -a -o lv_name,pool_lv,lv_size,data_percent | head -30

Read-only / Safe

$ lvs -o lv_name,lv_size,data_percent,metadata_percent,lv_metadata_size

  LV            LSize    Data%  Meta%  LMSize
data          <1.72t   39.96  94.12  120.00m
root          96.00g
swap           8.00g
vm-141-disk-0 500.00g  22.40
vm-142-disk-0 500.00g  18.05

Metadata full at 40% data usage

A cluster with an aggressive snapshot policy — hourly snapshots retained for a week, on thirty guests — ran a 1.7 TiB thin pool at around 40% data usage for months. The capacity dashboard was green throughout.

Metadata usage climbed instead, because every snapshot multiplies the mapping the pool must track. At 100% metadata, the thin pool stopped accepting writes. Guests began failing writes exactly as if the data area were full, and the dashboard still showed 40%.

The first diagnosis is always “the pool is full”, and then someone looks at the data percentage, sees 40%, and concludes there is a bug in the storage layer. There is not. There are two counters and one of them was never displayed.

The recovery is genuinely awkward. Extending metadata with lvextend --poolmetadatasize needs free extents in the volume group, and repairing damaged metadata with lvconvert --repair needs the pool inactive, which means every guest on it stopped. On a pool backing thirty guests, that is a full outage.

The prevention is one extra field in one alarm:

Alarm on metadata_percent as well as data_percent, at 75% and 85%.
Size the metadata area generously at creation — the default is often too small for a snapshot-heavy workload.
Treat a snapshot retention policy as a capacity decision, because it is one.

Under the hood

Why does PVE 9’s LVM snapshot change exist, and why is it not the same mechanism as LVM-thin’s?

Classic LVM snapshots have a specific defect the documentation names: they cause I/O interference across the entire volume group during snapshot operations, producing substantial performance degradation, and LVM therefore cannot support linked clones. That is why shared LVM went so long without usable snapshots — the mechanism existed and was not safe to offer.

PVE 9 solves it from a different direction. Rather than using LVM’s snapshot feature, the storage plugin manages snapshots itself: it allocates a separate logical volume per snapshot and layers them, using qcow2’s backing-chain capability inside those volumes. The documentation calls this “snapshots as a volume chain”, enabled by the snapshot-as-volume-chain property.

Three consequences follow:

The chain is walked on read. A read that misses in the top layer falls through to the one below. More snapshots means a longer chain, so chains want pruning rather than accumulating — the opposite of the “snapshots are free, keep them” instinct that ZFS encourages.

The image format changes. LVM with volume-chain snapshots stores qcow2 inside logical volumes rather than raw, which is why the LVM plugin’s format list gained qcow2.

It is not thin provisioning. The volumes are still thick from LVM’s point of view; only the snapshot layers are incremental. Thin behaviour on a SAN LUN, if you have it, comes from the array underneath, and PVE has no visibility into the array’s real free space. That is a genuinely dangerous blind spot on a thin- provisioned array: pvesm status reports the LUN as having space because LVM says so, while the array behind it is full.

LVM-thin, by contrast, uses the kernel’s dm-thin target directly. Allocation and snapshots are both properties of the pool, which is why it gets thin provisioning, cheap snapshots and linked clones — and why it cannot be shared, since dm-thin’s metadata has exactly one writer.

`saferemove`, and what it costs

Both plugins accept saferemove, documented as zeroing out data when removing logical volumes so that it cannot be accessed by other LVs created later.

That is a real concern on multi-tenant or regulated estates: without it, a newly created volume can contain the previous volume’s blocks, and a guest that reads its own unwritten disk sees somebody else’s data.

The cost is time and I/O. Removing a 2 TB volume means writing 2 TB of zeroes, throttled by saferemove_throughput (up to 10 MiB/s by default) and stepped by saferemove-stepsize (up to 32 MiB by default). At the default throughput, that 2 TB removal takes days.

The decision is therefore not “should data be erased” but “where”. On a multi-tenant estate, enable it and size the throughput deliberately. Elsewhere, consider whether encryption at rest, or the array’s own crypto-erase, answers the same question at a fraction of the cost.

Choosing between these and the alternatives

Situation	Answer	Why
Single node, or local fast tier	LVM-thin	Thin, snapshots, linked clones, and the default already exists
Existing SAN, storage team, 3+ nodes	LVM over the LUN, `shared 1`	Shared access without running a distributed storage system; snapshots since PVE 9
Commodity servers, 3+ nodes, want shared	Ceph/RBD	Shared, thin, snapshots, clones, scales out; more to learn
2–3 nodes, no SAN, RPO of minutes acceptable	ZFS + `pvesr`	Checksums, compression, cheap snapshots, replication instead of sharing
Anything where you want `raw` performance and no rollback	LVM (thick)	Simplest possible layout, and now with a rollback option if you want one

Two cautions specific to this pair. Never mark an LVM-thin storage shared — dm-thin has one writer and the result is corruption rather than an error. And do not assume free space on a thin-provisioned array: LVM reports the LUN’s size, not the array’s remaining capacity, so the array needs its own monitoring and its own alarm.

What happens if a node loses quorum while guests run on shared LVM?

Less than people fear, and the shape is worth knowing.

Running guests keep running. Their logical volumes are already active and their I/O goes straight to the LUN. Losing quorum does not close a block device.

Metadata operations stop. Creating a disk, removing one, resizing one, taking a volume-chain snapshot — all need the cluster lock, and the lock needs a writable /etc/pve, which a node without quorum does not have. Attempts fail with a lock or a permission error rather than proceeding unsafely. That is correct behaviour: it is precisely the case the lock exists for.

Starting a guest may fail, because starting one usually needs to write to cluster configuration.

HA does what HA does. If the node is running HA services and stays without quorum past the watchdog interval, it fences. Part XII owns that behaviour, and nothing about LVM changes it.

What to do: treat it as a quorum incident, not a storage incident. Restore quorum — fix the network, bring the missing node back, check corosync links — and the storage resumes accepting metadata operations by itself. Do not reach for pvecm expected 1 to “unstick the storage”: on a shared LUN that is exactly how two nodes end up writing volume group metadata with no lock between them, which is the one failure this whole design is built to prevent.

Key takeaways

lvmthin and lvm are different products with similar names. Thin is local, thin-provisioned, snapshot- and linked-clone-capable. Plain LVM over a shared LUN is shared and thick.
Shared LVM is safe because PVE implements cluster-wide locking when the storage is marked shared, using pmxcfs rather than cLVM or a separate lock manager. The lock therefore depends on quorum.
That lock protects PVE’s operations from each other and nothing else. The LUN must be presented to this cluster alone; a second writer destroys the volume group metadata.
A thin pool has a data area and a separate metadata area, and metadata fills first under snapshot-heavy workloads. Alarm on metadata_percent, not only data_percent. Repairing metadata needs the pool inactive.
PVE 9 gave LVM snapshots as volume chains — a separate volume per snapshot, layered with qcow2 — rather than classic LVM snapshots, which degrade the whole volume group. Chains are walked on read, so prune them.
LVM over a thin-provisioned array reports the LUN’s size, not the array’s free space. The array needs its own capacity alarm.
saferemove zeroes removed volumes and is throttled to 10 MiB/s by default, so a large removal takes a very long time. Decide it deliberately.
Never mark an LVM-thin storage shared.

Knowledge check

Why this matters

Shared LVM over a SAN is unfashionable and it is quietly one of the best answers in the product for a specific kind of organisation: one that already owns a storage array, already has people who understand it, and does not want to take on operating a distributed storage system as well.

It gives you the two capabilities that matter most — shared access for live migration and HA — with a very small amount of new machinery, and since PVE 9 it gives you rollback too. The cost is a rule you must actually enforce at the array: that LUN belongs to this cluster and nothing else.

LVM-thin is the opposite trade and just as valid. Everything local, everything cheap, snapshots that cost nothing, and a hard ceiling at one node.

The mistake worth avoiding is not choosing either of them. It is reading a document that says “LVM storage” and assuming it means whichever of the two you have used before.