ZFS as Proxmox storage (local-zfs, replication) · Proxmox VE

Why this matters in production

ZFS is the default local storage for most Proxmox deployments. Using it correctly means predictable performance and the ability to replicate VMs to peer nodes for HA-style recovery without shared storage.

The two ZFS storage modes

Proxmox exposes a ZFS pool in two ways:

Mode	What it is	Used for
`zfspool`	The whole pool is a Proxmox storage; VMs use `zvol` (block)	Direct VM disk creation
ZFS dataset + Directory	A dataset under a pool, exposed as directory storage	Templates, ISOs, file-based workloads

The default local-zfs storage on a Proxmox install is zfspool mode — every VM disk is a zvol.

Configuring local-zfs

The installer usually creates local-zfs automatically. Verify:

cat /etc/pve/storage.cfg

zpool status rpool && zfs list -t volume

If you want a separate pool for VM storage (recommended for larger deployments):

pvesm add zfspool vmdata --pool tank --content images,rootdir

Zvols

A zvol is a ZFS block device. Proxmox creates one per VM disk.

zfs list -t volume -o name,volsize,refreservation,compression

Properties to know:

volsize — virtual size of the block device.
refreservation — actual space reserved on the pool (for thick provisioning).
compression — default lz4 (very fast, modest ratio).
volblocksize — block size. Defaults to 16 KiB on OpenZFS 2.2 and later, which includes the ZFS 2.4 that PVE 9.2 ships; it was 8 KiB before that. Fixed once the volume has been written.
primarycache / secondarycache — caching settings; defaults are usually correct.
thin provisioning — leave refreservation unset; the zvol grows on demand.

The Proxmox storage option that controls this is sparse, documented as “Use ZFS thin-provisioning. A sparse volume is a volume whose reservation is not equal to the volume size.” The blocksize option sets the volblocksize of new zvols on this storage; the block-size tuning lesson covers how to choose it.

Read-only / Safehow over-provisioned is the pool actually?— Read-only. Sum the volsize column and compare it against the pool size. On a thin-provisioned storage that sum will exceed the pool, which is the point - the question is by how much, and whether the growth rate closes the gap before you notice.

POOL=tank

zpool list -o name,size,alloc,free,capacity,fragmentation "$POOL"

zfs list -t volume -o name,volsize,used,refreservation -r "$POOL"

# total provisioned vs total available
zfs list -Hp -t volume -o volsize -r "$POOL" |
awk '{s+=$1} END {printf "provisioned: %.1f TiB\n", s/1024/1024/1024/1024}'

What happens when a thin-provisioned ZFS pool reaches 100%?

Every VM on it stops, at the same instant, in the way that is hardest to recover from.

A full ZFS pool cannot accept writes. Guests do not receive an I/O error they can handle — the write blocks, the guest’s I/O stack waits, and processes enter uninterruptible sleep. A database mid-transaction stays mid-transaction. qm status reports running for every one of them.

The recovery is harder than it should be for a specific reason: deleting files inside a guest does not free pool space. The zvol still occupies the blocks; the guest merely marked them unused. And zfs destroy on a snapshot needs to write metadata, which a completely full pool may not be able to do.

The order that actually works:

Destroy a snapshot. This is usually possible even on a full pool and is the fastest space to recover.

zfs list -t snapshot -o name,used -s used -r tank | tail -20
zfs destroy tank/vm-101-disk-0@old-snapshot

Remove a reservation. If any zvol is thick-provisioned, clearing refreservation releases the reserved-but-unwritten space instantly and without writing anything.

zfs set refreservation=none tank/vm-140-disk-0

Add a vdev. A pool can always grow. If there is a spare disk, adding a mirror vdev is a real fix rather than a stopgap — and unlike almost everything else in ZFS, it takes effect immediately.

Then, once the pool is writable again, make the guests give the space back: enable discard on the VM disks and run fstrim inside each guest, so that files deleted in the guest actually release blocks in the pool. Without that, the pool refills at the same rate.

The prevention is a threshold alert, and the number matters: ZFS performance degrades noticeably above 80% because the allocator starts working harder to find contiguous space. Alert at 80%, act at 80%, and treat 90% as an incident rather than as a warning.

ZFS replication

ZFS replication copies a VM’s zvol to a peer node at regular intervals. It uses zfs send / zfs receive over SSH.

flowchart LR
  A[pve-01 source zvol] -->|zfs send| B[SSH]
  B --> C[pve-02 destination zvol]

Replication schedule:

pvesr create-local-job 100 pve-02 --schedule '*/15' --rate 50

pvesr status

After replication, the VM can be recovered on the peer node even if the source is gone — this is the foundation for “manual HA” without shared storage.

Snapshot + send/receive

A more flexible pattern:

zfs snapshot tank/vm-100-disk-0@before-upgrade

zfs send tank/vm-100-disk-0@before-upgrade | ssh pve-02 'zfs receive tank/vm-100-disk-0@before-upgrade'

Production considerations

Common mistakes

Putting VMs on the root pool.
Setting refreservation on every zvol (thin pool never over-commits but capacity is hidden).
Treating ZFS replication as HA.
Copying volblocksize=8K from advice written before OpenZFS 2.2. The default is 16 KiB now, and 8 KiB costs capacity on RAIDZ.

Key takeaways

zfspool exposes a ZFS pool as block storage for VMs.
Zvols are ZFS block devices; use them for VM disks.
ZFS replication copies a VM’s zvol to a peer; recovery is manual.
Compression is free; use it.