volblocksize, recordsize and guest alignment · Proxmox VE

Nearly every ZFS mistake is recoverable. ARC is too big, so you cap it. The scrub schedule is wrong, so you change it. Compression is off, so you turn it on and new writes get it.

volblocksize is not in that category. The property page states it plainly: “The blocksize cannot be changed once the volume has been written, so it should be set at volume creation time.” Every VM disk Proxmox creates on a zfspool storage is a zvol, and every one of them takes its block size from the storage’s blocksize setting at the moment it is created.

Which means the number in /etc/pve/storage.cfg on the day you built the cluster is baked into every disk created since, and correcting it means rewriting all of them.

The default moved, and old advice did not

OpenZFS changed the default volblocksize from 8 KiB to 16 KiB in version 2.2. PVE 9.2 ships ZFS 2.4, so a zvol created today with no explicit setting gets 16 KiB.

The upstream reasoning is worth reading in full, because it is the whole trade in two sentences: “Current default (16KB since v2.2) balances the metadata overhead, compression opportunities and decent space efficiency on majority of pool configurations due to 4KB disk physical block rounding (especially on RAIDZ and DRAID), while incurring some write amplification on guest FSes that run with smaller block sizes.”

Blog posts and old runbooks still say 8K, and copying that number is a real cost on RAIDZ

The advice “set volblocksize=8K for VM storage” was correct when the default was 8K and most pools were mirrors. It circulated widely, and it is still in a lot of internal runbooks.

On a mirror it is defensible. On RAIDZ it is expensive, and the expense is capacity rather than performance, which means nobody notices until the pool is fuller than the capacity plan said it would be. The mechanism is in the Under the hood section below; the short version is that RAIDZ rounds every allocation up, and the smaller the block, the larger that rounding is as a fraction of what you stored.

If you inherit a pool with blocksize 8k in storage.cfg and RAIDZ vdevs, that is worth measuring before you plan the next capacity purchase.

The three numbers that have to agree

volblocksize sits between two other block sizes and should not be chosen without both of them in view.

guest filesystem block size   (ext4/XFS/NTFS: 4 KiB; InnoDB pages: 16 KiB)
        |
        v
volblocksize                  (the zvol's allocation unit)
        |
        v
ashift                        (the pool's sector size: 2^12 = 4 KiB)

The hard floor comes from upstream and it is absolute: “It’s meaningless to set volblocksize less than guest FS’s block size or ashift.”

Below the guest’s block size, you have not made I/O smaller — the guest still writes 4 KiB — you have only made ZFS do more work per guest write. Below ashift, ZFS cannot address the storage that finely anyway.

Above the guest’s block size, every partial write becomes a read-modify-write: ZFS must read the whole block, apply the guest’s 4 KiB, recompute the checksum, and write the whole block back. On a size-3 mirror that is one read and two writes for a 4 KiB guest operation.

Guest workload	Guest block	Sensible `volblocksize`	Why
General-purpose Linux, ext4/XFS	4 KiB	16 KiB	The default. Good compression, acceptable amplification
Windows Server, NTFS default	4 KiB	16 KiB	Same reasoning
MySQL / InnoDB	16 KiB	16 KiB	Exact match; no read-modify-write on a page write
PostgreSQL	8 KiB	16 KiB	8 KiB is defensible on mirrors, costly on RAIDZ
File server, media, backups	4 KiB, sequential	64–128 KiB	Sequential access; compression ratio improves markedly
Small random I/O on a mirror pool	4 KiB	8 KiB	Only when the pool is mirrors and IOPS matter more than space

The right way to read that table is that 16 KiB is correct until you can name why it is not. The exceptions are real, and there are fewer of them than the tuning discussions suggest.

Setting it in Proxmox

Proxmox exposes this as the blocksize option on a zfspool storage, documented as “Set ZFS blocksize parameter”. No default is documented, which means when it is unset the pool’s own volblocksize default applies — 16 KiB on ZFS 2.2 and later.

Read-only / Safewhat is actually configured right now— Read-only. The absence of a blocksize line means the ZFS default applies, which is 16K on ZFS 2.2+ and 8K on anything older - so an upgraded host can have two eras of zvol on one pool.

grep -A8 '^zfspool:' /etc/pve/storage.cfg
zfs version

Read-only / Safewhat the existing zvols actually got— Read-only. This is the authoritative answer; storage.cfg tells you what new disks will get, this tells you what the current ones have. A mixture is normal on a pool that predates a settings change.

POOL=tank
zfs list -t volume -o name,volsize,volblocksize,used,logicalused,compressratio -r "$POOL"

Read-only / Safe— Illustrative. vm-140 predates the storage change and still has 8K. Note its USED against LUSED: 412G on disk for 287G of logical data is the RAIDZ rounding cost, not compression failing.

$ zfs list -t volume -o name,volsize,volblocksize,used,logicalused,compressratio -r tank

NAME                  VOLSIZE  VOLBLOCK   USED  LUSED  RATIO
tank/vm-101-disk-0        100G       16K  41.2G  38.9G  1.06x
tank/vm-102-disk-0        200G       16K  88.7G  84.1G  1.05x
tank/vm-140-disk-0        500G        8K   412G   287G  1.04x

Illustrative output

That last row is the whole lesson in one line. USED is what the pool gave up; LUSED is what the guest actually stored. A ratio far above compressratio means space is going somewhere other than data, and on RAIDZ with a small volblocksize that somewhere is parity and padding.

To change the setting for future disks:

Configuration changeset the blocksize on a storage— Affects only zvols created after this change. Existing disks keep the block size they were created with; nothing about them changes and nothing is at risk.

STORAGE=local-zfs
pvesm set "$STORAGE" --blocksize 16k

grep -A8 "^zfspool: $STORAGE" /etc/pve/storage.cfg

Changing it on a disk that already exists

You cannot. What you can do is create a new zvol with the right setting and copy the data, which Proxmox will do for you.

Service impact possiblemove a disk to a storage with the right blocksize— Creates a new zvol on the target storage, copies every block, then switches the VM to it. The VM may run during the move on most storage combinations, but the copy competes for pool I/O. Keep --delete 0 on the first pass so the original disk survives until you have confirmed the new one boots.

VMID=140
DISK=scsi0
TARGET=local-zfs-16k

qm move-disk "$VMID" "$DISK" "$TARGET" --delete 0

# confirm the new zvol has the block size you wanted
zfs get volblocksize "tank16k/vm-$VMID-disk-0"

The trick that makes this practical: define a second zfspool storage entry pointing at a different dataset of the same pool, with the blocksize you want. Then qm move-disk between them is a local copy at pool speed, and you can work through the estate a VM at a time rather than planning an outage.

Once the VM has booted from the new disk and you are satisfied, remove the old one — qm move-disk with --delete 0 leaves it attached as an unused disk, visible in the GUI, which is the correct place for it until you are sure.

`recordsize` is the same idea for datasets, with one important difference

Zvols have volblocksize. Datasets — a ZFS filesystem exposed to Proxmox as directory storage, holding qcow2 or raw files, ISOs, templates, backups — have recordsize, default 128 KiB.

The difference that matters operationally: “Changing the recordsize on a dataset will only take effect for new files.” Existing files keep their record size, but new ones get the new value, so a dataset is correctable by rewriting files rather than by moving storage. Upstream names the mechanism: “A cp followed by a mv on each file is sufficient.”

Where it earns attention:

A dataset holding PBS chunk data or backup files: leave it at 128 KiB. Large sequential writes, and a smaller record size only adds metadata.
A dataset holding qcow2 VM images: the qcow2 cluster size (64 KiB by default) is the alignment that matters, not the guest’s filesystem.
A dataset holding a database directly (a bind mount into an LXC container rather than a VM disk): upstream gives concrete numbers — recordsize=16K for InnoDB data files with 128 KiB left on the logs, and recordsize=32K for PostgreSQL data and WAL, noting 64 KiB and the 128 KiB default also work well.

Under the hood

Why a small volblocksize costs capacity on RAIDZ and not on a mirror.

On a mirror the arithmetic is trivial: a logical block of size B is written to each side of the mirror as B, rounded up to the sector size set by ashift. Two disks, size bytes each, 50% efficiency, and it does not matter whether B is 8 KiB or 128 KiB. This is why mirror pools are forgiving of the block-size decision.

RAIDZ does something else. Each logical block is split into sectors, parity sectors are computed per block, and the result is written across the vdev’s disks. Two rounding effects stack up.

Parity is per block, not per stripe. A RAIDZ2 vdev writes two parity sectors for every block, however small that block is. With ashift=12 (4 KiB sectors), an 8 KiB block is 2 data sectors — and it still needs 2 parity sectors. That is 16 KiB consumed for 8 KiB stored: 50% efficiency, on a vdev whose capacity table promised 67%. Widening the vdev does not help, because the parity count is fixed by the RAIDZ level, not by the disk count.

Allocations are rounded to a multiple of (parity + 1) sectors. ZFS does this so that freeing a block cannot leave a hole too small for any future allocation. On RAIDZ2 that is a multiple of 3 sectors, so a block occupying 4 sectors is charged 6. This padding is invisible in every tool except the gap between used and logicalused.

Now run the same arithmetic at 16 KiB on the same RAIDZ2 vdev: 4 data sectors plus 2 parity is 6 sectors, already a multiple of 3, so no padding. 24 KiB for 16 KiB stored: 67% efficiency, which is the number the table promised. The only thing that changed was the block size.

This is what the upstream note means by “may have worse space efficiency (especially on RAIDZ and DRAID)”, and it is why the same advice gives opposite answers on two pools that look similar from the Proxmox GUI.

The performance side runs the other way, which is what makes this a genuine trade rather than a right answer. A 4 KiB guest write into a 16 KiB block is a read-modify-write: read 16 KiB, verify its checksum, merge 4 KiB, recompute, write 16 KiB. Into an 8 KiB block it is half as much. For a workload that is overwhelmingly small random writes and sits on mirrors, 8 KiB genuinely is faster.

One more interaction worth holding on to: compression happens before allocation. A 16 KiB block that compresses to 5 KiB is allocated as 2 sectors, not 4. That is part of why larger blocks give better compression ratios — there is more redundancy to find in 16 KiB than in 8 KiB — and part of why the capacity penalty of a small block size is not fully predictable from the arithmetic alone. Measure used against logicalused on your actual data rather than trusting a table.

What happens if you find out eighteen months in that volblocksize is wrong?

Nothing breaks, which is exactly why it stayed wrong for eighteen months. You now have a decision about how much effort the correction is worth, and the honest answer is often “not much”.

Start by measuring rather than assuming. The question is not “is 8 KiB suboptimal” — it is “what is it costing us here”.

zfs list -t volume -o name,volsize,volblocksize,used,logicalused,compressratio -r tank
zpool list -o name,size,alloc,free,capacity,fragmentation

Divide used by logicalused per zvol and compare it against compressratio. On a mirror pool those two numbers should roughly agree. On RAIDZ with an 8 KiB block size, used/logicalused well above 1.3 while compressratio sits near 1.05 is the padding and parity overhead, and multiplying that gap by the pool size gives you the capacity you are paying for the setting.

Then decide, per workload rather than for the whole estate:

Leave it alone. If the pool is mirrors, the cost is close to zero and the correct action is to fix the storage setting so new disks are right, and never think about it again.

Fix it opportunistically. Every VM that gets rebuilt, restored or migrated for any other reason lands on the corrected storage and gets the right block size for free. Over a year most of the estate corrects itself at no cost. This is the right answer for most people.

Fix it deliberately. Worth the effort when the measurement shows the gap is large, when the pool is approaching a capacity purchase, or when one specific guest is both large and badly aligned. qm move-disk to a second storage entry with the right blocksize, a VM at a time, with --delete 0 so nothing is destroyed until it boots.

What not to do is the middle path people default to: schedule a project to fix the whole estate, discover it needs as much free space as the largest disk plus a maintenance window per VM, do a third of it, and leave the pool with three block sizes and no record of which is which. If you are going to do it deliberately, do it with an inventory and finish it.

The thing genuinely worth doing today, whichever you choose, takes one command: set blocksize correctly on the storage so the problem stops growing.

Common mistakes

Copying volblocksize=8K from a runbook written before ZFS 2.2. The default is 16 KiB now, and 8 KiB on RAIDZ costs real capacity.
Setting it below the guest filesystem block size. Upstream calls this meaningless, and it is: the guest still writes 4 KiB and ZFS just does more work.
Tuning it for the pool and ignoring the guest. An InnoDB VM and a file server on the same storage want different numbers; that is what a second storage entry is for.
Assuming pvesm set --blocksize fixed the existing disks. It applies to zvols created afterwards. Check with zfs get volblocksize.
Diagnosing the used-versus-logicalused gap as a compression problem. On RAIDZ with small blocks it is parity and padding, and no compression setting will change it.
Running qm move-disk with --delete 1 on the first pass. Keep the original until the VM has booted from the new disk.
Changing recordsize on a dataset and expecting existing files to follow. They do not; only new files get it.

Key takeaways

volblocksize is fixed when the zvol is written. It is the one ZFS setting with no in-place correction.
The default is 16 KiB since ZFS 2.2, and it is the right answer until you can name why it is not.
Never set it below the guest filesystem’s block size or below ashift.
On mirrors the choice barely affects capacity. On RAIDZ a small block size loses capacity to per-block parity and to allocation rounding.
Proxmox sets it per storage with blocksize; new zvols only.
Correct an existing disk with qm move-disk to a second storage entry, or by restoring it from PBS onto the corrected storage.
recordsize is the dataset equivalent and is correctable by rewriting files, which volblocksize is not.

Knowledge check

Why this matters

Most performance tuning is reversible, which is why most performance tuning can be deferred. Measure first, change later, change again if the measurement was wrong. That is the correct instinct and it is why experienced operators resist tuning things at build time.

volblocksize is the exception, and it is worth knowing that it is the exception, because the usual instinct produces the wrong behaviour here. Deferring the decision does not leave it open; it takes the default, which is then baked into every disk created in the meantime. The decision gets made whether or not you make it.

That would matter less if the cost were small. On a mirror pool it genuinely is, and the honest advice for most home labs and small clusters is to leave the default alone and spend the attention elsewhere. On RAIDZ it is not small: the difference between 8 KiB and 16 KiB on a RAIDZ2 vdev is the difference between 50% and 67% usable capacity, which on a 100 TB raw pool is 17 TB — and it presents as “ZFS is using more space than we calculated” rather than as anything identifiable.

The reason to teach this alongside the pool layout lesson is that both decisions have the same shape. Pool geometry and zvol block size are made once, before there is any data to measure, and enforced forever afterwards. Everything else about ZFS you can change on a Tuesday afternoon.

So the practical takeaway is a five-minute one. When you build a ZFS storage, look at the vdev type, look at what will run on it, write the blocksize down deliberately even if the answer is the default, and define a second storage entry the day a workload needs a different one. That is the entire intervention, and it is only available before the disks exist.