Filesystems for VM disks: ext4, xfs, and ZFS

Proxmox supports several filesystems underneath the qcow2 / raw abstraction. The right choice depends on your workload, your recovery story, and your willingness to operate a more complex stack.

The three options in practice

The “used by default” column is the strongest signal: PVE ships with both local (ext4 on a directory on the installer disk) and local-zfs (ZFS pool on dedicated disks). xfs is supported but not the default for either.

ext4 — the simple default

ext4 is a mature journaling filesystem with two-decade-old production history. It is what PVE puts on its installer disk by default when you don’t choose ZFS.

Strengths:

• Stable, predictable, well-understood. Every Linux admin has recovered an ext4 partition at 3am.
• Low overhead — no CoW, no checksums, no snapshots, no background scrub. The filesystem is essentially a fast allocator on top of the block device.
• No memory cost for caching (other than the page cache).
• fsck recovery is fast on small filesystems.

Weaknesses:

• No snapshots. To take a backup of a running VM on ext4, you must pause it (qm suspend 100) or use qemu-guest-agent to filesystem-freeze it. A “snapshot” is just a copy of the qcow2 file, and is only consistent if the guest cooperates.
• No checksums. Silent corruption (bit rot) propagates from the disk up to the VM. ext4 trusts the storage layer to deliver correct blocks.
• No send/receive. Replication between hosts is a copy or rsync of the qcow2 file.

Right choice for: small / dev clusters, NFS shares used as local storage, environments where you want the simplest possible stack and you’re willing to accept some operational limitations.

xfs — the alternative journaling filesystem

xfs is also a mature journaling filesystem with excellent performance characteristics on large filesystems. Red Hat Enterprise Linux uses xfs by default, which is why some shops standardize on it.

Strengths:

• Excellent performance on large filesystems (multi-TB).
• Robust against unclean shutdowns.
• Online defragmentation and expansion.
• Reflink snapshots since Linux 6.10 — not yet exposed by PVE but coming.

Weaknesses:

• Same snapshot limitations as ext4 from PVE’s perspective.
• Less common in PVE environments, so less community knowledge.
• Recovery tools (xfs_repair) require unmount and are slower than ext4’s fsck.

Right choice for: very large storage pools where ext4’s scaling limits become a concern, RHEL-skill-set shops.

ZFS — the production-grade choice

ZFS brings copy-on-write, snapshots, scrubs, checksums, and send/receive. It is the recommended filesystem for any production PVE cluster.

Strengths:

• Cheap snapshots — a snapshot is a property of the dataset, takes almost no space, and is instantaneous regardless of dataset size.
• Block-level checksums — every block is checksummed, and ZFS will detect and heal silent corruption from a redundant copy.
• Send/receive — replication between hosts is built in. zfs send | zfs recv streams the deltas of a snapshot.
• Scrub — a periodic background scrub verifies every block’s checksum and rebuilds from redundancy where needed. Catch silent corruption before it reaches your VM.
• ARC, L2ARC, ZIL — tunable caching layers for read-heavy and write-heavy workloads.

Weaknesses:

• More memory — ARC consumes RAM. Rule of thumb: 1 GB per TB of storage, plus working set. On a 64 GB host with 8 TB of fast SSDs, ARC can take 8 GB.
• More complex — vdevs, ashift, recordsize, compression ratio, dedup. Each has trade-offs.
• Hardware RAID is incompatible — ZFS wants direct access to disks to manage redundancy itself. A hardware RAID controller hides individual disk failures from ZFS, defeating its self-healing model.
• No production-grade dedup. The dedup table is in-memory; with large working sets it overflows to disk and tanks performance. Use compression instead.

Right choice for: any production cluster, especially with redundant disks (mirror, RAIDZ), shared storage scenarios, and wherever backups and replication matter.

When the choice is made for you

Some workloads force one or the other:

• Ceph OSD backing — must be a raw block device. The filesystem above doesn’t matter; OSD manages its own btrfs / bluestore. ext4, xfs, or ZFS all work as the host filesystem on top of which OSDs run.
• ZFS-over-ZFS — running ZFS on top of a ZFS pool (e.g., ZFS volume exported as iSCSI to another PVE host running ZFS) is fine but adds overhead. Don’t do it unless you have a specific reason.
• NFS / CIFS share — the filesystem on the NFS server is invisible to PVE. PVE sees a flat directory of files. Choose whatever the NFS server runs.
• Hardware RAID controller in JBOD mode — the controller passes individual disks through. ZFS sees them as separate devices. This is fine and recommended.

Practical recommendations

For a typical production PVE cluster of 3 nodes with local storage:

1. The boot disk: ext4 is fine. It’s a small filesystem that mostly holds the OS and PVE packages. ZFS boot is also fine but adds complexity for a small win.
2. The VM storage pool: ZFS mirror on dedicated SSDs. Use zfs create -o recordsize=64K -o compression=lz4 tank/vm-storage. 64K recordsize matches ZFS’s default volblocksize, which matches VM I/O patterns.
3. The PBS storage pool: ZFS mirror on dedicated HDDs. Recordsize doesn’t matter for chunk storage; PBS writes ~4 MB chunks sequentially. ARC caching helps for repeat reads of the same backup.
4. The NAS / cold storage: ext4 on a large disk. Cheap, simple, effective.

CLI walkthrough

Inspect what filesystems you’re using:

df -hT | grep -v tmpfs
lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT
zpool list
zpool status -v

For a new ZFS pool for VM storage:

# Identify the disks (replace with your actual device IDs)
ls -la /dev/disk/by-id/ | grep -v part

# Create a mirror pool
zpool create -f -o ashift=12 tank mirror \
  /dev/disk/by-id/ata-Samsung_SSD_870_EVO_1TB_S6PXNG0R200001 \
  /dev/disk/by-id/ata-Samsung_SSD_870_EVO_1TB_S6PXNG0R200002

# Create the dataset with VM-tuned properties
zfs create -o recordsize=64K \
  -o compression=lz4 \
  -o atime=off \
  -o primarycache=all \
  tank/vm-storage

# Add it to PVE
pvesm add zfspool vm-storage --pool tank/vm-storage --content images,rootdir

Verify the pool is healthy:

zpool status tank
# Expected: state ONLINE, no errors, scrub completed recently

Schedule a weekly scrub:

# Modern PVE: this is set via systemd timer
systemctl enable --now zfs-scrub-weekly@tank.timer
# Or via the legacy cron
( crontab -l 2>/dev/null | grep -v 'zpool scrub' ; \
  echo '0 3 * * 0 zpool scrub tank' ) | crontab -

Production considerations

• Different filesystems on the same host are fine. ext4 on the boot disk, ZFS for the VM pool, NFS mounted for cold storage — no conflict.
• Snapshot retention on ZFS — snapshots accumulate until you prune them. zfs list -t snapshot shows the count. Use pvesm retention policies or zfs-auto-snapshot to bound growth.
• Compression — lz4 is essentially free on modern CPUs and gives 1.2x–1.5x compression on VM workloads. Always enable.
• Recordsize — keep at 64K unless you have a specific reason. 128K helps sequential workloads (large databases), 16K helps small-file workloads (mail servers). The default is wrong for VMs.

Common mistakes

• Running ZFS on top of hardware RAID — defeats ZFS’s self-healing. Use the controller in JBOD / IT mode.
• Enabling dedup on production — the memory cost is enormous. Use compression instead.
• Mixing ashift values in a pool — older disks with 512-byte sectors combined with newer 4K disks force ashift=9, which wastes space. Create pools with ashift=12 on modern disks.
• Forgetting to set up scrubs — without a scrub schedule, silent corruption is detected only when ZFS tries to read a corrupt block, which can be after the second copy is also bad. Schedule a weekly scrub.

Key takeaways

• ext4 is the simple default. ZFS is the production-grade choice.
• ZFS requires direct disk access; no hardware RAID.
• Schedule weekly scrubs. Compression is free. Dedup is expensive.
• Match recordsize to your workload; 64K is the safe default.

Knowledge check