NFS in detail: versions, performance, and PVE integration

NFS is the most common shared-storage backend in mixed environments because every operating system speaks it and the configuration is simple. But “NFS is just NFS” hides a stack of version differences, mount-option trade-offs, and performance pitfalls that bite you at scale. This lesson covers what you actually need to know.

NFS versions

For PVE 9.x use NFSv4.1 or NFSv4.2. NFSv3 works but has lower throughput and no pNFS for parallel I/O.

The default mount in modern Linux is NFSv4.1. To force a specific version: mount -t nfs -o vers=4.1 server:/export /mnt.

Export configuration for PVE

A correct /etc/exports for VM storage:

/vm-storage  10.0.0.0/24(rw,sync,no_subtree_check,no_root_squash,fsid=1)

The options matter:

• rw — read-write. PVE needs this for VM disks.
• sync — the server replies only after writes hit stable storage. Required for data integrity; enables safe snapshots on the server side. Tradeoff: lower throughput than async.
• no_subtree_check — disables subtree checking, which is a legacy safeguard that breaks on busy exports.
• no_root_squash — the NFS root (uid 0) on the client maps to NFS root (uid 0) on the server. Required because QEMU runs as root on the PVE node and creates files owned by root.
• fsid=1 — explicit filesystem ID. Without this, NFSv4 can get confused about which export is which if multiple exports share a filesystem.

For an asynchronous (faster but less safe) export — only use this if you have a battery-backed write cache on the NFS server:

/vm-storage  10.0.0.0/24(rw,async,no_subtree_check,no_root_squash,fsid=1)

After editing /etc/exports:

exportfs -rav    # Re-export without bouncing nfsd
showmount -e     # Verify the export is visible

Mount options

PVE auto-mounts NFS with sensible defaults, but you can override. The key options for VM storage:

mount -t nfs -o \
  vers=4.1,        # NFS version
  rsize=1048576,   # 1 MB read size — max for most networks
  wsize=1048576,   # 1 MB write size — max for most networks
  hard,            # Retry forever on hangup (not soft)
  timeo=600,       # 60-second timeout per request
  retrans=3,       # Retry 3 times before giving up
  actimeo=30       # 30-second attribute cache
  server:/vm-storage /mnt/pve/nfs-storage

The defaults rsize=32768 and wsize=32768 (32 KB) are too small for VM workloads and throttle your IOPS. Always set them to the network MTU or larger.

For an NFS-mounted PBS datastore, increase the read-ahead:

mount -t nfs -o vers=4.1,rsize=1048576,wsize=1048576,noatime \
  pbs.example.com:/backups /backup-pool

Authentication: NFS and Kerberos

NFSv4 supports Kerberos authentication in three modes:

• krb5 — authentication only (the data is still unencrypted on the wire)
• krb5i — authentication + integrity checking (each request is signed, preventing tampering)
• krb5p — authentication + integrity + privacy (data is encrypted)

For production, krb5p is the right choice. The performance overhead is real but manageable on a 10 GbE network with AES-NI.

Setting up Kerberos NFS is non-trivial and out of scope for this lesson. The high-level steps:

# On the NFS server
apt install -y krb5-kdc krb5-admin-server krb5-user
krb5_newrealm
kadmin.local -addprinc -randkey nfs/server.example.com
kadmin.local -kt /etc/krb5.keytab nfs/server.example.com

Export with krb5p — this is a line in /etc/exports on the server, not a command:

/vm-storage  10.0.0.0/24(rw,sync,sec=krb5p,no_subtree_check,no_root_squash)

Then on the client:

# On the PVE client
apt install -y krb5-user
kadmin.local -addprinc -randkey nfs/client.example.com
mount -t nfs -o sec=krb5p server:/vm-storage /mnt/pve/nfs

For most clusters, plain NFS over a private VLAN is acceptable. Use Kerberos when sharing NFS across trust boundaries or when compliance requires it.

NFS performance tuning

Three knobs matter for NFS throughput:

1. rsize / wsize

The block size used for read and write operations. Default 32 KB. On 10 GbE networks, the optimal size is 1 MB (the maximum the NFSv3 and v4 protocols support). Anything smaller wastes network bandwidth; anything larger is silently capped.

# Verify the actual size used (cat /proc/mounts)
mount | grep nfs
# /mnt/pve/nfs on /mnt/pve/nfs type nfs (...,rsize=1048576,wsize=1048576,...)

# Adjust on remount
mount -o remount,rsize=1048576,wsize=1048576 /mnt/pve/nfs

2. Number of NFS daemons

nfsd runs as multiple kernel threads. Default is 8, which is usually too few for VM workloads. Increase to match the number of concurrent VMs:

# /etc/nfs.conf
[nfsd]
threads=64

Then apply and confirm:

systemctl restart nfs-server
cat /proc/fs/nfsd/threads   # Verify

3. TCP vs UDP (and modern TCP tuning)

NFSv4 is TCP-only. UDP is gone. To get the most out of a 10 GbE link, tune TCP:

# /etc/sysctl.d/10-nfs-network.conf
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 1048576 16777216
net.ipv4.tcp_wmem = 4096 1048576 16777216
net.core.netdev_max_backlog = 30000
net.ipv4.tcp_fastopen = 3

sysctl --system

Diagnosing NFS problems

When NFS feels slow, run this triage:

# 1. Check the server is responsive
ping -c 5 nfs-server

# 2. Check the export is mounted
mount | grep nfs
df -h /mnt/pve/nfs

# 3. Check the network bandwidth
iperf3 -c nfs-server

# 4. Check NFS-specific stats
nfsstat -c     # Client stats — look at "badxids" (should be 0)
nfsstat -s     # Server stats — look at "badxids", "requests"

# 5. Check I/O stats at the mount
iostat -dx 1 /mnt/pve/nfs   # r/s, w/s, await, %util

# 6. Check mount options
mount | grep nfs
cat /proc/mounts | grep nfs
nfsstat -m

Common issues and what they mean:

Production considerations

• Use a dedicated VLAN for NFS traffic. VM I/O competes with everything else on the management network. A separate 10 GbE link with jumbo frames (MTU 9000) gives the best throughput.
• Tune the NFS server’s storage. NFS is just a transport; if the server’s disks are slow, NFS is slow. Use SSDs or NVMe for the backend.
• Watch for silent failures. NFS servers can become unresponsive for tens of seconds during heavy I/O. The default hard mount retries forever (correct); the soft mount returns errors to applications (catastrophic for VMs).
• Snapshot on the server side. Many NFS appliances (TrueNAS, NetApp, etc.) support snapshots via their own tooling. Use those for the local backup story; use PBS for off-host.

Common mistakes

• Using NFSv3 by accident. Modern Linux defaults to v4.1 but old exports can downgrade. Always check mount | grep vers.
• Forgetting no_root_squash. QEMU creates files as root; if the NFS server squashes root to nobody, all writes fail.
• Setting rsize/wsize too small. 32 KB default caps throughput at ~2.5 Gbps even on a 10 GbE link.
• Putting NFS exports on the same network as live migration. Live migration saturates the network, NFS I/O stalls, VM disk operations time out. Separate network or QoS.

Key takeaways

• Use NFSv4.1 or v4.2. Set rsize/wsize to 1 MB.
• no_root_squash and no_subtree_check are mandatory for PVE.
• Increase nfsd threads to match VM count.
• Use a dedicated VLAN with jumbo frames.

Knowledge check