ZFS ARC, ZIL/SLOG, L2ARC · Proxmox VE

Why this matters in production

Misconfigured ZFS caching layers are responsible for two production failures: sync writes that are inexplicably slow (no SLOG), and read workloads that thrash (L2ARC misconfigured). Worse, many ZFS myths (“you need 1 GB RAM per TB”) waste budget and lead to over-spec.

The four caches

Cache	Where	What it does	Configuration
ARC	Host RAM	In-memory read cache; adaptive replacement	`zfs_arc_max`
ZIL	Pool or separate	Intent log for sync writes	Default on pool
SLOG	Separate fast device	Backs the ZIL with low-latency, durable storage	`zpool add tank log DEVICE`
L2ARC	Separate device	Disk-backed read cache	`zpool add tank cache DEVICE`

flowchart LR
  A[Read request] --> B[ARC hit?]
  B -->|yes| C[RAM: ~1 ms]
  B -->|no| D[L2ARC hit?]
  D -->|yes| E[L2ARC: ~0.1 ms]
  D -->|no| F[Pool: ~5 ms+]
  G[Sync write] --> H[ZIL in pool]
  H --> I[SLOG present?]
  I -->|yes| J[SLOG: ~0.1 ms]
  I -->|no| K[Pool sync write]

ARC — the in-memory read cache

ARC adapts to the workload: hot data stays in RAM.

The default depends on how the node was installed, and this catches people. The upstream ZFS default is 50% of host RAM (62.5% on some versions), which is far too much for a hypervisor that also runs VMs. Since Proxmox VE 8.1, a fresh installation sets ARC to 10% of host memory, clamped to a maximum of 16 GiB — so a node installed recently is already sensible, and a node upgraded from an older release may still be running the upstream default.

Read-only / Safewhat is ARC actually limited to on this node?— Read-only. A value close to half of total RAM means the node predates the Proxmox default and nobody has tuned it. arc_summary gives the same figures with hit ratios beside them.

cat /sys/module/zfs/parameters/zfs_arc_max
cat /sys/module/zfs/parameters/zfs_arc_min
free -g
arc_summary | head -30

Setting it permanently:

Configuration changelimit ARC to 16 GiB— Writes the module parameter. It takes effect at the next boot, and on a root-on-ZFS system the initramfs must be regenerated as well or the boot-time value is the old one. Nothing changes until you reboot.

echo 'options zfs zfs_arc_max=17179869184' > /etc/modprobe.d/zfs.conf

# required when ZFS is the root filesystem
update-initramfs -u -k all

Configuration changeapply it now, without rebooting, to test the effect— Writes the running value directly. It is lost at reboot, which makes it the right way to test a number before committing to it. ARC does not shrink instantly - it releases memory as pressure requires.

echo "$[16 * 1024*1024*1024]" > /sys/module/zfs/parameters/zfs_arc_max

`zfs_arc_min` silently overrides a low `zfs_arc_max`

zfs_arc_min defaults to 1/32 of system memory. If the zfs_arc_max you set is lower than or equal to it, the setting does not do what you asked.

This bites on large-memory nodes specifically. On a host with 512 GiB of RAM, zfs_arc_min is 16 GiB — so an attempt to cap ARC at 8 GiB is below the floor and has no effect. The node keeps consuming memory and the operator concludes the parameter does not work.

Lower the minimum first, then the maximum:

echo "$[8 * 1024*1024*1024 - 1]" > /sys/module/zfs/parameters/zfs_arc_min
echo "$[8 * 1024*1024*1024]"     > /sys/module/zfs/parameters/zfs_arc_max

The Proxmox sizing guidance for the other direction — how much ARC a pool actually needs — is 2 GiB base plus 1 GiB per TiB of storage. An 8 TiB pool wants around 10 GiB. That figure and the 10%-of-RAM default disagree on small-memory nodes with large pools, and when they do, the pool sizing is the one to follow.

ZIL and SLOG

The ZIL (ZFS Intent Log) records sync writes before they hit the pool. By default the ZIL lives on the pool itself, which means sync writes compete with the bulk pool for I/O.

A SLOG (Separate Log) device moves the ZIL to a small, low-latency, durable SSD. Sync writes go to the SLOG, then are written to the pool asynchronously.

SLOG device	Sync write latency
None (ZIL on pool)	~5 ms+
SATA SSD with PLP	~0.5 ms
NVMe with PLP	~0.1 ms

zpool add tank log /dev/disk/by-id/nvme-SLOG-Device

L2ARC

L2ARC extends ARC to a disk-backed cache. Useful when:

The working set exceeds ARC (RAM).
Reads dominate the workload.

L2ARC is not useful when:

Writes dominate (L2ARC caches reads only).
The ARC is already large enough.

Common myths debunked

Myth	Reality
ZFS needs 1 GB RAM per TB of storage	A 2010-era rule of thumb. With modern ARC tuning, 8 GB minimum plus 1 GB per 100 TB if that
ZFS requires ECC RAM	ZFS detects bit-rot. Without ECC, the host memory can flip a bit and ZFS will use the wrong checksum — but the checksum itself will be wrong and ZFS will report it. ECC helps but is not strictly required
ZFS needs a battery-backed RAID controller	ZFS must see raw disks. RAID controllers hide errors
SLOG must be huge	SLOG only needs to hold ~5 seconds of sync writes. A small SSD is fine

Production considerations

Common mistakes

Letting ARC grow unbounded.
Setting zfs_arc_max below zfs_arc_min and concluding the parameter does not work. Lower the minimum first on large-memory nodes.
Forgetting update-initramfs -u -k all on a root-on-ZFS node. The boot-time ARC limit is the old one.
Putting swap on a zvol. The failure mode is a deadlock under memory pressure, not slow swapping.
Using a consumer SSD as SLOG.
Adding L2ARC to a write-heavy workload.
Over-provisioning RAM because of “1 GB per TB.”

Key takeaways

ARC: in-RAM read cache. Tune zfs_arc_max to leave room for VMs.
SLOG: separate device for sync writes. Mandatory for high-sync workloads.
L2ARC: optional read cache. Avoid on write-heavy workloads.