Tuning per backend: ZFS ARC, Ceph OSD, LVM-Thin

The generic storage advice — use virtio-scsi-single, enable iothread, benchmark with the right block size — applies everywhere and is covered in xvii-performance-storage. This lesson is about the levers that exist on exactly one backend, where getting them wrong produces a failure the generic advice cannot explain.

There are three backends worth treating separately on a Proxmox node, and each has one setting that matters more than all its others combined:

ZFS: the ARC is not page cache, and that is the whole problem

The ARC is ZFS’s read cache. On a file server it is unambiguously good — give it everything spare. On a hypervisor it is competing directly with guest memory, and the two are reclaimed by different mechanisms with different urgency.

Proxmox’s default acknowledges this. From the Proxmox documentation, ZFS “uses 10% of the host memory, clamped to a maximum of 16 GiB” for the ARC by default. That is a deliberately conservative choice for a hypervisor and it is usually the right starting point — but “usually” is doing work, and the measurement below tells you which case you are in.

Measure before sizing

# current size and target, sampled
arcstat 1 5

# the configured ceiling, in bytes (0 means "use the default")
cat /sys/module/zfs/parameters/zfs_arc_max

# the full picture
arc_summary | head -40

# arcstat 1 5

    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
14:22:01  4.1K    3.9K    97     212    88     11   45  15.8G  16.0G  84.2G
14:22:02  3.8K    3.6K    96     198    91      9   44  15.8G  16.0G  84.2G
14:22:03  4.4K    4.2K    97     221    86     14   50  15.9G  16.0G  84.1G

Two readings, two different decisions:

• Hit ratio above ~90% — the ARC is already serving nearly everything from RAM. Making it larger buys you very little, and every gigabyte you give it is a gigabyte a guest cannot have.
• Hit ratio below ~70% with size pinned at c — the working set does not fit. More ARC would genuinely help, and now you have a number to justify it with.

A hit ratio below 70% with size well below c is a third case: ZFS wanted more and could not get it, which means the host is under memory pressure and the ARC is already losing the fight. That is a capacity problem, not a tuning one.

Sizing it

# 24 GiB, in bytes
echo $(( 24 * 1024 * 1024 * 1024 )) > /sys/module/zfs/parameters/zfs_arc_max

cat /sys/module/zfs/parameters/zfs_arc_max
arcstat 1 3

cat > /etc/modprobe.d/zfs.conf <<'EOF'
options zfs zfs_arc_max=25769803776
EOF

# only needed on a ZFS-root node:
update-initramfs -u -k all

cat /etc/modprobe.d/zfs.conf

The sizing rule

The Proxmox documentation gives a rule of thumb for ZFS memory: “at least 2 GiB Base + 1 GiB/TiB-Storage”. On a hypervisor, run that alongside the guest arithmetic:

ARC ceiling = physical RAM
            - sum of guest maximum memory
            - host overhead (services, PVE daemons: allow 8-16 GiB)
            - failover reserve (what N x u / (N-1) says you must absorb)

If that arithmetic leaves less than the 2 GiB + 1 GiB/TiB floor, the node does not have enough memory for the storage you have put on it. That is a finding, not a number to fudge.

The Proxmox ZFS documentation also recommends lowering vm.swappiness to 10 on servers, and warns that swap on a zvol “may generate some troubles, like blocking the server or generating a high IO load” — recommending a physical partition instead. Both are covered in the host tuning lesson; they matter more on a ZFS node than anywhere else.

Ceph: mClock replaced the levers you have read about

This is the section most likely to correct something you already “know”, because the settings almost every Ceph performance guide names have not done anything since Reef.

The mClock scheduler is the default OSD scheduler. The Ceph documentation is explicit that it overrides osd_max_backfills, osd_recovery_max_active, osd_recovery_max_active_hdd and osd_recovery_max_active_ssd, and forces to zero the entire osd_recovery_sleep family along with osd_scrub_sleep, osd_delete_sleep and osd_snap_trim_sleep.

So ceph config set osd osd_recovery_sleep 5 — the single most-repeated piece of Ceph throttling advice on the internet — accepts your value, stores it, reports it back to you when you ask, and changes nothing.

ceph config get osd osd_op_queue
ceph config get osd osd_mclock_profile

ceph -s
ceph osd perf

The three built-in profiles:

ceph config set osd osd_mclock_profile high_client_ops

ceph config get osd osd_mclock_profile
ceph -s

ceph config set osd osd_mclock_profile balanced

ceph config get osd osd_mclock_profile

The legacy settings can be re-enabled with osd_mclock_override_recovery_settings, which defaults to false. The Ceph documentation’s own guidance is that “the recommendation is to retain the defaults as is on a running cluster as modifying them could have unexpected performance outcomes” — so this is a lever of last resort, not the way back to familiar territory.

ceph -s | grep -A2 'io:'
ceph osd perf
ceph osd pool stats

LVM-Thin: the setting that is a data-loss control, not a performance one

LVM-Thin’s tuning surface is small. Its one critical setting is not about speed at all.

A thin pool over-provisions: the sum of the volumes allocated from it can exceed its physical size. That is the feature. The consequence is that the pool can run out of physical extents while every guest believes it has space, and when that happens, writes fail in a way guest filesystems are not designed to survive. Filesystems remount read-only in the good case and corrupt in the bad one — and it happens to every guest on the pool simultaneously.

lvs -o lv_name,vg_name,lv_size,data_percent,metadata_percent,lv_attr

vgs
pvesm status

# lvs -o lv_name,vg_name,lv_size,data_percent,metadata_percent

  LV   VG  LSize    Data%  Meta%
data pve  <1.64t   91.42  62.18
root pve   96.00g
swap pve    8.00g

grep -nE 'thin_pool_autoextend' /etc/lvm/lvm.conf

# set thin_pool_autoextend_threshold = 80
# and thin_pool_autoextend_percent   = 20
# then confirm the daemon that acts on it is running:
systemctl status lvm2-monitor --no-pager

For performance specifically, LVM-Thin’s one meaningful lever is the chunk size, which is fixed when the pool is created and cannot be changed afterwards. Small chunks give better space efficiency and worse sequential throughput; large chunks the reverse. If an existing pool has the wrong chunk size, the remedy is a new pool and a storage migration, which makes it a Part V decision rather than a tuning one.

Measuring the effect of any of this

Every change above should produce a number that moved. If you cannot show one, the change was a guess.

STAMP=$(date +%Y%m%d-%H%M%S)
OUT="/root/storage-baseline-$STAMP.txt"

{
echo "== zfs =="
arcstat 1 5
zpool iostat -v 1 3

echo "== ceph =="
ceph -s
ceph osd perf

echo "== lvm-thin =="
lvs -o lv_name,vg_name,data_percent,metadata_percent

echo "== host =="
cat /proc/pressure/io
} > "$OUT" 2>/dev/null

echo "written: $OUT"

The single most useful line in that capture is /proc/pressure/io. A tuning change that lowers I/O pressure has done something for every guest on the node. One that leaves it unchanged has not, whatever the throughput numbers say.

Common mistakes

• Leaving zfs_arc_max unset on a hypervisor. The default of 10% capped at 16 GiB is deliberately conservative and usually fine; an unbounded ARC on a node whose guests need the memory is a capacity plan with a variable in it.
• Reading free -h on a ZFS node and believing it. The ARC is not reclaimable at page-cache speed. arcstat size plus MemAvailable is the honest figure.
• Adding ARC without checking the hit ratio. Above ~90%, more ARC buys almost nothing and costs a guest its memory.
• Setting osd_recovery_sleep or osd_max_backfills on Squid or Tentacle. mClock overrides them. The value is stored, reported back, and ignored.
• Leaving a cluster on high_client_ops permanently. Every future recovery is then slower than it needs to be — a continuous durability cost for an occasional performance benefit.
• Trusting thin-pool autoextend without checking the volume group. Autoextend cannot create extents that do not exist. Alert on pool usage regardless.
• Ignoring Meta% on a thin pool. Metadata exhausts independently and produces the same failure as data exhaustion.
• Changing something without a before capture. “It feels faster” is not a result. /proc/pressure/io before and after is.

Key takeaways

• ZFS ARC defaults to 10% of host memory capped at 16 GiB on Proxmox. Set zfs_arc_max explicitly and persist it in /etc/modprobe.d/zfs.conf; update-initramfs -u -k all is needed only on a ZFS-root node.
• ARC is not page cache: its eviction is asynchronous, so a fast large allocation can OOM a host that looks like it has free memory.
• Decide ARC size from the hit ratio, not from a rule. Above ~90%, more ARC buys nothing.
• The Proxmox floor is 2 GiB + 1 GiB per TiB of storage; if guest memory plus failover reserve leaves less than that, the node is undersized for its pool.
• On Ceph Squid and Tentacle, mClock is the default and overrides osd_max_backfills, the osd_recovery_max_active family and every *_sleep option. Use osd_mclock_profile instead.
• high_client_ops lengthens the degraded window. It is a durability trade, and the wrong choice on a pool already down to one copy.
• LVM-Thin’s critical setting is autoextend, and it is inert unless the volume group has free extents. Alert on pool Data% and Meta% independently.
• Prove every change with a before-and-after capture. /proc/pressure/io is the line that speaks for all guests at once.

Knowledge check