Boot and root layout: ZFS root, systemd-boot and GRUB · Proxmox VE

Two screens of the Proxmox installer decide things you cannot change afterwards without reinstalling: the root filesystem, and the disks it spans. Everything else on a Proxmox node is adjustable in production. This is not.

The decision is usually made in about forty seconds, and it is usually made by picking the option someone remembers from a blog post. This lesson is about making it on purpose.

What the installer writes to disk

The installer creates three partitions on every disk it selects, regardless of filesystem:

#	Size	Type	Purpose
1	1 MB	`EF02`	BIOS boot partition, for GRUB on legacy BIOS
2	512 MB	`EF00`	EFI System Partition (ESP)
3	remainder, or `hdsize`	—	The actual data: LVM PV or ZFS vdev member

Both boot partitions are created on every disk whatever the firmware mode, which is why a node installed under UEFI can still be booted after being moved to a legacy-BIOS chassis, and why every disk in a mirrored root has its own bootable ESP rather than only the first.

That last property is the one that matters operationally: a two-disk ZFS mirror has two ESPs, and keeping them identical is a job someone has to do. That job is proxmox-boot-tool, and it is the subject of the next lesson.

hdsize limits partition 3 so the rest of the disk is left free. On the answer-file path it is a key under [disk-setup.zfs] or [disk-setup.lvm]; interactively it is under Advanced Options.

The bootloader is derived, not chosen

There is no bootloader selection in the installer. It falls out of two facts about the machine:

Root filesystem	Firmware	Secure Boot	Bootloader
ZFS	UEFI	off	systemd-boot
ZFS	UEFI	on	GRUB
ZFS	legacy BIOS	—	GRUB
ext4 / xfs	any	any	GRUB

Two readings of that table are worth having.

ZFS root plus UEFI without Secure Boot is the only path to systemd-boot. Everything else lands on GRUB.

Enabling Secure Boot changes the bootloader on a ZFS-root system. It is not a firmware setting with no consequences above it. Deciding to turn Secure Boot on after installation means the machine boots by a different mechanism than it was installed with, and the ESP contents that mechanism expects are different.

ZFS root or LVM

This is the real decision. Both are supported, both are used in production, and they are good at different things.

	ZFS root	LVM on ext4 or xfs
Redundancy for the OS	Built in: mirror or raidz across the selected disks	None from the installer; needs hardware RAID
Bit-rot detection	Checksummed, and repaired from a good copy on scrub	Undetected
Snapshots of the host OS	Yes, cheap, and rollback-capable	No
Guest storage on the same pool	Yes, with snapshots and replication	LVM-Thin, snapshots but no replication
Built-in ZFS replication between nodes	Yes	No
Host RAM cost	ARC — significant, and must be bounded	Negligible
Behaviour with a RAID controller in front	Actively harmful	Expected
Boot complexity	Two or more ESPs to keep in sync	Usually one

Choose ZFS root when the node has direct-attached disks in HBA or IT mode, you want the OS itself to survive a disk failure without a hardware RAID controller, and you intend to use ZFS features — snapshots, zfs send replication, checksums — for guests as well.

Choose LVM on ext4 when there is a battery-backed hardware RAID controller you cannot or should not reconfigure, when host RAM is tight enough that ARC is a real cost, or when guest storage lives somewhere else entirely — Ceph, a SAN, an NFS array — and the local disks only need to hold the OS.

If you choose ZFS, bound the ARC

The single most common ZFS-root misconfiguration on a hypervisor is an unbounded ARC. Left alone, ZFS will grow its cache using memory the guests need, and the reclaim path under memory pressure is slower than the guest allocations competing with it.

Set it at install time — arc-max in the answer file, in MiB — or immediately after:

Configuration changebound the ARC persistently— Writes the module parameter for future boots and applies it to the running system. The value here is 16 GiB expressed in bytes; choose yours against total host RAM and the guest allocation you intend.

ARC_MAX_BYTES=17179869184

echo "options zfs zfs_arc_max=$ARC_MAX_BYTES" \
> /etc/modprobe.d/zfs.conf
update-initramfs -u -k all

echo "$ARC_MAX_BYTES" > /sys/module/zfs/parameters/zfs_arc_max
cat /sys/module/zfs/parameters/zfs_arc_max

The update-initramfs step is not optional on a ZFS-root system: the pool is imported from the initramfs, so the parameter has to be present there to apply from the first moment ZFS is active.

RAID level for the root pool

raid1 across two disks is the default answer and the right one for most nodes. The OS is small, the redundancy requirement is “survive one disk”, and rebuild time is minutes.

raid10 is worth it when guest storage shares the pool and you want the IOPS. raidz-1 and raidz-2 on a boot pool are a poorer trade than they look: write amplification is higher, and resilver on a raidz vdev is slower and heavier than on a mirror, which matters most when you are already down a disk.

Inspecting what you actually got

Read-only / Safethe layout as installed— Read-only. lsblk shows the three-partition pattern per disk; proxmox-boot-tool status names the bootloader on each synced ESP; findmnt confirms what root actually is.

lsblk -o NAME,SIZE,FSTYPE,PARTTYPENAME,MOUNTPOINT

proxmox-boot-tool status

findmnt /
[ -d /sys/firmware/efi ] && echo 'booted in UEFI mode' || echo 'booted in legacy BIOS mode'

Read-only / Safea two-disk ZFS mirror on UEFI without Secure Boot— Illustrative. Two ESPs, both synced, both using systemd-boot — the layout the bootloader table predicts for this combination.

# proxmox-boot-tool status

Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
E4C7-9B21 is configured with: uefi (versions: 7.0.2-6-pve)
E4CA-1F03 is configured with: uefi (versions: 7.0.2-6-pve)

Illustrative output

uefi in that output means systemd-boot. A GRUB-managed ESP reports grub instead, and a system with grub on some ESPs and uefi on others is a node that has been partially re-initialised — worth investigating before the next kernel upgrade.

What is reversible and what is not

Choice	Change later?
Root filesystem (ZFS / ext4 / xfs)	No. Reinstall.
Which disks the root spans	No for the vdev geometry; you can add a mirror to a ZFS vdev, not convert raidz
ZFS `ashift`	No. Pool-creation only.
Swap size, `maxroot`, `maxvz` on LVM	Partly — LVM volumes can be resized within the VG
ARC maximum	Yes, any time
ZFS compression	Yes, but only applies to newly written blocks
Bootloader	Only by re-initialising the ESPs
Secure Boot	Firmware toggle, but it changes the expected bootloader on ZFS root
Hostname and IP	Yes, with care

The three genuinely irreversible ones — filesystem, vdev geometry, ashift — are all decided on the same installer screen, in the same forty seconds. That asymmetry is the argument for the answer file: it turns those forty seconds into a reviewed document.

Under the hood

Why ZFS root needs a bootloader that can read ZFS, and why Proxmox picked the one it did.

A bootloader has to find and load a kernel and initramfs before any filesystem driver from the running system exists. On ext4 with GRUB, GRUB has an ext4 reader built in and reads /boot directly from the root filesystem.

ZFS is harder. GRUB has a ZFS reader, but it lags OpenZFS: a pool with feature flags GRUB does not implement becomes unreadable to it, and “enable a new pool feature and the node stops booting” is a genuinely unpleasant class of incident. Proxmox avoids it by not asking the bootloader to read ZFS at all.

On ZFS root with systemd-boot, the kernels and initramfs images are copied onto the ESP, which is plain FAT32 that firmware can read natively. The bootloader never touches the pool. proxmox-boot-tool is what performs that copy, and it is the reason the tool exists rather than being an optional convenience.

Three consequences follow directly.

/boot on a ZFS-root system is not where the booting kernel lives. The authoritative copies are on each ESP. Deleting a kernel from /boot does not remove it from the boot menu, and adding one there does not make it bootable.

The 512 MB ESP is a real constraint. Each kernel plus its initramfs is a few hundred megabytes, so the ESP holds a small number of them. This is why Proxmox keeps a bounded set of kernels rather than every one ever installed, and why an ESP that fills produces a failed update-initramfs during an ordinary apt upgrade — a confusing symptom whose cause is capacity rather than corruption.

Every ESP must be kept in sync. With one bootable disk this is invisible. With a mirror it is the whole point: if only the first disk’s ESP was updated, losing that disk means the survivor boots an older kernel, or does not boot at all. The next lesson is entirely about this.

On the GRUB paths — legacy BIOS, or ext4/xfs root — the picture is simpler. GRUB is installed to the BIOS boot partition or the ESP, /boot is an ordinary directory on the root filesystem, and update-grub regenerates the menu. Proxmox still uses proxmox-boot-tool to manage multiple ESPs where they exist, and the init subcommand takes a grub argument to force that mode explicitly.

The kernel command line differs accordingly: /etc/kernel/cmdline for systemd-boot, applied with proxmox-boot-tool refresh, and GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, applied with update-grub. Editing the wrong one is a change that appears to have been made and has no effect — the commonest symptom being an IOMMU or hugepages parameter that never took.

What happens if you install to a single disk and add a second one later?

You get redundancy for the data and, unless you do one more step, none for the boot path.

zpool attach converts a single-disk ZFS root into a mirror, and it works on a running system:

zpool status rpool
zpool attach rpool /dev/disk/by-id/existing-disk-part3 \
  /dev/disk/by-id/new-disk-part3

The resilver runs in the background and the pool is redundant when it finishes. At this point the pool survives losing either disk.

The node does not. The new disk was added as a bare vdev member; it has no partition table matching the original, no BIOS boot partition, and no ESP. Lose the original disk and the pool imports fine on hardware that can find something to boot — and there is nothing to boot, because the only ESP was on the disk that died.

The complete procedure has three parts, and only the first is the one people remember. Copy the partition table from the surviving disk so the new one has all three partitions. Attach partition 3 to the pool. Then format and initialise the new ESP with proxmox-boot-tool so it is included in the synced set. The next lesson covers those commands in detail; what matters here is knowing the second and third steps exist.

The verification is one command: proxmox-boot-tool status should list both ESPs. If it lists one, the mirror protects your data and not your ability to start the machine, and you will discover that during the disk failure rather than before it.

The same trap applies in reverse when replacing a failed disk in an existing mirror. zpool replace restores the pool. It does not restore the ESP, and a node that has silently been down to one bootable disk since its last disk replacement is a common finding on clusters that have been running for a few years.

Common mistakes

ZFS behind a RAID controller. ZFS cannot repair what it cannot see, and the failure surfaces long after the cause. HBA or IT mode.
Unbounded ARC on a hypervisor. It competes with guest memory. Set arc-max at install time, and remember update-initramfs on ZFS root.
ashift = 9 copied from an old guide. Permanent write amplification, fixable only by recreating the pool.
Assuming /boot is authoritative on ZFS root. The booting kernels live on the ESPs.
Adding a mirror disk and stopping after zpool attach. The pool is redundant; the boot path is not.
Enabling Secure Boot after a ZFS-root install. It changes the expected bootloader.
Editing /etc/default/grub on a systemd-boot system. The change appears to have been made and does nothing.

Key takeaways

The installer writes three partitions to every selected disk: 1 MB BIOS boot, 512 MB ESP, and the remainder.
The bootloader is derived: systemd-boot only on ZFS root under UEFI without Secure Boot; GRUB in every other combination.
ZFS root buys OS redundancy, checksums, snapshots and replication, at the cost of ARC memory and two ESPs to keep in sync. It requires HBA/IT mode.
LVM on ext4 is the right answer behind a hardware RAID controller, or when guest storage lives elsewhere.
Filesystem, vdev geometry and ashift are decided once and are not changeable afterwards.
On ZFS root the ESPs hold the kernels, /boot does not, and the 512 MB size is a real limit.

Knowledge check

Why this matters

Most production decisions on a Proxmox node are recoverable. Storage can be added, networks re-plumbed, guests migrated, kernels pinned and unpinned, and a wrong choice costs a change window.

The root layout is not in that category. Getting the filesystem or the vdev geometry wrong means evacuating the node and rebuilding it, which on a cluster means moving every guest twice and living with reduced capacity in between. The ashift case is worse: it is invisible, it degrades every small write for the life of the pool, and the only fix is to recreate the pool and move the data.

That asymmetry is why the boot layout deserves more thought than its forty seconds of installer time suggests, and why it is one of the strongest arguments for the answer file. A layout expressed in TOML gets reviewed by someone other than the person at the console, gets diffed against the last node, and gets rebuilt identically at 03:00 by whoever is awake.

The second theme is the one the ESP keeps raising. On ZFS root the boot path and the data path are genuinely separate systems that happen to share disks — the pool protects your data, and only the ESP set protects your ability to start the machine. Every operation that touches disks has to consider both, and the failure of the second is silent until the day it matters. That is the whole subject of the next lesson.