proxmox-boot-tool: ESP synchronisation and kernel pinning · Proxmox VE

proxmox-boot-tool exists because a mirrored root protects your data and not your ability to start the machine. The pool survives a disk failure by design. The boot path survives it only if somebody kept the second EFI System Partition current, and nothing in ZFS does that.

This is also the tool you reach for when a kernel upgrade produces a node that boots into something worse than it had before — which is a live concern right now, because PVE 9.2 moved the default kernel from the 6.x series to 7.0.

What it manages

Two things, and understanding that they are separate makes the tool straightforward.

The synced ESP set. A list of partition UUIDs in /etc/kernel/proxmox-boot-uuids. Every ESP on that list receives an identical copy of the bootloader configuration, the kernels and the initramfs images on every refresh.

The kernel selection. Which kernels get copied onto those ESPs, and which one boots by default.

Everything the tool does is one of those two.

Reading the current state

Read-only / Safethe first command in any boot investigation— Read-only. Names the firmware mode the node booted in, then each synced ESP with its bootloader type and the kernel versions present on it.

proxmox-boot-tool status

Read-only / Safea healthy two-disk mirror— Illustrative. Two ESPs, both listed, both carrying the same two kernel versions. This is the state to compare against after any disk operation.

# proxmox-boot-tool status

Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
E4C7-9B21 is configured with: uefi (versions: 6.17.2-4-pve, 7.0.2-6-pve)
E4CA-1F03 is configured with: uefi (versions: 6.17.2-4-pve, 7.0.2-6-pve)

Illustrative output

Three things to read off that output every time:

How many ESPs are listed. A two-disk mirror showing one ESP is a node with a single point of failure that its owners believe is redundant.

Whether the versions match. Different kernel lists on different ESPs means a refresh did not complete, and the node’s behaviour after a disk failure depends on which disk survived.

uefi versus grub. uefi means systemd-boot. A mix across ESPs on one node means a partial re-initialisation.

The kernel list is a separate question:

Read-only / Safewhich kernels are managed, and is one pinned?— Read-only. Lists the kernel versions proxmox-boot-tool will copy to the ESPs, and marks a pinned version if one is set.

proxmox-boot-tool kernel list
uname -r

uname -r alongside it is deliberate. The list is what is available; uname -r is what is running. When those disagree after a reboot, the node did not boot what you expected.

Adding a replacement disk to the boot set

This is the procedure that gets skipped, and it is the reason nodes end up with one bootable disk.

After zpool replace or zpool attach restores the pool, the new disk has a vdev member and nothing else. Give it the same three-partition layout and add its ESP to the synced set.

Read-only / Safestep 1 — identify both disks unambiguously— Read-only. Use by-id paths from here on: kernel names such as sda can move between reboots, and this procedure writes partition tables.

lsblk -o NAME,SIZE,MODEL,SERIAL,PARTTYPENAME
ls -l /dev/disk/by-id/ | grep -v part

Destructivestep 2 — copy the partition layout to the new disk— Overwrites the partition table on the target. Verify both paths before running: sgdisk -R replicates SOURCE onto TARGET, and the argument order is target first. Getting it backwards destroys the surviving disk.

HEALTHY=/dev/disk/by-id/ata-SAMSUNG_MZ7L3480_S6WPNE0T123456
NEW=/dev/disk/by-id/ata-SAMSUNG_MZ7L3480_S6WPNE0T654321

sgdisk "$HEALTHY" -R "$NEW"
sgdisk -G "$NEW"

lsblk "$NEW"

sgdisk -G assigns new random GUIDs to the copied table. Without it both disks carry identical partition GUIDs, which confuses anything that identifies partitions that way — including the boot-uuid list this procedure is about to write to.

Destructivestep 3 — format and initialise the new ESP— format erases partition 2 and writes a FAT filesystem. init adds the partition UUID to /etc/kernel/proxmox-boot-uuids and copies the current bootloader and kernels onto it.

NEW=/dev/disk/by-id/ata-SAMSUNG_MZ7L3480_S6WPNE0T654321

proxmox-boot-tool format "$NEW-part2"
proxmox-boot-tool init "$NEW-part2"

On a system using GRUB rather than systemd-boot, pass the mode explicitly: proxmox-boot-tool init "$NEW-part2" grub. Check proxmox-boot-tool status first — it tells you which one this node uses.

Read-only / Safestep 4 — verify, and do not skip this— Read-only. Both ESPs must appear with matching kernel version lists. This is the only evidence that the node can boot from either disk.

proxmox-boot-tool status

The genuinely complete verification is to boot from the new disk once, by selecting it in the firmware boot menu during a scheduled reboot. Nobody does this and it is the only thing that proves the work. On a cluster with HA and a maintenance window it costs one node reboot.

Kernel pinning

Proxmox keeps several kernels available and boots the newest by default. When the newest one is the problem, pinning is how you stay on the one that works.

This is not hypothetical for PVE 9.2. The 6.x-to-7.0 default kernel transition is exactly the kind of change that surfaces a driver or firmware interaction on a subset of hardware — a NIC that stops passing traffic under load, an HBA that resets, a passthrough device that no longer binds — while every other node in the cluster is fine.

Read-only / Safewhat can be pinned?— Read-only. The exact version strings from this list are what pin accepts. Copy one rather than typing it: a version that does not match is rejected.

proxmox-boot-tool kernel list

Configuration changepin a known-good kernel— Sets the default boot entry to a specific version and refreshes every synced ESP. Takes effect on the next boot and persists across upgrades until unpinned.

proxmox-boot-tool kernel pin 6.17.2-4-pve

proxmox-boot-tool kernel list
proxmox-boot-tool status

The pin persists. New kernels continue to install and appear in the list, and none of them becomes the default while the pin is set. That is the desired behaviour during an investigation and a hazard afterwards: a node pinned in March and forgotten is a node that has not taken a kernel security fix since March, while apt reports everything as up to date.

Testing a kernel without committing to it

--next-boot applies the pin to exactly one boot. The boot after that returns to the normal default.

Configuration changetry a kernel for one boot only— Sets a one-shot boot entry. If the node does not come back, a power cycle returns it to the previous default with no state left behind.

proxmox-boot-tool kernel pin 7.0.2-6-pve --next-boot

reboot

This is the right tool for testing a suspect kernel, and the wrong tool for a rollback you intend to keep. A one-shot boot that fixes production is a reprieve: the next reboot — weeks later, during an unrelated incident — silently returns to the kernel that broke. A rollback you mean to keep is a plain pin without --next-boot.

Configuration changerelease the pin— Restores the default behaviour of booting the newest available kernel. Use --next-boot to clear only a one-shot pin.

proxmox-boot-tool kernel unpin

proxmox-boot-tool kernel list

The rest of the kernel subcommands

kernel add and kernel remove control which versions are copied to the ESPs, independently of what is installed. kernel remove is the answer to a full ESP: dropping an old version frees a few hundred megabytes on every synced partition.

refresh re-copies the current configuration and kernel set to every ESP. It is what apt triggers automatically after a kernel change, and it is what you run by hand after editing /etc/kernel/cmdline, because that file is not consulted at boot — its contents are copied onto the ESPs, and a change that has not been refreshed has not happened.

Configuration changechange the kernel command line on a systemd-boot system— Edits the command line and propagates it to every ESP. Without the refresh the file is changed and the node boots with the old parameters.

cat /etc/kernel/cmdline
# edit the file, then:
proxmox-boot-tool refresh

# after the next reboot, confirm it took:
cat /proc/cmdline

clean removes entries from /etc/kernel/proxmox-boot-uuids whose partitions no longer exist. Run it after physically removing a disk; without it, every refresh tries to write to a partition that is gone and reports errors that look worse than they are.

Under the hood

Why synchronisation is a copy rather than a mount, and what that costs.

An ESP is FAT32 with no redundancy of its own. It cannot be part of a ZFS pool or an mdadm array, because firmware has to read it before any of that exists. So Proxmox keeps N independent copies and makes them identical by writing to each one in turn.

proxmox-boot-tool refresh mounts each UUID in /etc/kernel/proxmox-boot-uuids under a temporary path, writes the bootloader configuration and the selected kernels and initramfs images, and unmounts it. The re-exec into a private mount namespace that the tool announces on every run exists so those temporary mounts are invisible to the rest of the system and cannot be left behind by an interrupted run.

Four consequences that matter operationally.

The UUID list is the source of truth, not the disks present. An ESP that exists but is not in the list receives nothing. This is exactly the state after zpool replace without init — the partition is there and looks plausible under lsblk, and it is empty of anything bootable.

A refresh is not atomic across ESPs. They are written sequentially, so a crash or a power loss mid-refresh leaves some ESPs updated and others not. That is survivable by design — each ESP is independently bootable — and it is why the version lists in status can legitimately disagree for a few seconds and must not disagree afterwards.

The ESP is 512 MB and each kernel plus initramfs is a few hundred megabytes. That bounds the set to a small number of versions. When it fills, update-initramfs fails during an ordinary apt upgrade, with an out-of-space error that points at the ESP rather than at the root filesystem — a confusing message whose remedy is proxmox-boot-tool kernel remove for a version you no longer need.

The tool is not only for ZFS root. Any Proxmox installation with more than one ESP benefits, and the init subcommand takes a grub argument to manage GRUB-mode ESPs the same way. What differs is where the kernel command line lives: /etc/kernel/cmdline with refresh for systemd-boot, /etc/default/grub with update-grub for GRUB.

The pin itself is stored alongside the boot configuration and written onto each ESP during refresh, which is why pinning ends with a refresh and why a pin set while one disk was absent does not apply to that disk until it is refreshed again.

What happens if a node will not boot after a kernel upgrade?

You need console access. Not SSH — the node is not up. IPMI, iDRAC, iLO, or a crash cart. If your estate has no out-of-band access, this is the incident that establishes the requirement, and it will take hours rather than minutes.

First, try the boot menu. systemd-boot shows a menu at boot; press a key during the countdown to interrupt it and select an older kernel. GRUB has the same under Advanced options. If an older kernel boots, you have a running node and time to think, and the immediate next step is proxmox-boot-tool kernel pin on that version so a subsequent reboot does not undo the fix.

If nothing in the menu boots, or there is no menu, boot the Proxmox ISO and choose Advanced Options → Rescue Boot. That imports the root pool and gives you a shell on the installed system rather than on the installer.

From there the question is which of two failures you have.

The kernel is broken but the boot path is intact. The menu appeared, entries existed, the selected kernel panicked or hung. Pin a working version and refresh:

proxmox-boot-tool kernel list
proxmox-boot-tool kernel pin 6.17.2-4-pve
proxmox-boot-tool refresh

The boot path itself is broken. No menu, firmware reporting no boot device, or an ESP that failed with its disk. Establish what the tool believes first, because the state it reports drives everything after it:

proxmox-boot-tool status
lsblk -o NAME,SIZE,FSTYPE,PARTTYPENAME
cat /etc/kernel/proxmox-boot-uuids

If an ESP is listed that no longer exists, proxmox-boot-tool clean removes it. If a disk is present with an ESP that was never initialised, format and init bring it into the set. If the surviving ESP is intact but stale, refresh rewrites it.

The case that needs care is a node whose only ESP failed with its disk while the pool survived on the mirror. The data is fine; there is nothing to boot. Recovery is the disk-replacement procedure from earlier in this lesson, run from the rescue environment: copy the partition table from the surviving disk, format, init, refresh, then confirm status lists both.

Two habits make all of this shorter. Keep proxmox-boot-tool status output in your monitoring or your configuration-management facts, so “how many bootable disks did this node have yesterday” is answerable without the node. And do the kernel upgrade on one node first, with a soak, so the version that broke is known before the second node takes it — which turns this from an outage into a pin on one machine.

Common mistakes

Replacing a disk and stopping at zpool replace. The pool is redundant; the boot path is not. format, init, then verify.
sgdisk -R with the arguments reversed. It overwrites the surviving disk’s partition table. Read it aloud first.
Skipping sgdisk -G. Duplicate partition GUIDs confuse the tools that identify partitions by GUID, including the boot-uuid list.
Editing /etc/kernel/cmdline without refresh. The file changed; the ESPs did not; the node boots with the old parameters.
--next-boot used as a rollback. It reverts at the following reboot, weeks later, during something unrelated.
A pin with no review date. The node stops taking kernel security fixes while apt reports success.
Never verifying status after a disk operation. It is one read-only command and it is the only evidence the work took.
Assuming SSH will be available. It will not be. Out-of-band console access is the prerequisite.

Key takeaways

proxmox-boot-tool manages two things: the set of synced ESPs, and which kernels land on them.
status is the first command in any boot investigation. Count the ESPs, compare the version lists, note uefi versus grub.
A replaced disk needs sgdisk -R, sgdisk -G, format, init, then status to verify. zpool replace alone leaves one bootable disk.
kernel pin is a persistent rollback; kernel pin --next-boot is a single-boot test. Do not use the second as the first.
A pin needs a ticket with a review date, because nothing else will remind you.
refresh is what makes /etc/kernel/cmdline take effect on systemd-boot; clean removes ESPs whose disks are gone.
The ESP is 512 MB. kernel remove is the answer when it fills.

Knowledge check

Why this matters

Boot redundancy is the clearest example in Proxmox of a control that looks present and is not.

A two-disk ZFS mirror gives every visible signal of redundancy. zpool status says ONLINE. The GUI shows a healthy pool. Monitoring is green. Pull a disk in a test and the node keeps running, which appears to confirm the whole thing. None of that says anything about whether the machine can be started again, because ZFS does not manage the ESP and nothing checks it.

The gap opens quietly. It opens the first time a disk is replaced by somebody following the ZFS procedure correctly and completely, because the ZFS procedure has nothing to say about partition 2. It stays open because it has no symptom. And it closes only during an unplanned reboot of a node that has since lost its other disk — which is to say, during the exact scenario the mirror was bought for.

proxmox-boot-tool status is a read-only command that takes a second and answers the question directly. Running it after every disk operation, and collecting it as a fact in whatever inventory you keep, converts boot redundancy from an assumption into something you can point at.

The pinning half of the tool matters for a related reason: it is the only thing that makes a kernel upgrade reversible without a rebuild. Upgrading one node, soaking it, and pinning back if it misbehaves turns the 6.x-to-7.0 transition from a fleet-wide gamble into a bounded experiment — provided the pin gets an expiry, because a node pinned and forgotten has swapped an availability risk for a security one.