Firmware, BIOS and microcode in a maintenance window

Below the kernel there is another operating system on every device in the server, and most Proxmox maintenance procedures stop at the kernel. The gap shows up in a recognisable way: a NIC that drops packets under a specific load pattern, an NVMe drive whose latency climbs after months of uptime, a CPU errata that only matters once you run untrusted guest code — which is the definition of a hypervisor.

The layer is also where a new kernel most often exposes an old problem. A firmware bug the 6.x kernel worked around can become a visible fault on 7.0, which makes firmware an active concern during the PVE 9.2 kernel transition rather than a background one.

Five things, and they are not the same

Two distinctions carry most of the practical weight.

Microcode is not persistent. It is re-applied on every boot from the initramfs. That is why the fix for a CPU errata is a package, why it is undone by removing the package, and why it needs the initramfs to be rebuilt.

The BMC updates independently of the host. It has its own operating system, its own network stack and its own vulnerabilities, and it is running right now on every node whether the host is powered on or not. It is also not in apt, so nothing in your patch process touches it unless you built that separately.

CPU microcode

This is the one Proxmox operators most often miss, and it is the one most relevant to a hypervisor, because a large share of CPU errata are speculative-execution issues whose exploitation model is precisely “untrusted code running on the same core”.

The repository prerequisite

Microcode packages are in the non-free-firmware component. A node whose debian.sources lists only main cannot install them, and there is no warning — apt install intel-microcode simply reports that the package does not exist.

# /etc/apt/sources.list.d/debian.sources
Types: deb deb-src
URIs: http://deb.debian.org/debian/
Suites: trixie trixie-updates
Components: main non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb deb-src
URIs: http://security.debian.org/debian-security/
Suites: trixie-security
Components: main non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Installing it

grep -m1 vendor_id /proc/cpuinfo
grep microcode /proc/cpuinfo | uniq

apt update

# Intel:
apt install intel-microcode

# AMD:
apt install amd64-microcode

Verifying it after the reboot

Installing is not applying. Two checks, and both are needed:

dmesg | grep -i microcode

grep microcode /proc/cpuinfo | uniq

# dmesg | grep -i microcode

[    0.000000] microcode: updated early to revision 0x2b000603, date = 2026-03-11
[    1.204118] microcode: Current revision: 0x2b000603
[    1.204120] microcode: Microcode Update Driver: v2.2.

If the revision has not moved, the usual causes are that the initramfs was not rebuilt, that the node has not actually rebooted, or that the BIOS already carries a newer revision than the package — the last being a legitimate no-op rather than a failure.

When microcode is the problem

Microcode updates occasionally cause regressions. Two documented levers:

Disable the early load for one boot by appending dis_ucode_ldr to the kernel line in the boot menu — press e in GRUB, add it to the linux line, and boot with Ctrl-X. That is a diagnostic, not a fix: it tells you whether microcode is implicated, and it does not persist.

Downgrade rather than remove. apt install intel-microcode=3.20230512.1 with the version you want, followed by apt-mark hold. Removing the package entirely leaves the node without any microcode updates at all, including the ones that were working.

Runtime device firmware

pve-firmware is installed by default. From the Proxmox documentation, it “stores firmware on the Proxmox VE operating system and will pass it to a device if its persisted firmware is less recent.”

That is the useful property: the device loads it at every boot, so an ordinary apt update can deliver a device fix without touching the device’s flash — and it is undone by rolling the package back, which makes it far less risky than a persistent update.

dmesg | grep -iE 'firmware|fw ver'
dmesg | grep -i 'firmware.*fail'

dpkg -l pve-firmware linux-firmware 2>/dev/null

A device whose firmware request failed usually still works, in a reduced mode — a NIC without offload support, a wireless device without the regulatory database. It is worth grepping for after any kernel change.

Persistent firmware: fwupdmgr and vendor tools

Proxmox ships its own fwupd package with Secure Boot support, requiring UEFI systems manufactured after 2014.

fwupdmgr refresh
fwupdmgr get-devices
fwupdmgr get-updates

fwupdmgr update

fwupdmgr get-history

fwupdmgr coverage on server hardware is partial. For the devices that matter most on a hypervisor you will usually be using the vendor’s own path:

The Proxmox documentation notes the convenience of Dell’s Lifecycle Manager and HPE’s Service Packs specifically — a single vendor bundle that updates BIOS, BMC, NIC and controller firmware together is far less error-prone than four independent updates, and it is what the vendor tests as a set.

Rolling it across a cluster

Firmware is a per-node change with a reboot, which makes it structurally the same as the patching procedure — with two additions.

NODE=$(hostname)

dmidecode -t bios -t system -t baseboard > "/root/fw-before-$NODE.txt"
grep microcode /proc/cpuinfo | uniq >> "/root/fw-before-$NODE.txt"
fwupdmgr get-devices >> "/root/fw-before-$NODE.txt" 2>/dev/null

cat "/root/fw-before-$NODE.txt"

NODE=$(hostname)
ha-manager crm-command node-maintenance enable "$NODE"
ha-manager status

Step 3 — apply the firmware by the vendor’s route, and let it reboot as many times as it needs to. Vendor bundles frequently reboot two or three times; do not intervene partway through, which is how a device ends up half-flashed.

NODE=$(hostname)

dmidecode -t bios -t system -t baseboard > "/root/fw-after-$NODE.txt"
grep microcode /proc/cpuinfo | uniq >> "/root/fw-after-$NODE.txt"
diff "/root/fw-before-$NODE.txt" "/root/fw-after-$NODE.txt"

dmesg | grep -iE 'DMAR|IOMMU|AMD-Vi' | head
lspci -nnk | grep -A3 -i 'ethernet'
zpool status

Step 5 — soak it. An hour under real load, not five minutes. Firmware faults are frequently load-dependent, which is the whole reason they were not caught by the vendor.

Step 6 — return it to service with ha-manager crm-command node-maintenance disable, and then move to the next node.

Common mistakes

• debian.sources without non-free-firmware. Microcode packages are not merely unavailable, they appear not to exist.
• Installing microcode and not verifying after the reboot. Installed is not applied; dmesg | grep microcode is the proof.
• Removing the microcode package to fix a regression. Downgrade and hold instead; a hypervisor without microcode has lost its only mitigation for a class of CPU errata.
• Treating a BIOS update as a firmware-only change. It resets device configuration, including IOMMU and HBA mode.
• Interrupting a vendor bundle mid-flash. Multiple reboots are normal; intervening leaves a device half-flashed.
• Flashing a whole cluster in one window. No rollback, no control group, no capacity.
• A five-minute soak. Firmware faults are load-dependent, which is why the vendor did not catch them.
• Never updating the BMC. It is not in apt, it is always on, and it is a full management-plane compromise if it falls.

Key takeaways

• Five distinct layers: CPU microcode, runtime device firmware, persistent device firmware, BIOS and BMC. Only the first two arrive through apt.
• Microcode is volatile and re-applied at every boot from the initramfs; it needs non-free-firmware and a reboot, and dmesg is the proof it loaded.
• pve-firmware supplies runtime firmware that the device loads at boot, which makes it low-risk and reversible.
• Persistent firmware, BIOS and BMC updates come from the vendor. Bundles are safer than piecemeal updates because they are tested as a set.
• A BIOS update can reset IOMMU, HBA mode, boot order and power settings. Export the configuration, and verify from a written list afterwards.
• Roll it one node at a time with HA maintenance mode, record before and after, soak under real load.
• Firmware is the only layer where a failed update can leave a machine no software can reach. The BMC is the recovery path, so check that it works first.

Knowledge check