Firmware lifecycle - the layer your package manager cannot see · Linux

Ask a well-run fleet what version of openssl it is running and you will get an exact answer in seconds. Ask the same fleet what version of NIC firmware is on its hosts and the honest answer is usually that nobody knows.

That gap matters because firmware is where a specific and frustrating class of production bug lives. An NVMe drive that drops off the bus under sustained write load, a NIC that corrupts checksums when a particular offload is enabled, a RAID controller that stalls during a rebuild, a CPU errata that needs a microcode update to fix — none of these are visible to apt list --upgradable, and all of them present as “the application is broken” for as long as nobody thinks to look below the operating system.

What counts as firmware on a server

Component	Typical update path	Reboot needed
System BIOS / UEFI	fwupd (UEFI capsule) or vendor bundle	Yes
CPU microcode	OS package, loaded every boot, or BIOS	Yes (or a fresh boot)
BMC	Vendor tool or the BMC web interface	BMC restarts; host keeps running
RAID controller	Vendor tool (`storcli`, `ssacli`)	Usually
NVMe and SAS drives	`nvme fw-download`/`fw-commit`, or vendor bundle	Usually
NIC	fwupd or vendor tool	Usually
Backplane, PSU, expander	Vendor bundle	Sometimes

The column that surprises people is the third one. Firmware updates are not uniformly “reboot the box”; a BMC update restarts only the BMC, and a drive firmware commit may take effect on the next drive reset rather than the next host boot. Knowing which is which is what lets you batch the disruptive ones into a single window.

Auditing what you have

Start with the platform identity, which you already have from the inventory lesson:

sudo dmidecode -t bios      # BIOS vendor, version, release date
sudo dmidecode -t system    # product name and service tag

Then ask fwupd, which enumerates every device that has a firmware version it can read — whether or not it can update it:

Read-only / Safefwupd audit— get-devices and get-updates change nothing; refresh only downloads metadata. get-devices is the useful one even on hardware fwupd cannot update, because it still reports versions for devices it can read. The Update Description is not decoration - it is the only place you learn whether an update is a security fix, a stability fix for a configuration you do not have, or a feature you do not want.

$ fwupdmgr get-devices
fwupdmgr refresh
fwupdmgr get-updates

ACME R740xd
│
├─System Firmware:
│     Device ID:          8b6b5c1e...
│     Current version:    2.19.0
│     Vendor:             ACME (DMI:ACME)
│     Update State:       Success
│
├─Embedded Controller:
│     Current version:    1.4.2
│
└─SSD 960GB:
    Current version:    QDV1RE14
    Vendor:             ACME (NVME:0x1234)

Successfully downloaded new metadata: 3 local devices supported

ACME R740xd
│
└─System Firmware:
    Device ID:          8b6b5c1e...
    Current version:    2.19.0
    Update Version:     2.22.1
    Update Remote ID:   lvfs
    Update Checksum:    SHA256(9c1e...)
    Update Description: Fixes a memory training failure on
                        hosts populated with 16 DIMMs, and
                        includes microcode for CVE-2025-XXXXX.

Illustrative output

fwupd draws its metadata from the Linux Vendor Firmware Service, which covers a growing share of enterprise hardware. Where it does not, the vendor bundle is the path — a Dell System Update repository, an HPE Service Pack, a Lenovo update utility — and the discipline in this lesson applies unchanged.

Applying an update

# Update everything fwupd knows about, with confirmation prompts
sudo fwupdmgr update

# Or one device at a time, which is what you want in production.
# Take the ID from the get-devices output above.
DEVICE_ID=8b6b5c1e
sudo fwupdmgr install "$DEVICE_ID"

# For updates staged into the ESP and applied on the next boot
sudo fwupdmgr get-history

UEFI capsule updates do not apply when the command returns. fwupd writes the capsule to the EFI system partition and sets a flag; the firmware picks it up and flashes during the next boot, before any operating system runs. So the sequence is install, then reboot, then verify — and the verification is mandatory, because a capsule that the firmware rejected leaves you on the old version with a successful-looking command in your history.

# After the reboot
sudo dmidecode -t bios | grep -i version
fwupdmgr get-history            # did the capsule report Success or Failed?

Microcode is a special case

CPU microcode arrives two ways and the difference is operationally significant. The BIOS carries a copy, applied at POST. The distribution also ships one — intel-microcode or amd64-microcode — which the kernel loads from the initramfs very early in boot, overriding the BIOS copy when it is newer.

# What did the kernel load this boot?
sudo journalctl -k --grep='microcode'
grep -m1 microcode /proc/cpuinfo

# After installing the package, the initramfs must be rebuilt
sudo apt install intel-microcode
sudo update-initramfs -u

The OS-supplied microcode is re-applied on every boot and never persists in the CPU, so it is genuinely reversible: remove the package, rebuild the initramfs, reboot. That makes it the one firmware-adjacent change in this lesson with a real rollback, and it is why security microcode is normally shipped through the package manager rather than through a BIOS campaign.

Firmware version drift is a defect

Two nodes with identical package sets, identical configuration and different NIC firmware are not identical nodes. They will diverge under load, and the divergence will look like a load balancer problem or a noisy-neighbour problem for as long as nobody compares firmware.

Capture firmware versions in the same inventory export as everything else, and diff across the fleet:

# On the control node
ansible -i /etc/ansible/prod webservers -b \
  -m ansible.builtin.command \
  -a "dmidecode -s bios-version" \
  | sort | uniq -c

Any group of hosts that is supposed to be interchangeable and reports more than one BIOS version has a finding. It is not always worth fixing immediately, but it should be recorded, not discovered during an incident.

Firmware rollback is usually not available, whatever the vendor tool implies

Package rollback is imperfect but it exists. Firmware rollback frequently does not, and the reasons are deliberate:

Many platforms enforce a minimum version, so a downgrade is rejected outright to prevent an attacker reintroducing a patched vulnerability.
Some use one-time-programmable anti-rollback fuses, which make the restriction physical and permanent.
Even where a downgrade is technically accepted, a failed flash can leave the device unable to initialise, and the recovery is a service call rather than a command.

fwupdmgr downgrade exists and works on the subset of devices that permit it. Do not plan around it.

The practical consequences shape the whole change:

Test on hardware you can afford to lose before the fleet. A single spare node of the same model, or a decommissioned one, is the cheapest insurance in this lesson.
Read the release notes properly. “Fixes a memory training failure on 16-DIMM configurations” is not a reason to flash a host with 8 DIMMs.
Update because you have a reason — a CVE, a bug you have actually hit, a vendor requirement for a support case — not because a newer version exists.

A BMC firmware update can remove the access path you would use to recover

BMC firmware is the one to think hardest about, because the BMC is what you would reach for if the update went wrong.

Two specific hazards. First, the BMC restarts during the update, so any SOL session drops and any fence agent pointed at it fails for the duration — on a Pacemaker cluster that is a window in which no node can be fenced, exactly as covered in linux-oob-architecture. Second, some BMC updates reset configuration to factory defaults: network settings, user accounts, SOL enablement and cipher suite configuration all revert, and the host that was reachable this morning has a BMC on DHCP with the default credentials.

So the order is:

1. Export the BMC configuration where the platform supports it.
2. Record the network settings, user list and SOL settings by hand
   in any case (ipmitool lan print 1, user list 1, sol info).
3. Confirm you have a second access path: the host is up and
   reachable in-band, or somebody can reach the rack.
4. Put the cluster node in standby first. A BMC restart during
   the update means it cannot be fenced.
5. Update. Then re-verify every setting from step 2, and log in
   with the break-glass credential, not just the daily one.

Step 5 is the one that catches a factory reset, and the break-glass credential is what proves the account survived.

Firmware is a change like any other

Everything the rolling-maintenance part says applies here, and more strictly, because there is no rollback:

One node at a time. A firmware bug that bricks a NIC will brick every NIC you flash in that batch.
Health-gate between nodes rather than working to a timer.
Hold the first node longer than feels necessary. Storage and network firmware bugs are load-dependent, and a node that has not carried peak traffic has not been tested.
Record the versions before and after, per host, in the change record.