Firmware, BIOS and controller configuration before install

The previous lesson chose the hardware. This one configures it, and the distinction matters more than it sounds: a node can be built from exactly the right components and still be unable to run ZFS safely, unable to pass through a GPU, and unable to meet a latency target — because of settings made in a firmware menu that nobody opened.

What makes this a planning lesson rather than an installation one is that most of these decisions are effectively permanent. Changing the storage controller mode on a node that already holds data means destroying the data. Changing the boot mode means reinstalling. These are choices you make once, before the node has anything on it, and the cost of getting them wrong is measured in rebuilds.

The one that cannot be undone: controller mode

If you take one thing from this lesson, take this one.

The Proxmox documentation is unambiguous: “Do not use ZFS on top of a hardware RAID controller which has its own cache management. ZFS needs to communicate directly with the disks.” The recommendation is to use HBA adapters, or LSI controllers flashed in IT mode.

This is widely read as a performance recommendation. It is not. It is a data-integrity requirement, and the reasoning is worth having in full because it is what lets you defend the position when someone points out that the RAID controller has a battery and is very fast.

Why RAID in front of ZFS breaks ZFS

ZFS’s central promise is end-to-end integrity: every block carries a checksum, every read verifies it, and when verification fails ZFS repairs the block from a redundant copy. Both halves matter, and a RAID controller breaks the second one.

Present eight disks to ZFS through a hardware RAID-6 volume and ZFS sees one device. You can create a pool on it, and that pool has no redundancy that ZFS controls. When ZFS reads a block and the checksum fails — which it will, eventually, because that is what checksums are for — it has detected corruption it cannot repair. The RAID controller believes the volume is healthy, because its own parity checks passed; the corruption happened above or beside the layer it protects.

You end up in the worst configuration available: full detection, no repair. A pool that reports permanent errors and names files you have lost.

The second failure is subtler and worse. ZFS relies on write ordering: a transaction group is committed only once its data is genuinely on stable media, and ZFS issues cache flushes to enforce that. A RAID controller with its own write cache may acknowledge a flush that has not reached the disks. Under normal operation nothing is wrong. After an unexpected power loss, the pool can come back having lost the last few transaction groups — and unlike a journalled filesystem, ZFS’s recovery assumes its ordering guarantees held.

Why Ceph needs it too

Ceph’s requirement is different but lands in the same place. An OSD is built on a whole raw block device, and Ceph’s failure domain is the OSD. Put four disks behind a RAID-5 volume and you get one OSD instead of four: the CRUSH map loses the granularity it needs to place replicas sensibly, a single disk failure now degrades an entire OSD rather than being handled inside it, and per-device SMART data disappears behind the controller.

You have replaced Ceph’s distributed redundancy with local redundancy underneath it, paid for both, and made the failure domain larger.

Determining what you actually have

This is worth checking on every node you inherit, because it is not visible from the outside and the answer is frequently not what the documentation for the estate says.

lspci -nnk | grep -A3 -iE 'raid|sas|sata|nvme'

lsblk -o NAME,SIZE,MODEL,SERIAL,ROTA,TRAN

cat /sys/class/scsi_host/host*/proc_name 2>/dev/null

# lsblk -o NAME,SIZE,MODEL,SERIAL,ROTA

NAME   SIZE MODEL           SERIAL      ROTA
sda    8.0T LOGICAL VOLUME              0
├─sda1 1007K
├─sda2  512M
└─sda3  8.0T

# lsblk -o NAME,SIZE,MODEL,SERIAL,ROTA

NAME   SIZE MODEL              SERIAL           ROTA
sda    1.8T MZ7L31T9HBLT-00A07 S6R1NE0T203841       0
sdb    1.8T MZ7L31T9HBLT-00A07 S6R1NE0T203847       0
sdc    1.8T MZ7L31T9HBLT-00A07 S6R1NE0T203852       0
sdd    1.8T MZ7L31T9HBLT-00A07 S6R1NE0T203858       0

smartctl -a /dev/sda | head -20

# if that fails and this works, the disk is behind a RAID controller:
smartctl -a -d megaraid,0 /dev/sda | head -20

# vendor tooling; names vary by controller family
# LSI/Broadcom:      sas2flash, sas3flash, storcli
# Dell PERC:         perccli, or HBA mode in the iDRAC storage menu
# HPE Smart Array:   ssacli, or HBA mode in the RBSU

storcli /c0 show
sas3flash -listall

Many modern controllers offer an HBA mode or pass-through mode switch in firmware that achieves the same result without cross-flashing. Prefer that where it exists: it is reversible, supported, and does not risk bricking the card. Check the vendor’s documentation before assuming your controller needs a flash.

IOMMU, SR-IOV and the passthrough settings

These are needed only by some workloads, and they cost nothing to enable, which makes the correct policy simple: enable them on every node whether or not you currently need them. A node without IOMMU cannot host a passthrough guest, and discovering that during a migration is worse than having enabled it up front.

grep -c -E 'vmx|svm' /proc/cpuinfo

dmesg | grep -iE 'DMAR|IOMMU|AMD-Vi' | head

ls /sys/kernel/iommu_groups/ | wc -l

lspci -vvv 2>/dev/null | grep -c 'Single Root I/O Virtualization'

A count of zero IOMMU groups means IOMMU is not active, regardless of what the firmware menu claims — the kernel command line may also need intel_iommu=on or amd_iommu=on, which is covered in i-foundations-hardware-virt.

Power profile and C-states

This is where planning meets the performance work in Part XVII, and the reason it belongs here is that the firmware setting is the durable one.

Server firmware ships with a power profile that is usually a balanced or efficiency-oriented default, and on Dell and HPE machines that profile also controls package C-state depth, memory patrol scrub behaviour, and turbo policy — settings the operating system either cannot reach or can only partially override.

The decision to make before install is not “performance or efficiency” in the abstract. It is:

• Latency-sensitive nodes — databases, brokers, anything where tail latency is the SLO — set the firmware profile to the performance-oriented option and limit package C-states. C6 exit latency is on the order of a hundred microseconds and applies to every wakeup.
• General-purpose nodes — leave the balanced default. The demand following governor handles it, and the power difference on a dense rack is real money.

The critical constraint is uniformity within a class. A cluster where one node has deep C-states limited and its neighbours do not has guests whose performance depends on where HA happened to place them, with no configuration difference to point at and nothing in any log. Decide per class of node, and keep the class identical.

Boot mode, and why it is a permanent decision

The firmware boot mode determines which bootloader Proxmox installs, and that determines where kernel command-line changes go for the rest of the node’s life.

proxmox-boot-tool status

[ -d /sys/firmware/efi ] && echo "booted in UEFI mode" || echo "booted in legacy BIOS mode"

dmidecode -t bios | grep -iE 'vendor|version|release'

Choose UEFI. Legacy BIOS boot is deprecated across the industry, Secure Boot requires it, and changing the mode later means reinstalling because the partition layout differs. If Secure Boot is a requirement in your environment, enable it now rather than later — Proxmox supports it, and enabling it on a node already running can leave a bootloader the firmware refuses.

The BMC baseline

The out-of-band controller is the thing that recovers a node that no software can reach, and it is configured before the operating system exists. Three items belong in the pre-install checklist.

Change the default credentials. Every vendor ships known defaults and they are the first thing scanned for. This is not a hypothetical risk; BMCs are a documented compromise path and a BMC gives complete control of the host, including its console and its power.

Put it on a management network that is not the guest network. A BMC reachable from a VM is a privilege escalation from guest to hardware.

Set NTP and a syslog target. A BMC with the wrong time produces hardware event logs you cannot correlate against anything, which removes most of their value at the moment you need them.

The pre-install checklist

Record these per node, before the installer runs, in something you can diff later. A BIOS update can reset every one of them.

NODE=$(hostname)
OUT="/root/firmware-baseline-$NODE.txt"

{
dmidecode -t bios -t system -t baseboard -t processor
echo "== virt =="
grep -c -E 'vmx|svm' /proc/cpuinfo
echo "== iommu groups =="
ls /sys/kernel/iommu_groups/ 2>/dev/null | wc -l
echo "== boot mode =="
[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
echo "== disks =="
lsblk -o NAME,SIZE,MODEL,SERIAL,ROTA,TRAN
echo "== controllers =="
lspci -nnk | grep -A3 -iE 'raid|sas|nvme'
} > "$OUT"

wc -l "$OUT"

Alongside it, keep a written list of the settings themselves — controller mode, IOMMU, SR-IOV, Above 4G, power profile, boot mode, Secure Boot, SMT. The command capture proves what the system saw; the written list records what you chose, and after a BIOS update you need both.

Common mistakes

• Accepting the controller’s factory default. Most Dell PERC and HPE Smart Array cards default to RAID mode, and the Proxmox installer will cheerfully put ZFS on a logical volume.
• Believing a battery-backed cache makes RAID safe for ZFS. It addresses write ordering and does nothing about ZFS’s inability to repair what it detects.
• Putting Ceph OSDs on RAID volumes. It enlarges the failure domain, hides per-device SMART and pays for redundancy twice.
• Verifying IOMMU from the firmware menu. The menu records a request; /sys/kernel/iommu_groups records the result.
• Leaving IOMMU off on nodes that “do not need passthrough”. It costs nothing and a guest may need to migrate there.
• Tuning the power profile on some nodes and not others. Guest performance then depends on HA placement, with nothing in any log.
• Choosing legacy BIOS boot. It rules out Secure Boot and changing it later means a reinstall.
• Leaving BMC defaults in place, or putting the BMC on a guest-facing network. A BMC is complete control of the host.
• Not recording the settings. A BIOS update resets them and nobody can say what they were.

Key takeaways

• Controller mode is the decision that cannot be undone with data in place. ZFS and Ceph need direct access to disks: HBA, or IT-mode firmware, or a vendor HBA/pass-through mode switch.
• RAID in front of ZFS is a data-integrity problem, not a performance one: ZFS still detects corruption and can no longer repair it.
• A protected write cache addresses ordering and does not restore redundancy.
• lsblk showing real model strings and serial numbers, and smartctl working without -d megaraid, are the tests for direct attachment.
• Enable VT-x/AMD-V, VT-d/AMD-Vi, SR-IOV and Above 4G Decoding on every node; verify from /sys/kernel/iommu_groups, not from the menu.
• Set the power profile per class of node and keep the class uniform, or HA placement becomes an unexplainable performance variable.
• Choose UEFI. It decides the bootloader, which decides where kernel parameters live, and changing it later means reinstalling.
• Change BMC defaults, isolate it from guest networks, and give it NTP and a syslog target.
• Capture a firmware baseline and keep a written list of the settings. A BIOS update resets them.

Knowledge check