Hardware virtualisation, IOMMU, and CPU features · Proxmox VE

Why this matters in production

A surprising fraction of “Proxmox won’t install” and “PCI passthrough doesn’t work” tickets trace back to firmware settings that were never enabled. This lesson shows how to verify the features Proxmox depends on, and what the symptoms of their absence look like.

Mental model

Proxmox needs three classes of hardware feature:

CPU virtualisation extensions — Intel VT-x (also called vmx) or AMD-V (svm). Without these, KVM cannot run a guest at full speed; QEMU falls back to TCG (pure emulation) and is dramatically slower.
IOMMU — Intel VT-d (intel-iommu) or AMD-Vi (amd-iommu). Without this, the host cannot safely give a guest direct access to a PCI device (GPU, NVMe controller, NIC).
Optional but recommended — SR-IOV (for NIC virtual functions), APICv, Posted Interrupts. These affect throughput and latency for high-performance networking and passthrough.

How to verify in the kernel

The firmware menu records what you asked for. The kernel records what you got, and they disagree often enough that only the second is worth trusting.

Read-only / Safethe four checks, in order of what they rule out— Read-only. The cpuinfo flag says the CPU supports it and firmware has not hidden it. /dev/kvm existing says the kernel module loaded. The dmesg lines say the IOMMU initialised. /proc/cmdline says what the kernel was actually asked to do, which is not always what /etc/default/grub contains.

grep -c -E 'vmx|svm' /proc/cpuinfo

ls -l /dev/kvm

dmesg | grep -iE 'DMAR|IOMMU|AMD-Vi' | head

cat /proc/cmdline

Read-only / Safewhat a working IOMMU looks like at boot— Illustrative, Intel. The 'DMAR: IOMMU enabled' line is the confirmation. On AMD the equivalent lines say AMD-Vi. Absence of any DMAR or AMD-Vi line at all means the IOMMU never initialised.

# dmesg | grep -iE 'DMAR|IOMMU'

[    0.008431] ACPI: DMAR 0x000000006C7E4000 0000A8 (v01 INTEL  EDK2 )
[    0.291204] DMAR: IOMMU enabled
[    0.412887] DMAR: Host address width 46
[    0.412889] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    0.412901] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466
[    1.884012] DMAR: Intel(R) Virtualization Technology for Directed I/O

Illustrative output

If /dev/kvm is missing, the most common causes are:

Hardware virtualisation disabled in firmware.
Kernel booted with kvm.enable_virt_at_load=0 (uncommon).
The kernel does not have the KVM module loaded (modprobe kvm).
Running inside a guest without nested virtualisation enabled.

IOMMU groups: the check that actually predicts passthrough

Enabling VT-d is necessary and not sufficient. What decides whether you can pass a particular device to a guest is which IOMMU group it lands in — and that is determined by the motherboard’s PCIe topology, not by any setting.

Read-only / Safelist every IOMMU group and what is in it— Read-only. A count of zero groups means the IOMMU is not active regardless of what the firmware menu says. A group containing several unrelated devices means all of them must be passed to the same guest, or none can be.

ls /sys/kernel/iommu_groups/ | wc -l

for G in /sys/kernel/iommu_groups/*/devices/*; do
N=${G#*/iommu_groups/}; N=${N%%/*}
printf 'group %s: ' "$N"
lspci -nns "${G##*/}"
done | sort -V

Read-only / Safea group that makes passthrough impossible as it stands— Illustrative. Group 14 holds the GPU, its audio function, and an unrelated network controller. Passing the GPU means passing the NIC too — and that NIC may be the host's management interface.

# (the loop above)

group 13: 00:1f.3 Audio device [0403]: Intel Corporation HD Audio [8086:a348]
group 14: 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation [10de:2484]
group 14: 01:00.1 Audio device [0403]: NVIDIA Corporation [10de:228b]
group 14: 02:00.0 Ethernet controller [0200]: Realtek RTL8125 [10ec:8125]
group 15: 03:00.0 Non-Volatile memory controller [0108]: Samsung [144d:a80a]

Illustrative output

The IOMMU group is the reason passthrough works on one machine and not another

Two hosts with identical CPUs, identical BIOS settings and identical Proxmox versions can behave completely differently here, and the usual conclusion — that something is misconfigured — is wrong.

An IOMMU group is the smallest set of devices the hardware can isolate from one another. Everything in a group must be assigned to the same guest, because the platform cannot guarantee that one device in the group cannot reach another’s memory. That grouping comes from the PCIe root complex layout and from whether the platform implements ACS (Access Control Services), which is what lets the chipset report finer-grained isolation.

Server boards generally implement ACS properly and give you one device per group. Consumer boards frequently route several slots through a single PCIe switch with no ACS, so the GPU, an onboard NIC and a USB controller all land together.

Why the first diagnosis is wrong. The symptom is a VM that will not start, or that starts and takes the host’s network with it, and the instinct is to check intel_iommu=on again — which is already correct. The evidence that identifies it is the group listing above, and it takes one command.

What you can actually do. Move the card to a different physical slot, which frequently lands it in a different group and costs nothing to try. Pass the whole group, if everything in it is genuinely disposable. Or choose hardware with proper ACS support, which is one of the concrete reasons the hardware lesson recommends server boards for passthrough work.

The “ACS override” kernel patch circulates as a fix. It does not improve isolation; it tells the kernel to assume isolation the hardware has not claimed. On a host running untrusted guests that is a security decision, not a workaround, and it should be made deliberately if at all.

Firmware checklist

Every server-class motherboard Proxmox runs on should have these settings:

Setting	Where	Required for
Intel VT-x / AMD-V	CPU configuration	All KVM VMs
Intel VT-d / AMD-Vi	Chipset / Northbridge	PCI/GPU passthrough
SR-IOV	PCIe configuration	NIC virtual functions
Above 4G Decoding	PCIe configuration	Large-memory GPUs
ACS	PCIe configuration	Finer-grained IOMMU groups

The exact wording varies between vendors (Dell iDRAC, HPE iLO, Supermicro IPMI, Lenovo XClarity). Searching the firmware manual for “virtualisation” or “IOMMU” usually surfaces the right option. ii-planning-firmware-bios treats these as pre-install decisions, which is where they belong.

Kernel command line

For PCI passthrough, the kernel command line on each Proxmox node should include:

intel_iommu=on iommu=pt

or for AMD:

amd_iommu=on iommu=pt

iommu=pt enables pass-through mode, in which the IOMMU only translates for devices assigned to guests and leaves host-owned devices on a direct mapping — lower overhead than translating everything.

Where you put that line depends on the bootloader, and Proxmox nodes differ. Editing the wrong file produces a change that looks applied and is not, which is why the first step is always to ask the node.

Read-only / Safeask the node which bootloader it uses— Read-only. proxmox-boot-tool status is authoritative. Do not infer it from the Proxmox version: it depends on boot mode, root filesystem and when the node was installed.

proxmox-boot-tool status

[ -d /sys/firmware/efi ] && echo "UEFI" || echo "legacy BIOS"

cat /proc/cmdline

Service impact possibleapply the change, on whichever bootloader this node has— Both require a reboot to take effect, so drain the node first. Verify afterwards from /proc/cmdline, which reports what the kernel actually received, rather than from the file you edited.

# GRUB installs: edit GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, then
update-grub

# systemd-boot installs: append to the single line in /etc/kernel/cmdline, then
proxmox-boot-tool refresh

# after the reboot, on either:
cat /proc/cmdline
dmesg | grep -iE 'DMAR|AMD-Vi' | head -3

CPU models for VMs

Model	What it exposes	When to use
`host`	All host CPU features verbatim	Maximum performance; migration only between identical hosts
`x86-64-v2-AES`	Westmere-era baseline plus AES	The UI default for new VMs. Good portability and performance
`x86-64-v2` / `v3` / `v4`	Specified x86-64 micro-architecture levels	Raise the level when the workload benefits and the fleet supports it
`kvm64`	The x86-64-v1 baseline	Maximum compatibility, lowest performance. Legacy
Named Intel/AMD models	Specific CPU vendor lines	Rarely needed

The documentation is explicit about the cost of host: “If you want to do a live migration of VMs between different hosts, your VM might end up on a new system with a different CPU type or a different microcode version. If the CPU flags passed to the guest are missing, the QEMU process will stop.”

That is a stop, not a degradation. It is the reason host belongs only on guests that will never migrate, or on clusters whose nodes are genuinely identical — including their microcode revision, which a firmware update can change on one node and not another.

Read-only / Safewhat CPU models does this node offer, and what is each guest using?— Read-only. The capabilities endpoint lists every CPU model this node can offer, including any custom ones. Custom models are backed by /etc/pve/virtual-guest/cpu-models.conf and are referenced from a VM configuration as custom-NAME; that file does not exist on a node with no custom models, which is the normal case.

pvesh get /nodes/localhost/capabilities/qemu/cpu --output-format json | jq -r '.[].name'

cat /etc/pve/virtual-guest/cpu-models.conf 2>/dev/null || echo "no custom CPU models defined"

for VMID in $(qm list | awk 'NR>1 {print $1}'); do
printf '%s ' "$VMID"; qm config "$VMID" | grep -E '^cpu:' || echo "cpu: (default)"
done

Production considerations

Under the hood

What VT-x and the IOMMU actually do, and why each has a cost worth knowing.

VT-x and AMD-V add a CPU privilege axis. Before them, a hypervisor had to trap and emulate every privileged instruction a guest kernel executed, because a guest kernel expects to run at ring 0 and cannot be allowed to. The extensions add a separate mode — VMX root and non-root — so the guest kernel really does run at ring 0, in a context the CPU knows is a guest. A VM exit transfers control back to the hypervisor only when something genuinely requires it: an I/O port access, a page fault the host must resolve, an interrupt.

That is why the presence of the flag changes performance by an order of magnitude rather than a few percent, and why its absence is unmistakable — a guest under TCG emulation does not run 20 % slower, it runs at a small fraction of native speed.

The counting unit that matters for tuning is the VM exit. Each one is a context switch costing hundreds to low thousands of cycles. Most virtualisation performance work — virtio drivers instead of emulated hardware, APICv for interrupt delivery, posted interrupts — is ultimately about removing exits from a hot path.

The second address translation is where hugepages come in. A guest virtual address becomes a guest physical address through the guest’s own page tables, and that becomes a host physical address through the EPT (Intel) or NPT (AMD). Both walks happen in hardware, but a full walk with 4 KiB pages at both levels is up to 24 memory references for one uncached translation. This is the mechanism that makes transparent hugepages worth having on a KVM host: 2 MiB host pages remove a level from the host-side walk.

The IOMMU is the same idea applied to devices. A PCI device performing DMA writes to a physical address it was given. Without an IOMMU there is nothing stopping a device from writing to any physical address at all — which is fine when the operating system programs the device and catastrophic when a guest does, because the guest could program it to write into the host’s memory or another guest’s.

The IOMMU sits between the device and memory and translates device addresses through tables the host controls. A guest given a device programs it with guest physical addresses; the IOMMU maps those to the host physical pages actually backing that guest, and a device programmed to reach anywhere else simply faults.

Two consequences follow directly.

The isolation guarantee is per group, not per device, because the platform can only enforce what its topology allows it to distinguish. Devices behind a switch that does not implement ACS can talk to each other without the transaction reaching the root complex, so the IOMMU never sees it and cannot prevent it. Hence the grouping rules earlier in this lesson: they are a statement about what the hardware can actually promise.

And iommu=pt is a performance choice with a scope. It leaves host-owned devices on an identity mapping, avoiding translation overhead for them, while devices assigned to guests are still fully translated. It does not weaken guest isolation, which is the common concern about it.

What happens if one node in the cluster has different CPU features from the others?

You get a cluster where live migration works in one direction and not the other, and the failure is a stopped guest rather than a rejected request.

The asymmetry is the confusing part. A guest started on a node with a rich feature set — cpu: host on a newer CPU — has been told it has instructions the older node does not provide. Migrating it there does not degrade it; QEMU stops, because the guest may have already compiled a code path using an instruction that is about to disappear. The reverse direction works fine. So the cluster appears to work until the day HA chooses the wrong target, or until someone drains the newer node.

A microcode update can create this without any hardware change. Microcode revisions can alter which CPU feature bits are advertised. A cluster of physically identical nodes, patched on a rolling schedule, spends the patch window as a mixed-feature cluster — which is one of the less obvious reasons to keep firmware uniform rather than merely current.

How to detect it before it bites. Compare the flags across nodes directly:

pvesh get /cluster/resources --type node --output-format json | jq -r '.[].node'

then on each node, grep '^flags' /proc/cpuinfo | head -1 and diff the results. Any difference is a migration constraint. Doing this once after every firmware or kernel roll is cheap; discovering it during a failover is not.

The fixes, in order of preference. Set an explicit cpu: model that every node supports — x86-64-v2-AES is the UI default for good reason, and x86-64-v3 is safe on any reasonably modern fleet. This costs a few percent against host and buys unconditional migration.

Where a guest genuinely needs host for a specific instruction set, fence it with an HA group restricted to nodes that provide it, so neither HA nor an operator can move it somewhere it cannot run.

And when adding a node to an existing cluster, check its feature set against the others before it takes any guests. A node that is newer is as much of a problem as one that is older, because guests started there acquire features the rest of the cluster lacks.

Common mistakes

Verifying from the firmware menu rather than the kernel. The menu is a request; /proc/cpuinfo, /dev/kvm and dmesg are the result.
Checking intel_iommu=on is set and stopping there. An empty /sys/kernel/iommu_groups means it did not take, and a group containing several devices means passthrough is blocked for a different reason entirely.
Editing /etc/default/grub on a systemd-boot node. Nothing reads it. proxmox-boot-tool status says which applies.
Using cpu: host on a heterogeneous cluster. Migration to a node with fewer features stops the QEMU process rather than degrading it.
Assuming identical hardware means identical CPU features. A rolling microcode update makes a uniform cluster temporarily mixed.
Buying consumer motherboards for passthrough. Many lack VT-d, and many that have it lack ACS, which puts unrelated devices in one IOMMU group.
Forgetting “Above 4G Decoding”, which causes large GPUs to fall back to legacy addressing and lose performance.
Looking for custom CPU models in /etc/pve/cpu-models.conf. The path is /etc/pve/virtual-guest/cpu-models.conf, and it does not exist unless you created it.

Key takeaways

Proxmox needs VT-x/AMD-V (always) and VT-d/AMD-Vi (for passthrough).
Verify from the kernel: /proc/cpuinfo flags, /dev/kvm, dmesg | grep -iE 'DMAR|AMD-Vi', and /proc/cmdline.
A non-empty /sys/kernel/iommu_groups is the real test that IOMMU is active.
The IOMMU group is the unit of isolation. Everything in a group goes to the same guest, and the grouping comes from PCIe topology and ACS support, not from configuration.
The kernel command line lives in /etc/default/grub plus update-grub, or /etc/kernel/cmdline plus proxmox-boot-tool refresh. Ask proxmox-boot-tool status which.
x86-64-v2-AES is the UI default for new VMs. host gives maximum performance and stops the QEMU process on migration to a node with fewer features.
Custom CPU models live in /etc/pve/virtual-guest/cpu-models.conf and are referenced as custom-<name>.
Keep the CPU model uniform across a cluster so migration either always works or never does.