PCI passthrough and isolated NICs · OPNsense

Virtual NICs (virtio) on Proxmox deliver most of the performance a small or medium firewall needs. But at 10 Gbit/s and beyond, or when the operator wants strict isolation between the hypervisor and the firewall’s traffic, the right answer is PCI passthrough: give the OPNsense VM direct ownership of a physical NIC. This lesson covers what PCI passthrough is, what IOMMU groups mean, how to configure it on Proxmox, and the trade-offs versus virtio.

What PCI passthrough (VT-d) is

A NIC bound to a virtual machine via PCI passthrough is entirely controlled by that VM. The host operating system does not see the device; the guest’s driver talks to the NIC directly via DMA. The CPU’s IOMMU (Intel VT-d or AMD-Vi) enforces memory isolation: the NIC can only DMA into memory pages the hypervisor has mapped for the guest.

The result:

No hypervisor on the data path. Every packet goes NIC → guest memory → guest driver → guest kernel, with no involvement from the Proxmox host networking stack.
No shared ring buffers or vhost-net translation. Native driver performance.
Strong isolation. The host cannot inspect, modify, or even observe packets flowing through the passthrough NIC. This is exactly what a firewall operator wants.

The cost:

The NIC is dedicated to one VM. The host cannot use it for anything else (no other VMs can be attached to it).
Live migration becomes harder. When the VM moves to a different host, the passthrough NIC must be available on the target; Proxmox supports this but the constraints are tighter than for virtio NICs.
IOMMU group constraints. The passthrough must include every device in the IOMMU group, not just the NIC.

Read-only / SafeIOMMU groups— The list of IOMMU groups on the host. Each group is a set of devices the IOMMU treats as a single isolation unit — every device in the group must be passed through together. Group 5 has three devices (a multifunction NIC); group 7 has three devices (likely the chipset and a USB controller). For a single-function NIC like group 3 (0000:01:00.0), passthrough is straightforward. For a multifunction NIC like group 5, the operator must pass through all three functions or none.

root@pve:~# find /sys/kernel/iommu_groups/ -type l | sort -V

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/3/devices/0000:01:00.0
/sys/kernel/iommu_groups/4/devices/0000:01:00.1
/sys/kernel/iommu_groups/5/devices/0000:02:00.0
/sys/kernel/iommu_groups/5/devices/0000:02:00.1
/sys/kernel/iommu_groups/5/devices/0000:02:01.0
/sys/kernel/iommu_groups/6/devices/0000:03:00.0
/sys/kernel/iommu_groups/7/devices/0000:04:00.0
/sys/kernel/iommu_groups/7/devices/0000:04:00.1
/sys/kernel/iommu_groups/7/devices/0000:04:01.0

Illustrative output

IOMMU groups explained

The IOMMU partitions PCI devices into groups. Every device in a group shares the same IOMMU isolation boundary — the CPU cannot grant one device in the group DMA access to memory without granting it to all of them. To pass through a device, the operator must pass through every device in its group.

On modern server hardware with proper ACS (Access Control Services) support, each PCI device is in its own group. On consumer hardware (and older servers), groups can be larger, containing multiple unrelated devices that must be passed through together. A group that contains the NIC you want plus the SATA controller and the USB controller is a passthrough deal-breaker — you cannot pass through the NIC without also passing through the SATA controller, which the host needs.

To verify passthrough is possible:

dmesg | grep -e DMAR -e IOMMU    # Intel: look for "DMAR: IOMMU enabled"
dmesg | grep -e AMD-Vi            # AMD: look for "AMD-Vi: IOMMU performance counters"
find /sys/kernel/iommu_groups/ -type l   # enumerate groups

If find shows groups with many unrelated devices, the host may need a BIOS update (for ACS) or a different CPU/chipset combination.

Configuring PCI passthrough on Proxmox

Three steps on the Proxmox host:

Enable IOMMU in the kernel command line. Edit /etc/default/grub:
- Intel: GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
- AMD: GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
Then update-grub and reboot.
Load the VFIO modules. In /etc/modules, add:
```
vfio
vfio_iommu_type1
vfio_pci
```
Reboot. Verify with dmesg | grep -i vfio — look for “VFIO - User Level meta-driver” and the bound device names.
Bind the NIC to VFIO. Either via the Proxmox GUI (VM → Hardware → Add → PCI Device → select the NIC) or via /etc/pve/qemu-server/<vmid>.conf:
```
hostpci0: 01:00.0,pcie=1
hostpci1: 01:00.1,pcie=1
```
The 01:00.0 is the PCI address from lspci or find /sys/kernel/iommu_groups/. The pcie=1 flag enables PCIe passthrough (vs legacy PCI).

After reboot, the NIC appears inside the guest as if it were a physical NIC attached directly. In OPNsense, pciconf -lv shows the device bound to the igb (Intel) or other driver without any virtio layer.

Performance: virtio multi-queue vs PCI passthrough

Both approaches can deliver line-rate throughput on 10 Gbit/s NICs. The differences:

Aspect	virtio multi-queue	PCI passthrough
Throughput	8–10 Gbit/s with 4+ vCPUs and queues	Native — 10 Gbit/s+, limited by NIC and CPU
CPU cost	Hypervisor context switches on each packet	None on the host (guest-only)
Latency	~10–50 µs additional per packet	Native NIC latency
Isolation	Host can inspect traffic (theoretically)	Host cannot observe or modify traffic
Live migration	Easy	Possible but constrained by target host hardware
Complexity	Low	Moderate (IOMMU groups, VFIO setup)

For most deployments up to 2.5 Gbit/s, virtio multi-queue is sufficient and simpler. For 10 Gbit/s line-rate with low latency, or when strict isolation is required (a firewall that must process sensitive traffic), PCI passthrough is the right answer.

Troubleshooting passthrough

The most common passthrough failures:

“Device is in use by another driver.” The host kernel driver is bound to the NIC. Blacklist the driver via /etc/modprobe.d/blacklist.conf (e.g. blacklist igb for Intel igb NICs).
“IOMMU not enabled.” The kernel command line change did not take effect. Verify with dmesg | grep -e DMAR and check /proc/cmdline.
“Group contains too many devices.” The hardware does not support ACS or the BIOS does not enable it. Update the BIOS or use a different slot/NIC.
Guest boots but does not see the NIC. The PCI address in the VM config is wrong. Verify with lspci -nn -d <vendor:device> on the host and compare to the address in /etc/pve/qemu-server/<vmid>.conf.

Production patterns

Three patterns cover most decisions:

Home / small office (1 Gbit/s): virtio multi-queue. Passthrough is overkill.
Small business (1–2.5 Gbit/s): virtio multi-queue with queues=4 or 8. Passthrough is optional.
Datacenter / ISP / 10 Gbit/s+: PCI passthrough with an Intel X710 or Chelsio T5. Isolation and performance both favour passthrough.

Summary

PCI passthrough gives the guest direct ownership of a NIC, bypassing the hypervisor’s data path entirely.
IOMMU (VT-d / AMD-Vi) enforces memory isolation; the NIC can only DMA into guest memory.
Verify IOMMU groups before buying hardware; multi-device groups are a passthrough deal-breaker.
Configuration: enable IOMMU in kernel command line, load VFIO modules, bind the NIC, add to VM config.
For most deployments, virtio multi-queue is enough. PCI passthrough is for 10 Gbit/s+ or strict isolation.