GPU passthrough for Jellyfin transcoding and AI workloads

Why passthrough a GPU?

Two common home-lab reasons:

1. Jellyfin / Plex transcoding — CPU transcoding works but eats cycles and power. GPU transcoding (Intel QuickSync, NVENC) is 10× more efficient.
2. AI workloads — running local LLMs (Llama, Mistral), Stable Diffusion, or Whisper benefits hugely from a discrete GPU.

The catch: GPU passthrough requires IOMMU support and a few kernel parameters. Once configured, it’s reliable but the initial setup is fiddly.

Step 1: Verify hardware support

# Check CPU virtualisation support
egrep -c '(vmx|svm)' /proc/cpuinfo
# Should output 1 or more (number of cores with VT-x / AMD-V)

# Check IOMMU support
dmesg | grep -i -e DMAR -e IOMMU
# Look for "Intel-IOMMU" or "AMD-Vi" — if present, you're good

# On modern systems with kernel 5.x+, IOMMU is enabled but needs kernel args
# to actually be used

Step 2: Enable IOMMU in BIOS

Reboot, enter BIOS (Del / F2 / F10 depending on manufacturer). Look for:

• Intel: VT-d (under “Advanced”, “CPU Configuration”, or “Security”)
• AMD: AMD-Vi or IOMMU (often under “Advanced” → “CPU Configuration”)

Enable both VT-x (or AMD-V) and VT-d / AMD-Vi. Save and exit.

Step 3: Configure kernel command line

# Edit /etc/default/grub
nano /etc/default/grub

# Find the GRUB_CMDLINE_LINUX_DEFAULT line and add:
#   For Intel:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
#   For AMD:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

# Save and update grub
update-grub

# Add VFIO modules to load at boot
# (heredoc replaced)
echo "vfio" >> /etc/modules
echo "vfio_iommu_type1" >> /etc/modules
echo "vfio_pci" >> /etc/modules
echo "vfio_virqfd" >> /etc/modules
update-initramfs -u -k all

# Reboot
reboot

Step 4: Verify IOMMU is active

# After reboot
dmesg | grep -e DMAR -e IOMMU
# Should show "Intel-IOMMU: enabled" or similar

# List IOMMU groups
find /sys/kernel/iommu_groups/ -type l | sort -V
# Each PCI device belongs to an IOMMU group; the group must be exclusive
# for clean passthrough

Step 5: Find the GPU’s PCI addresses

# Find the iGPU (usually 00:02.0 on Intel)
lspci | grep -i vga
# 00:02.0 VGA compatible controller: Intel Corporation ... (iGPU)

# Find the GPU's audio function (usually 00:02.1 or similar)
lspci | grep -i -A1 vga
# The second device is the HDMI audio controller — pass it through too

# Get the full address with vendor:device IDs
lspci -n -s 00:02
# 00:02.0 0300: 8086:9a49 (rev 01)  arrow iGPU VGA
# 00:02.1 0380: 8086:9a49 (rev 01)  arrow iGPU audio

Step 6: Bind the GPU to vfio-pci

# Get the vendor:device IDs (e.g., 8086:9a49)
# Configure vfio to bind those devices

# (heredoc replaced)
echo "options vfio-pci ids=8086:9a49" >> /etc/modprobe.d/vfio.conf
# If the iGPU audio is separate, add its ID too
# options vfio-pci ids=8086:9a49,8086:9a50

update-initramfs -u -k all
reboot

Step 7: Pass through to the VM

qm set 120 --hostpci0 0000:00:02.0,pcie=1,x-vga=1

# pcie=1 uses PCIe passthrough (vs legacy PCI)
# x-vga=1 marks it as the primary VGA (so you see console output)
# If there's an audio function, pass it too:
qm set 120 --hostpci1 0000:00:02.1

# Use machine type q35 for PCIe
qm set 120 --machine pc-q35-9.0

# UEFI required
qm set 120 --bios ovmf --efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=1

# Start the VM
qm start 120

Step 8: Verify in the guest

# Inside the guest
lspci | grep -i vga
# Should show the Intel GPU

# For Linux guests, install drivers
# Intel: usually in-kernel, just verify
lsmod | grep i915

# Check for /dev/dri/renderD128 (render node)
ls -la /dev/dri/

# For Jellyfin:
# - Install jellyfin-ffmpeg (custom build with hardware support)
# - Enable hardware acceleration in the Jellyfin dashboard
#   Playback → Transcoding → Hardware acceleration: Intel QuickSync

Step 9: Verify hardware transcoding works

# In Jellyfin, play a 4K HEVC file. Without hardware acceleration,
# CPU usage spikes to 100% and the stream stutters. With QuickSync,
# CPU stays low and the stream is smooth.

# From the host, watch the GPU usage
intel_gpu_top  # or radeontop for AMD

# You should see Video engine utilisation rise during transcoding

AI workloads (discrete GPU)

For a discrete GPU (RTX 3060, RX 6700 XT, etc.), the process is similar but:

• The GPU typically has its own IOMMU group (good)
• It needs more power and cooling — a real GPU in a mini-PC may not fit physically
• For LLMs, you need at least 8 GB VRAM for smaller models (Llama 3 8B)
• For Stable Diffusion, 8-12 GB VRAM is the sweet spot

# Same setup steps; just point --hostpci0 at the discrete GPU
qm set 130 --hostpci0 0000:01:00.0,pcie=1,x-vga=1
qm set 130 --hostpci1 0000:01:00.1  # audio

# For LLMs (e.g., ollama)
curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3:8b

# For Stable Diffusion, use ComfyUI or Automatic1111

Common issues and fixes

“Device is in use by driver”

# The host is still using the GPU. Blacklist the host driver:
# (heredoc replaced)
echo "blacklist i915" >> /etc/modprobe.d/blacklist-i915.conf
echo "blacklist snd_hda_intel" >> /etc/modprobe.d/blacklist-i915.conf
update-initramfs -u -k all
reboot

“IOMMU group not isolated”

This means your GPU is in a group with other devices. Options:

• BIOS may have an “ACS Enable” option that breaks groups apart (rare on consumer boards)
• Use the ACS override patch (community scripts provide this for some setups)
• Accept it and pass through all devices in the group

“No display output”

The VM starts but you see nothing on the console. Likely:

• --bios ovmf not set
• --x-vga=1 not set
• Display is connected to the host GPU, not the passed-through one (use VNC/SPICE console instead)

GPU passthrough works but performance is poor

• BIOS might not have ACS or may have a substandard IOMMU implementation
• Some Intel iGPUs only support H.264 / HEVC 8-bit, not 10-bit or AV1
• For full codec support, look at “Intel Arc” discrete GPUs

Key takeaways

• IOMMU must be enabled in BIOS and via kernel parameters
• The iGPU is fine for Jellyfin transcoding; a discrete GPU is needed for AI
• The GPU and its audio function must be passed through together
• Use UEFI (OVMF) and q35 machine type for modern passthrough
• Test with a simple workload first (a Linux VM with the GPU)

Knowledge check