Passing devices into containers: GPUs, dongles and the boundary you give up · Proxmox VE

The single most common reason to pass a device into a container is hardware video transcoding: a media server in an unprivileged container that wants /dev/dri/renderD128 so the iGPU does the work instead of the CPU. The same mechanism carries serial dongles, TPMs, tape drives and anything else with a device node.

It is genuinely easier than the equivalent for a VM. Where a VM needs VFIO, an IOMMU group, a driver unbind and a reboot - the material in GPU passthrough - a container needs one config line, because there is only one kernel and the device is already bound to it.

That ease is exactly the thing to be careful about.

Two mechanisms, one of which you should use

The supported one: `dev[n]`

Configuration changepass the render node into a container— Changes the container config. Requires a restart. The device on the host is not reconfigured or detached from anything.

set -euo pipefail
CTID=200

# What the device is on the host, and who owns it there.
ls -l /dev/dri/
stat -c 'major=%t minor=%T owner=%U:%G mode=%a' /dev/dri/renderD128

# Pass it in. uid/gid/mode describe the node as it will exist INSIDE
# the container, in the container's own ID space.
pct set "$CTID" --dev0 '/dev/dri/renderD128,uid=0,gid=104,mode=0660'

pct reboot "$CTID"
pct exec "$CTID" -- ls -l /dev/dri/

The documented sub-options are few, and each does one thing:

Option	Meaning
`path=`	Path to the host device to pass through.
`uid=`	User ID assigned to the device node inside the container.
`gid=`	Group ID assigned to the device node inside the container.
`mode=`	Octal access mode set on the device node.
`deny-write=`	`1` denies the container write access to the device.

gid=104 above is not a magic number - it is whatever GID the render group has inside that container’s userland, which is a property of the container’s distribution, not the host’s. Read it, do not assume it:

Read-only / Safefind the GID the container's own group database uses

# pct exec 200 -- getent group render video

render:x:104:
video:x:44:

Illustrative output

Get this wrong and the device node appears with an unresolvable group, the application’s user is not in it, and every attempt to open the device returns permission denied - which reads in most application logs as “no hardware acceleration available” and nothing more specific.

The legacy one: raw `lxc.*` keys

Most forum answers written before PVE 8.2 use the low-level form, appended to /etc/pve/lxc/<vmid>.conf:

lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir

This still works - PVE passes lxc.* settings straight to the low-level LXC tools - and you will meet it in inherited estates. It does two separate things, which is worth understanding even if you never write it:

The devices.allow lines are a cgroup policy: they permit the container’s cgroup to open a character device with major 226 and the named minors.
The mount.entry line puts the device node into the container’s mount namespace so there is something to open.

You need both. A device node with no cgroup permission gives EPERM on open; a cgroup permission with no node gives ENOENT. That two-part structure is the reason “I added the device and it still does not work” is such a common report against the legacy method.

Why a device node is not a capability, and the cgroup controller is

A device node in /dev is just an inode carrying a major and minor number. Creating one grants nothing by itself - mknod is a filesystem operation. What decides whether a process may open it is the cgroup device controller, which the kernel consults on every open of a device inode by a process in that cgroup.

That separation is why the container security model works at all. If device access were purely a filesystem question, then any container that could mknod could manufacture a node for /dev/sda - major 8, minor 0 - and read the host’s disk directly. Under cgroup v2 the container’s cgroup has no rule permitting major 8, so the open fails no matter what inode exists.

It also explains the mknod feature flag. Enabling it lets the container create device nodes; it does not let the container use nodes it was not already permitted to use. The flag is far less dangerous than its name suggests, and correspondingly far less useful than people hope.

Under cgroup v2 - which is all PVE 9 uses - the controller is implemented as BPF programs attached to the cgroup rather than the v1 devices.allow file, which is why there is no readable list of permitted devices to inspect. You verify by attempting an open, not by reading policy.

What you are actually giving up

Say it plainly, because the forum posts do not.

Passing /dev/dri/renderD128 into a container gives every process in that container direct access to the DRM driver’s ioctl interface for your GPU. That interface is large, is written in C, runs in the kernel, and is shared with every other container and with the host. A container that is unprivileged is still sharing one kernel with everything else on the box; passing a device hands it a specific, complex, attack-relevant part of that kernel.

Device	What it exposes	Reasonable for
`/dev/dri/renderD128`	GPU compute and video engines, no display control	A trusted media or ML workload
`/dev/dri/card0`	The above plus modesetting and display control	Almost nothing in a container
`/dev/ttyUSB0`, `/dev/ttyACM0`	One serial device	A home-automation controller
`/dev/net/tun`	Arbitrary tunnel interface creation	A VPN container, with the network implications understood
`/dev/sdX`, `/dev/nvme0n1`	A raw block device, bypassing the storage layer	Effectively never - use a device mount point, and think hard

The transcode that quietly stopped happening

The symptom. Three weeks after a routine host reboot, users complain that the media server is slow and the host’s CPU sits at 90% every evening. The container is running. The application logs no errors. Monitoring shows high CPU, which is treated as a capacity problem, and someone proposes adding cores.

What happened. The host gained a second GPU - or a firmware update changed probe order, or a kernel update changed module load order - and the render node the container was pinned to is now renderD129. dev0 still points at /dev/dri/renderD128, which either does not exist or is a different adapter.

Why the first diagnosis is wrong. Almost every hardware-transcoding application treats “cannot open the render device” as a reason to fall back to software transcoding, not as a fatal error. It is a deliberate design choice and it is the right one for a desktop application. In a container it converts a hard failure into a slow, expensive, silent one. The CPU graph is a symptom, and adding cores makes the symptom cheaper without touching the cause.

The check. Not “is the container running” but “does the device exist inside it and can it be opened”:

CTID=200
pct exec "$CTID" -- ls -l /dev/dri/ || echo 'NO RENDER DEVICE IN CONTAINER'
pct exec "$CTID" -- sh -c 'command -v vainfo >/dev/null && vainfo 2>&1 | head -5'

The fix that survives the next reboot. Stop referencing the enumerated name. /dev/dri/by-path/ contains stable, PCI-topology-derived symlinks; resolve one on the host and pass the target, or write a udev rule that gives your adapter a fixed name. Then add the check above to the container’s post-start verification so the fallback is loud instead of silent.

What happens if the USB dongle is unplugged and plugged back in?

The container loses it and does not get it back until it is restarted, and this is one of the most-reported LXC problems in home and light-industrial estates - Zigbee coordinators, Z-Wave sticks, UPS cables, serial-attached meters.

The mechanism: the device node was placed into the container’s mount namespace when the container started. Unplugging removes the host’s node. Replugging creates a new node - possibly at a different name, because ttyUSB numbering is allocation order, not topology - and udev’s event goes to the host. Nothing propagates that new node into the already-running container’s namespace.

Three things follow, in the order you should apply them:

Never reference /dev/ttyUSB0 directly. Use a udev-provided stable path. /dev/serial/by-id/ is derived from the device’s own vendor, product and serial strings and survives renumbering.
Accept that a restart is part of the recovery. The container has to be restarted for the node to reappear inside it. Write that into the runbook rather than rediscovering it.
Alert on the absence, not on the application. A cron or systemd timer on the host that checks the node exists inside the container, and pages if it does not, turns a silent multi-day gap into a ten-minute one.

CTID=200
DEV=/dev/serial/by-id/usb-Silicon_Labs_slae.sh_cc2652rb_stick-if00-port0
pct exec "$CTID" -- test -c "$DEV" \\
  || echo "CT $CTID: $DEV missing inside container"

If the dongle is genuinely critical, the honest architecture is a VM with USB passthrough, where the device is attached to the guest at the QEMU level and the guest’s own udev handles the replug. That is more machinery, and it is the reason the machinery exists.

Devices pin a container to a node

A container with a dev[n] entry can only do its job on a host that has that device. Nothing in PVE enforces this.

Manual migration to a node without the device succeeds. The container starts. The device is absent inside it.
HA failover does the same, faster and unattended, at the worst possible moment.

If a device-bearing container is HA-managed, constrain it. A node-affinity rule naming the nodes that actually have the hardware is the mechanism, and it is covered in HA rules:

Configuration changepin a device-bearing container to nodes that have the device— Creates an HA node-affinity rule. Strict means the CRM will leave the container stopped rather than start it somewhere it cannot work.

set -euo pipefail
CTID=200

ha-manager rules add node-affinity ct-gpu-hosts \
--resources "ct:$CTID" \
--nodes 'pve-01,pve-02' \
--strict 1

ha-manager rules config --resource "ct:$CTID"

Strict is the right choice here and it is worth being explicit about why. A non-strict rule lets HA start the container on a node without the GPU, where it will run and be wrong. A strict rule leaves it stopped, which is visible, alarming and correct. Choosing “up but degraded” over “down and obvious” is almost always the wrong trade for hardware-dependent workloads.

Verification that can fail

Read-only / Safeprove the device is present, permitted and usable— Read-only. The third step actually opens the device, which is the only step that proves the cgroup rule and the ownership are both right.

set -euo pipefail
CTID=200
DEV=/dev/dri/renderD128

# 1. The node exists inside the container.
pct exec "$CTID" -- test -c "$DEV" && echo 'node present'

# 2. Ownership resolves and the application's user is in that group.
pct exec "$CTID" -- stat -c '%U:%G %a %n' "$DEV"
pct exec "$CTID" -- id www-data

# 3. It can actually be opened read-write, which is the only test that
#    exercises the cgroup device rule as well as the file mode.
pct exec "$CTID" -- runuser -u www-data -- \
sh -c "exec 3<>'$DEV' && echo 'open OK' && exec 3>&-"

Step 3 is the one that distinguishes the three failure modes from each other. ENOENT means the node is missing - a dev[n] path problem. EPERM as root means the cgroup rule is missing - a legacy-config problem. EACCES as the service user means ownership or mode is wrong - a uid/gid/mode problem. Steps 1 and 2 alone cannot tell those apart.

Knowledge check

Why this matters

Device passthrough into a container is the place where the convenience of containers and the honesty of the security model pull hardest against each other. It is two lines of config and it works immediately, which makes it feel like a feature rather than a decision.

It is a decision. You are handing a specific kernel driver’s interface to a workload, on a machine where that kernel is also running everything else. For a trusted media server on a home lab that is a fine trade and nobody should agonise over it. For a container running third-party code on a shared cluster it is a trade nobody should make silently.

The operational half matters just as much and is easier to forget: a device turns a portable container into a hardware-bound one. It stops being something the cluster can place anywhere, and it acquires a failure mode - silent fallback - that ordinary monitoring will not catch. Both of those need to be written down where the next operator will find them, because neither is visible in pct config output to someone who does not already know to look.