Container networking and storage · Proxmox VE

Why this matters in production

Most production container problems are networking or storage problems. Configuring these correctly from the start avoids pain later.

Container networking

Containers can have one or more network interfaces. Each attaches to a bridge on the host.

pct create 200 local:vztmpl/debian-12-standard_12.2-1_amd64.tar.zst --hostname web-ct --net0 name=eth0,bridge=vmbr0,ip=dhcp --cores 2 --memory 1024 --rootfs local-zfs:8

Network options:

Option	Purpose
`bridge=`	Bridge on the host
`ip=dhcp`	DHCP on this interface
`ip=10.10.100.50/24`	Static IP with CIDR
`gw=10.10.100.1`	Default gateway
`tag=100`	VLAN tag (requires VLAN-aware bridge)
`firewall=1`	Enable container firewall
`rate=100`	Rate limit in Mbps

Multiple interfaces

Containers can have multiple NICs for separation:

pct set 200 --net1 name=eth1,bridge=vmbr1,ip=10.20.0.50/24

SDN VNets

Use the VNet name as the bridge:

pct set 200 --net0 name=eth0,bridge=vn-web,ip=dhcp

What is actually between eth0 and the wire

net0: name=eth0,bridge=vmbr0 reads like the container is plugged into the bridge. What exists is a veth pair: two virtual interfaces joined back-to-back, one end inside the container’s network namespace named eth0, the other end on the host named veth<vmid>i0, enslaved to the bridge.

Anything written to one end appears on the other. So the path from a process in the container to the physical network is:

process -> eth0 (container netns) -> veth200i0 (host) -> vmbr0 -> bond0 -> wire

You can see the whole thing, and seeing it is what turns container networking from configuration into troubleshooting:

CTID=200
ip -br link show master vmbr0 | grep "veth$CTID"
bridge link show | grep "veth$CTID"
pct exec "$CTID" -- ip -br addr show eth0

Two things follow that are not obvious from the config syntax.

The container’s traffic is bridged, not routed. It arrives on the physical network with the container’s own MAC address, so it is subject to whatever the switch does about MAC learning, port security and VLAN membership. A container that cannot reach anything while the host can is very often a switch-side policy, not a Proxmox problem.

The host sees each container’s traffic as a distinct interface. That is what makes per-container firewalling and rate limiting possible at all, and it is why rate= on the net[n] option is enforced on the host end of the veth rather than inside the container - a process in the container cannot remove its own rate limit, because the limit is not in its namespace.

Container firewall

The container firewall is a separate object from the host firewall, with its own enable flag, its own rule set and its own default policies. Enabling one does not enable the other, and this catches people repeatedly.

Three switches must all be on for a container rule to have any effect:

The datacenter firewall, in /etc/pve/firewall/cluster.fw.
The container’s firewall option, firewall=1 on the net[n] entry.
The rules themselves, in /etc/pve/firewall/<vmid>.fw.

Configuration changecheck all three layers before writing a rule— Reads the firewall configuration at each level. A rule set with firewall=0 on the interface is inert, and nothing warns you.

set -euo pipefail
CTID=200

# 1. Cluster level
grep -E '^enable' /etc/pve/firewall/cluster.fw 2>/dev/null \
|| echo 'cluster firewall not enabled'

# 2. Is the flag set on the interface itself?
pct config "$CTID" | grep -E '^net[0-9]+:' | grep -o 'firewall=[01]' \
|| echo 'firewall= not set on any interface (defaults to off)'

# 3. The rules
cat "/etc/pve/firewall/$CTID.fw" 2>/dev/null || echo 'no rules file'

Rules that exist, are correct, and do nothing

The symptom. A container has a carefully written rule set denying inbound traffic except on port 443. Someone tests from another container and reaches port 22 without difficulty. The rules are re-read, the syntax is fine, the order is right.

What happened. firewall=1 is not set on the net0 entry, so the container’s veth is not attached to the per-guest chain at all. The rules file is present, valid, and never consulted.

Why the first diagnosis is wrong. Everything visible about the rules is correct, so the investigation goes deeper into the rules - direction, macros, IP sets, ordering - which is a long way to look for something that is not there. The switch that is off is one level up, in the interface definition rather than in the firewall configuration.

The one-line check and fix:

CTID=200
pct config "$CTID" | grep -E '^net[0-9]+:'
pct set "$CTID" --net0 name=eth0,bridge=vmbr0,ip=dhcp,firewall=1

The general lesson, which applies to the host firewall too: a firewall that is not enforcing produces exactly the same output from every inspection command as one that is. Test it by attempting a connection that should fail, never by reading the rules.

Container storage

Root filesystem

Every container has a rootfs (rootfs=local-zfs:8). It is a ZFS subvolume, directory, or LVM volume.

Mount points come in three shapes

The value in the first position of mp[n] selects between three quite different things, and the difference decides what vzdump protects.

Shape	Written as	Backed up by vzdump
Volume - PVE allocates it on a storage	`mp0: local-zfs:32,mp=/data`	Yes, unless `backup=0`
Bind - a host directory	`mp0: /srv/ct-data,mp=/data`	No
Device - a host block device	`mp0: /dev/disk/by-id/...,mp=/data`	No

# A volume mount point: 32 GiB allocated on local-zfs, included in backups.
pct set 200 --mp0 local-zfs:32,mp=/data

# A bind mount point: an existing host directory, NOT in backups.
pct set 200 --mp1 /srv/ct-data,mp=/srv-data

There is no nfsmount= sub-option, and no way to bind an NFS export directly. The correct pattern is to mount the export on the host - through a PVE storage definition, or in the host’s /etc/fstab - and then bind the resulting host path into the container.

The full treatment of the ownership arithmetic, the idmap= option and the shared=1 migration trap is in bind mounts and mount points. The short version to carry into any design conversation: a bind mount is a statement that something other than vzdump protects that data, and if you cannot name what, use a volume mount point instead.

Backup behaviour

Mount points can be:

backup=1 (default): included in vzdump backups.
backup=0: excluded. Useful for ephemeral data or externally-managed storage.

backup applies to volume mount points only. Setting it on a bind mount point changes nothing - the content was never in scope - and the fact that PVE accepts the option without complaint is how a lot of people acquire a false sense of coverage.

Storage backends

Backend	Use
`local-zfs` (zfs subvolume)	Default; snapshots; performance
`local` (directory)	Simple; no ZFS features
`local-lvm` (LVM thin)	Thin provisioning; no native snapshots
NFS	Shared; no native snapshots from container’s POV
CephFS	Distributed shared filesystem
Ceph RBD (via krbd)	Block device inside container

Production considerations

What happens if the bridge named in a container's config does not exist on the node it starts on?

The container fails to start, with an error that names the bridge - which is the good case. The bad case is that the bridge exists with the same name and a different meaning.

Bridge missing. pct start fails in the pre-start hook. pct start --debug names the interface. This is loud, immediate and easy.

Bridge present but attached to a different VLAN or uplink. The container starts, gets an interface, and is on the wrong network. DHCP either fails - so the container has no address and the application times out - or succeeds against a different DHCP server, and the container is now reachable at an address nobody expects, possibly on a network it should not be on at all.

The second case is what makes bridge naming a cluster-wide discipline rather than a per-node one. vmbr1 must mean the same thing on every node, or migration and HA recovery become a way to move guests between networks by accident.

The check, before you trust a container to be migratable:

CTID=200
for BR in $(pct config "$CTID" | grep -oE 'bridge=[a-z0-9]+' | cut -d= -f2 | sort -u); do
  for NODE in pve-01 pve-02 pve-03; do
    printf '%-8s %-8s ' "$NODE" "$BR"
    ssh -o BatchMode=yes "root@$NODE" \
      "bridge link show 2>/dev/null | grep -c \"master $BR\" || echo 0"
  done
done

Equal port counts across nodes is not proof of equivalence, but a bridge with three ports on one node and none on another is a finding worth having before a failover produces it.

Verification

Read-only / Safedoes this container's networking and storage actually work— Read-only apart from one test file that is removed. The last two checks are the ones that fail on a container that looks configured and is not.

set -euo pipefail
CTID=200

echo '--- network ---'
pct exec "$CTID" -- ip -br addr show
pct exec "$CTID" -- ip route show default
pct exec "$CTID" -- getent hosts deb.debian.org || echo 'DNS RESOLUTION FAILS'

echo '--- the host end of the veth ---'
ip -br link show | grep "veth$CTID" || echo 'NO VETH ON HOST'

echo '--- mounts, and whether they are real ---'
pct exec "$CTID" -- findmnt -no TARGET,SOURCE,FSTYPE \
| grep -vE '^/(proc|sys|dev|run)'

echo '--- can the service account actually write ---'
pct exec "$CTID" -- runuser -u www-data -- \
sh -c 'touch /data/.wtest && rm /data/.wtest && echo "write OK"' \
|| echo 'WRITE FAILED - check ownership and the ID mapping'

The DNS check is there because it is the single most common container networking failure and the one least likely to be caught by pinging an IP. A container with a correct address, a correct route and no working resolver behaves like a container with a network problem, and every layer-3 test you run will pass.

Common mistakes

Using bridge=vmbr0 when the production design uses a separate storage or backup bridge.
Mounting host paths with permissions that conflict with the unprivileged UID mapping.
Forgetting to rate-limit tenant containers.
Writing firewall rules without setting firewall=1 on the interface, so nothing is enforced and nothing says so.
Expecting backup=0 or backup=1 to mean anything on a bind mount point.
Letting the same bridge name mean different networks on different nodes.

Key takeaways

Containers attach to bridges through a veth pair; the host end is a real interface you can inspect, filter and rate-limit.
Multiple NICs allow traffic separation.
Mount points come in three shapes and only one of them is backed up.
The container firewall needs three switches on, and a rule set that is not being enforced looks identical to one that is.

Knowledge check

Why this matters

Container networking and storage look simpler than the VM equivalents, and they are - right up to the point where something does not work, at which point the simplicity turns out to have been a thinner abstraction rather than a smaller problem.

Both halves of this lesson have the same shape. The configuration line is short and reads as a complete description; underneath it there is a veth pair with a host end you can filter, or a mount point that is one of three different things with three different backup outcomes. Operators who know what is underneath diagnose in minutes, because they know which layer to look at next. Operators who only know the config line have one place to look, and when the answer is not there they are stuck.

The two failures worth carrying away are both of the same species: something is configured, appears correct under every inspection, and is not in effect. A firewall rule set with firewall=0 on the interface. A bind mount with backup=1 set on it. In both cases the fix is trivial and the diagnosis is not, because the system’s own output cannot distinguish working from inert. Which is the argument for verification that attempts the thing rather than reads about it.