Network isolation between tenants · Proxmox VE

The previous lesson ended on the row that matters most in the table of things a pool does not isolate: network. Two guests attached to the same bridge can reach each other, and neither the pool they belong to nor the ACLs on that pool have any bearing on it.

Isolation between tenants is a property of two things — which layer-2 domain a guest is attached to, and who is allowed to attach a guest to it. Everything in this lesson is one of those two.

The permission that does the work

Start with the one that surprises people. Consider a tenant with a sensible role: VM.Allocate, the VM.Config.* privileges including VM.Config.Network, VM.Console, VM.PowerMgmt, scoped tightly to /pool/tenant-blue.

VM.Config.Network is documented as “add/modify/remove network devices”. So the tenant can add a network device. To what?

To whichever bridge or VNet they name — unless something stops them. The privilege that stops them is SDN.Use, documented as “access SDN vnets and local network bridges”, and its ACL path is what confines the tenant:

Configuration changeconfine a tenant to their own zone— Changes access control. Grants SDN.Use on one zone path only. Without a grant like this — and without removing any broader grant — a tenant with VM.Config.Network can attach a guest NIC to any network the cluster has.

POOL="tenant-blue"
ZONE="blue"

pveum role add TenantNetwork --privs "SDN.Use SDN.Audit"
pveum acl modify "/sdn/zones/$ZONE" --groups "$POOL-admins" --roles TenantNetwork

pveum user permissions someone@ldap --path "/sdn/zones/$ZONE"

This is the general shape of tenant confinement in PVE, and it is worth stating once as a principle: a tenant is confined by the set of paths their roles are granted on, not by the pool their guests are in. The pool is where the guests are; the grants are what they can touch.

Zone per tenant

With the permission model settled, the topology. Proxmox VE’s SDN offers five zone types and they suit different estates:

Zone type	Isolation mechanism	Suits
Simple	An isolated bridge per VNet, local to each node, with routing or NAT	Small estates, lab-adjacent tenancy, tenants who only need outbound
VLAN	802.1Q tags on an existing bridge	Most on-premises estates; requires switch configuration to match
QinQ	Stacked service and customer VLAN tags	Providers who have run out of VLAN space or need a per-customer tag space
VXLAN	Layer-2 overlay over UDP 4789 on an underlay network	Estates where the physical fabric must not know about tenant VLANs
EVPN	Routable layer-3 with BGP, VRF and anycast gateway per node	Larger providers; the only one giving each tenant a routing domain

The tenancy-relevant point is that the zone is the ACL object. A zone per tenant means a path per tenant — /sdn/zones/blue, /sdn/zones/green — and therefore a clean grant. A shared zone with per-tenant VNets inside it gives you one path to grant on and no way to distinguish the tenants using it.

Zone per tenant, VNets inside it as the tenant likes

The layout that scales:

One zone per tenant. It is the permission boundary and it costs nothing.
VNets inside that zone for the tenant’s own segmentation — front end, back end, management — named with the tenant prefix so a name collision across tenants is impossible.
Subnets on each VNet where you want IPAM to assign addresses and where a gateway or SNAT is needed.

Two operational notes. VNet names become interface names on the nodes, so keep them short and unambiguous — a 12-character tenant prefix leaves little room. And an SDN apply is a cluster-wide operation that reconfigures networking on every node; changes are staged as pending and applied atomically, so batch a tenant’s whole set of changes into one apply rather than applying six times.

Read-only / Safesee the zones, VNets and their pending state— Read-only. Lists SDN objects and shows whether any configuration is pending an apply. Run this before applying, so you know exactly what the apply is about to change.

pvesh get /cluster/sdn/zones --output-format json | jq -r '.[] | [.zone, .type, (.nodes // "all")] | @tsv'

pvesh get /cluster/sdn/vnets --output-format json | jq -r '.[] | [.vnet, .zone, (.tag // "-"), (.isolate_ports // 0)] | @tsv'

Read-only / Safe

$ pvesh get /cluster/sdn/vnets --output-format json | jq -r '.[] | [.vnet, .zone, (.tag // "-"), (.isolate_ports // 0)] | @tsv'

blue-front	blue	1201	0
blue-back	blue	1202	1
green-front	green	1301	0
green-back	green	1302	1

The three switches

The commonest firewall failure in Proxmox is not a wrong rule. It is a correct rule that is not being enforced, because the firewall has three independent enable flags and all of them must be on.

Cluster-wide: enable: 1 in the [OPTIONS] section of /etc/pve/firewall/cluster.fw. Off by default.
Per guest: enable in /etc/pve/firewall/<VMID>.fw, which defaults to 0.
Per network interface: the firewall=1 flag on the guest’s network device itself, in the guest configuration.

The per-host flag in host.fw is a fourth, though it defaults to enabled once the cluster flag is set.

The rules were perfect and the traffic went through

A tenant onboarding produced a careful <VMID>.fw for every guest: default drop inbound, explicit allows for the ports the application needs, a security group for the shared management access.

Six weeks later a penetration test reached a database port from another tenant’s guest. The rules were correct. pve-firewall status on the node reported the firewall as running.

The guests had been created by cloning a template whose network device predated the firewall standard, and carried no firewall=1 on the NIC. Every rule was parsed, stored and displayed in the GUI, and none of it was attached to a device.

The first diagnosis is always the rule set, because that is what you can read. Check enforcement instead: qm config <vmid> | grep ^net shows whether the flag is on the device, and pve-firewall compile prints the rules the node has actually generated. If a guest’s chain is absent from the compiled output, no rule in its .fw file matters.

Read-only / Safeverify enforcement rather than configuration— Read-only. Four checks in ascending order of trust: the cluster flag, the per-guest flag, the per-interface flag, and the rules the node has actually compiled. Only the last one is evidence.

VMID=141

grep -A5 '^\[OPTIONS\]' /etc/pve/firewall/cluster.fw

cat "/etc/pve/firewall/${VMID}.fw"

qm config "$VMID" | grep '^net'

pve-firewall status

pve-firewall compile | grep -A20 "${VMID}"

Stopping a tenant from lying about their address

Layer-2 isolation puts tenants on separate segments. Inside a shared segment — and there is usually at least one, for management or for a shared service — a guest can claim any IP or MAC it likes unless something prevents it.

PVE’s mechanism is the ipfilter IPSet. A per-guest IPSet named ipfilter-net<N>, matching the guest’s interface, restricts the source addresses that guest may use. The firewall chapter documents these ipfilter-net* sets as per-interface spoofing prevention.

Configuration changepin a guest to the address it was given— Changes the guest firewall configuration. Adds an ipfilter IPSet for net0 containing the one address the guest is entitled to use. Traffic from other source addresses on that interface is dropped. Get the address wrong and the guest loses connectivity — verify from the guest after applying.

VMID=141
NODE="$(hostname)"

pvesh create "/nodes/${NODE}/qemu/${VMID}/firewall/ipset" --name "ipfilter-net0"
pvesh create "/nodes/${NODE}/qemu/${VMID}/firewall/ipset/ipfilter-net0" --cidr 192.0.2.50

pvesh get "/nodes/${NODE}/qemu/${VMID}/firewall/ipset/ipfilter-net0"

Spoofing protection matters more in multi-tenancy than in a single-tenant cluster for a specific reason: on a shared segment, every other tenant’s firewall rules that permit a source address are only as good as the guarantee that the source address is real. Without ipfilter, an allow-list by IP is an allow-list by claim.

isolate-ports, and what it is not

The SDN documentation describes isolate-ports as marking all guest ports as isolated, preventing communication between guests while still permitting traffic to the gateway. That is exactly the semantic you want for a segment where tenants share a VNet but must not see each other — a hosting VLAN, for example.

Two limits, both documented, both important:

It is local to each host. Guests on the same VNet but on different nodes are not isolated from one another by this setting. The documentation says so directly and recommends the VNet firewall for cluster-wide isolation.

The VNet firewall needs the nftables backend. VNet-level rules live in /etc/pve/sdn/firewall/<vnet_name>.fw and require the proxmox-firewall nftables backend, enabled with nftables: 1 in the host configuration. That backend is documented as a tech preview, and the same applies to forward-direction rules — the iptables backend ignores them.

Do not build a tenancy boundary on a tech preview

The honest position for a production multi-tenant estate on PVE 9.2:

Use separate VNets with separate VLAN tags as the primary isolation. It is ordinary layer-2 separation, enforced by the bridge and the switch, and it does not depend on any preview feature.
Use guest-level firewall rules with a default drop as the second layer, on the stable iptables backend, with all three enable flags on.
Treat isolate-ports as a per-host hardening measure that helps and does not suffice.
Adopt the nftables backend, forward rules and VNet firewalls deliberately — on a validation cluster first, as the fleet lesson describes — and re-evaluate their preview status at each release rather than assuming it has changed.

Building the boundary a customer is paying for on top of a feature the vendor labels a tech preview is a decision, and it should be one somebody made on purpose and wrote down.

Under the hood

Where does isolation actually happen?

Follow a packet out of a guest. It leaves the emulated NIC into a tap device on the host, which is enslaved to a Linux bridge — vmbr0, or the bridge a VNet materialised on each node. The bridge is a layer-2 switch in software. If two tap devices are on the same bridge in the same VLAN, the bridge will forward between them, and it will do so without the packet ever touching the physical network.

That is the sentence that explains most tenant-isolation surprises. A switch ACL, a router filter, or a physical firewall between two racks sees nothing of that traffic, because it never left the host. The controls that can see it are the ones in the host: the bridge’s VLAN configuration, the netfilter hooks the PVE firewall installs on the tap device, and the bridge’s own port isolation.

Three consequences worth carrying:

Separate VLANs are the strongest of the three, because they change what the bridge is willing to forward rather than filtering what it forwards.

A firewall rule only exists where the hook is installed — which is why the per-interface firewall=1 flag is not a formality. No flag, no hook, no filtering, regardless of what the .fw file says.

isolate-ports is a bridge-level property on the local host, which is precisely why it cannot isolate two guests on different nodes: each node’s bridge knows only about its own ports, and the traffic between the nodes crosses the physical fabric where neither bridge’s isolation applies.

Tenant network onboarding checklist

Zone created for the tenant, named for the tenant, with the zone type matching the estate’s fabric.
VNets created inside that zone, prefixed with the tenant name, each with its own VLAN tag or VXLAN VNI from an allocated range.
Subnets defined where IPAM, gateway or SNAT are wanted.
SDN.Use granted on /sdn/zones/<tenant> only, and pveum acl list read to confirm no broader grant exists anywhere.
Cluster firewall enabled (enable: 1 in cluster.fw) with a default inbound policy the estate has agreed.
Per-guest firewall enabled and firewall=1 on every guest NIC, including in whatever template the tenant clones from.
ipfilter-net<N> populated for every guest on a shared segment.
Verified from the wrong side: from a guest in another tenant’s zone, attempt to reach this tenant’s guests, and record the result. A test run from the hypervisor proves nothing about tenant-to-tenant reachability.
Compiled rules checked with pve-firewall compile on at least one node holding a tenant guest.
Offboarding written down: zone, VNets, subnets, VLAN tags returned to the pool, ACL entries, IPSets.

What happens if two tenants end up on the same VLAN?

Assume it will happen once — a VLAN tag reused from a spreadsheet, a VNet created in the wrong zone, or a guest NIC attached to a shared bridge because a tenant had a broader SDN.Use grant than anyone realised.

What is true immediately. The two tenants’ guests are on one layer-2 domain. They can ARP for each other, reach every port not blocked by a guest firewall, and see each other’s broadcast traffic — which on many networks means discovery protocols, DHCP and sometimes cluster heartbeat traffic from applications inside the guests. If either tenant relies on IP allow-lists, those allow-lists are now trusting an unverified claim.

How you find out. Rarely from the network. Usually from a guest’s own logs — unexpected connection attempts, DHCP offers from an unknown server, or an application clustering with a stranger. The alternative is a scheduled check: from a guest in each zone, attempt to reach a known address in every other zone and alert on any success. That check is cheap and almost nobody runs it.

Resolution order.

Contain before you tidy. Enable the guest firewall with a default drop on the affected guests if it is not already enforced, and verify with pve-firewall compile rather than trusting the .fw files.
Establish exposure. How long, which guests, what was reachable. This is a disclosure question as much as a networking one, and for a customer tenancy it probably has a contractual notification attached.
Correct the topology, not the symptom. Move the misplaced VNet or guest into the correct zone with its own tag. Changing firewall rules to compensate for a shared segment leaves the shared segment.
Fix the cause. A reused VLAN tag means the tag allocation is not authoritative — put it in the baseline document from the standards lesson. A broader-than-expected SDN.Use grant means the ACL review at onboarding did not happen; make pveum acl list part of it.
Add the reachability check so the next occurrence is found by you rather than by a tenant.

Key takeaways

A pool does not isolate networks. Isolation is which layer-2 domain a guest is attached to, plus who may attach a guest to it.
SDN.Use — “access SDN vnets and local network bridges” — granted on /sdn/zones/<zone> is what confines a tenant. Without it constrained, a tenant holding VM.Config.Network can attach a NIC to any network the cluster has.
Permissions are additive across paths. Read pveum acl list and pveum user permissions <user> --path / at onboarding; a legacy grant at the root defeats a careful grant on a zone.
One zone per tenant, because the zone is the ACL object. VNets inside it, prefixed with the tenant name and each with its own tag.
The firewall has three independent enable flags: enable: 1 in cluster.fw, enable in <VMID>.fw which defaults to 0, and firewall=1 on the guest NIC. All three, or nothing is enforced.
Verify with pve-firewall compile, not by reading .fw files. A guest with no compiled chain has no firewall regardless of its configuration.
ipfilter-net<N> IPSets stop a guest spoofing source addresses. On a shared segment, an IP allow-list without them is an allow-list by claim.
isolate-ports is local to each host and does not isolate guests on different nodes. The VNet firewall (/etc/pve/sdn/firewall/<vnet>.fw) and forward-direction rules require the proxmox-firewall nftables backend, which is a tech preview.
Test isolation from another tenant’s guest, on a schedule. Nothing else proves it.

Knowledge check

Why this matters

Tenant isolation fails quietly. That is what makes it different from almost every other topic in this course.

A storage misconfiguration produces an error. A quorum problem produces an outage. A missing firewall=1 produces a working system, on a working network, with satisfied users, for as long as nobody looks — and the first evidence is usually somebody else’s audit.

So the discipline this subject needs is not more configuration. It is verification from the outside: a guest in tenant A’s zone that tries, on a schedule, to reach a known address in tenant B’s, and tells you when it succeeds. Ten lines of script, running weekly, is worth more than another layer of rules, because it is the only thing in the design that can distinguish “isolated” from “not yet tested”.

Everything else in this lesson is about making that check pass by construction: separate layer-2 domains so the bridge will not forward, SDN.Use scoped so a guest cannot be attached to the wrong one, and the firewall genuinely enforced rather than merely written down.