The firewall in depth: three levels, and the nftables backend

The Proxmox firewall is not one firewall. It is a rule generator that emits packet filter rules at several distinct points in the path a packet takes, from separate configuration files, with separate enable switches. Understanding which file governs which packet is most of the skill; the rule syntax itself is small.

The practical question this lesson answers is: given a thing you want to block or permit, which file do you edit? Getting that wrong produces a rule that is syntactically perfect, visible in the GUI, and never consulted for the traffic you care about.

The four levels

All four live in /etc/pve, which means all four are cluster-replicated. A datacenter rule written on node 1 is in effect on node 4 immediately. It also means all four stop being editable when quorum is lost.

Three rules of thumb resolve almost every placement question:

Traffic to the hypervisor is host.fw. SSH to the node, the web GUI on 8006, the Ceph public network, corosync. If the packet’s destination is the node’s own IP, the node firewall is what sees it.

Traffic to a guest is $VMID.fw. A rule permitting 443 to a web server belongs on that guest, not on the node it happens to run on today. This is the placement error with the longest half-life, because a node-level rule works perfectly until the guest migrates.

Definitions are cluster.fw. Security groups, IP sets and aliases defined at datacenter level are referenceable from every guest and node. Definitions written elsewhere are not shared.

Directions, and the FORWARD direction specifically

Every rule declares a direction: IN, OUT or FORWARD.

IN and OUT are relative to the zone the file governs. On host.fw, IN is traffic arriving at the node. On $VMID.fw, IN is traffic arriving at the guest — from anywhere, including another guest on the same bridge.

FORWARD is traffic passing through: routed by the host, or crossing a VNet. It is the direction that matters when a Proxmox node is doing SDN routing, and it is the one that did not exist as a first-class concept before the nftables backend gave VNet-level rules somewhere to live.

The syntax in the [RULES] section is positional:

[RULES]
IN  SSH(ACCEPT) -source +management -log nolog
IN  ACCEPT -p tcp -dport 443 -source +web-clients
OUT ACCEPT -p tcp -dport 514 -dest 192.0.2.30
|IN ACCEPT -p tcp -dport 8080

Direction, then either ACTION or MACRO(ACTION), then options. A rule prefixed with | is disabled — kept in the file, not compiled. That is the mechanism for staging a rule you are not ready to enable and for disabling one during an incident without losing it.

Macros such as SSH, HTTP, DNS and Ceph expand to the port and protocol sets those services need. They are worth using: SSH(ACCEPT) cannot be mistyped as the wrong port number, and a macro that covers multiple ports covers all of them.

The primitives that stop rules multiplying

A firewall becomes unmaintainable when the same fact is written in many places. Three constructs prevent that, all defined in cluster.fw.

Aliases — a name for an address

[ALIASES]
mgmt_net    192.0.2.0/24
monitoring  198.51.100.15
dr_site     203.0.113.0/24

Reference them as -source mgmt_net. When the management network is renumbered, one line changes.

local_network is defined automatically and covers the cluster communication network. Inspect what Proxmox worked out:

pve-firewall localnet

IP sets — a name for a group of addresses

[IPSET management]
192.0.2.10
192.0.2.11
198.51.100.0/28

[IPSET web-clients]
203.0.113.0/24

Reference them with a + prefix: -source +management. IP sets are matched in a single kernel-side set lookup rather than one rule per address, so a set of five hundred addresses costs roughly what one costs.

Three IP set names are special:

management holds the hosts permitted to do normal management — GUI, VNC, SPICE, SSH. It applies to host firewalls only, and it automatically includes the cluster network so nodes can talk to each other.

blacklist is dropped by every host and every guest, cluster-wide. It is the fastest lever during an active attack: one edit, effective everywhere.

ipfilter-net0, ipfilter-net1, … are per-interface anti-spoofing sets on a guest. With ipfilter enabled, a guest may only source packets from addresses in the corresponding set. For containers with configured IPs the set is implicitly populated. Both VMs and containers implicitly include the MAC-derived IPv6 link-local address so neighbour discovery keeps working. This is the control that stops a compromised guest impersonating another tenant on the same bridge, and it is off by default.

Security groups — a name for a set of rules

[group web-tier]
IN ACCEPT -p tcp -dport 80
IN ACCEPT -p tcp -dport 443
IN ACCEPT -p tcp -dport 22 -source +management

[group db-tier]
IN ACCEPT -p tcp -dport 5432 -source +app-servers
IN ACCEPT -p tcp -dport 22 -source +management

A guest then references the group in its own [RULES]:

[OPTIONS]
enable: 1
policy_in: DROP

[RULES]
GROUP web-tier
IN ACCEPT -p tcp -dport 9100 -source monitoring

Adding a port to the web tier is one edit in cluster.fw and it reaches every guest referencing the group. Without groups it is one edit per guest, and the guest somebody forgot is the one that breaks.

What happens when you set the policy to DROP

Setting policy_in: DROP at datacenter level does not sever your connection, and the reason is a documented set of always-permitted flows. With a DROP or REJECT input policy, the following still pass cluster-wide:

• Loopback traffic, and any established connection
• TCP 8006 from the management set — the web GUI
• TCP 5900–5999 from management — VNC consoles
• TCP 3128 from management — the SPICE proxy
• TCP 22 from management — SSH
• TCP 60000–60050 from management — live migration
• UDP 5405–5412 on the cluster network — corosync
• IGMP, and ICMP types 3, 4 and 11

Two readings of that list matter.

It is why enabling the firewall usually does not lock you out. The management path is preserved by construction.

It is also why “the firewall is enabled” is a weaker statement than it sounds. Everything above is permitted from the management set, and pve-firewall localnet tells you what that set currently contains. If it resolved to a broad network — a flat estate where the cluster network and the office network are the same subnet — then enabling the firewall permitted GUI, SSH, VNC and SPICE from the whole of it. The control that matters is narrowing the management set, not flipping the enable switch.

A separate list is dropped silently and never logged: invalid-state TCP, broadcast and multicast, TCP 43, UDP 135/445, UDP 137–139, UDP 1900, TCP 135/139/445, and UDP source port 53. This is why a Windows guest’s SMB traffic can vanish with no log line to explain it — worth knowing before you spend an hour on a rule that was never the problem.

The nftables backend

Since PVE 8.2 there is a second implementation. pve-firewall generates iptables rules; proxmox-firewall generates nftables rules from the same configuration files. In PVE 9.2 the nftables backend remains a technology preview.

apt update
apt install proxmox-firewall

Enable it per node, in host.fw:

[OPTIONS]
enable: 1
nftables: 1

Per node is the important property: you can convert one node, observe it, and convert the rest — or convert it back — while the cluster keeps running. The same GUI path is Host → Firewall → Options → nftables.

Four behavioural differences are worth knowing before the migration, not during it:

No firewall bridges. The iptables backend inserts an additional Linux bridge per guest interface to attach its rules. The nftables backend does not. Anything that inspected or depended on those interfaces — a monitoring check counting bridges, a script parsing brctl show — sees a different topology afterwards.

REJECT is unavailable for guest traffic and becomes a drop. A rule written as REJECT silently behaves as DROP. The difference is user-visible: a rejected connection fails immediately, a dropped one hangs until the client times out. Applications with short connect timeouts behave the same either way; anything with a long timeout gets slow rather than broken, which is a harder symptom to diagnose.

NDP, router advertisements and DHCP always get rules, regardless of the default policy. The iptables backend honours the policy for these. If your threat model depends on blocking rogue DHCP or RA within a bridge, verify it rather than assuming the policy still covers it.

Guest rules are evaluated even for existing conntrack entries. The iptables backend short-circuits established connections. A rule change under nftables therefore affects connections that are already open, where under iptables it applied only to new ones. This makes changes take effect more predictably and means a tightening rule can drop a live session that would previously have survived.

Verifying which backend is actually running

pve-firewall status
systemctl status proxmox-firewall.service --no-pager
nft list ruleset | head -40
iptables-save | head -40

pve-firewall compile prints the rules the current configuration would generate without applying them — the right thing to run before a change window, and the fastest way to find out that a rule you wrote is not being emitted at all.

pve-firewall compile

# Can my workstation still reach the GUI on this node?
pve-firewall simulate --from outside --to host \
--source 192.0.2.10 --dest 198.51.100.20 \
--protocol tcp --dport 8006 --verbose 1

# Can the app tier reach the database guest on 5432?
pve-firewall simulate --from vm201 --to vm310 \
--protocol tcp --dport 5432 --verbose 1

simulate is the tool people do not know exists. It answers “would this packet get through, and which rule decided” without generating the packet, which makes it usable against production from a laptop.

Note the shape of the arguments, because it is easy to get wrong. --from and --to name zones — host, outside, or a guest as vm100 / ct101 — while --source and --dest take the IP addresses. Passing an address to --from is rejected. The zone pair is what selects which chains the simulated packet traverses, which is precisely the question a placement error turns on.

Common mistakes

• Guest rules written in host.fw. They work until the guest migrates, which is exactly when nobody is looking at the firewall.
• Enabling the datacenter firewall without checking pve-firewall localnet. The management set may resolve to a network far broader than intended, and the permitted-by-default list then applies to all of it.
• Never enabling ipfilter on guests. Anti-spoofing between tenants on the same bridge is off by default.
• Per-guest copies of the same rules. Use a security group; the guest somebody forgot is the one that breaks.
• Editing iptables or nft directly during an incident. The next compile discards it. Edit the .fw file.
• Switching to nftables without restarting guests. Running guests keep the previous backend’s attachment.
• Expecting REJECT under nftables. It drops, and the symptom is slowness rather than a clean refusal.
• Enabling logging with a DROP policy and no rate limit on a busy node. The root filesystem fills.

Key takeaways

• Four levels, four files: datacenter for defaults and definitions, node for the host, VNet for forwarded SDN traffic, guest for the guest. A rule in the wrong file is never consulted, not merely outranked.
• Aliases, IP sets and security groups keep one fact in one place.
• The management IP set is what the permitted-by-default management flows are scoped to. Narrowing it is the control; the enable switch alone is not.
• ipfilter is the anti-spoofing control between guests, and it is off by default.
• pve-firewall compile and pve-firewall simulate answer “what would this do” without applying it.
• The nftables backend is per node and reversible, changes guest REJECT into a drop, always permits NDP/RA/DHCP, evaluates guest rules for established connections, and requires a guest restart.
• Never edit the generated ruleset. The next compile discards it.

Knowledge check