Bonds, LACP, VLANs, MTU · Proxmox VE

Why this matters in production

Most production outages caused by networking are not “the network broke.” They are “the network reconfigured itself because one link flapped and the bond didn’t failover.” This lesson teaches the configurations that survive real failures.

Bond modes

Mode	Behaviour	Failover	Load balancing	Switch cooperation
`balance-rr` (0)	Round-robin packets	No	Per packet	None required
`active-backup` (1)	One active, others standby	Yes	No	None required
`balance-xor` (2)	Hash-based	No	Per flow	Static EtherChannel
`802.3ad` / LACP (4)	Negotiated aggregation	Yes	Per flow	LACP required
`balance-tlb` (5)	Transmit load balancing	Yes	TX only	None required
`balance-alb` (6)	TLB + RLB	Yes	Both	None required

Configuring a bond with LACP

In /etc/network/interfaces:

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto bond0
iface bond0 inet manual
    bond-slaves eno1 eno2
    bond-miimon 100
    bond-mode 802.3ad
    bond-lacp-rate fast
    bond-xmit-hash-policy layer3+4

Reload with ifreload -a, and read the result rather than assuming it. Note that the edit goes to /etc/network/interfaces.new when made through the GUI; the lesson on applying network changes covers the staging model and the rollback timer that belongs around any bond change.

Service impact possibleapply, then verify the bond actually aggregated— The three fields that matter are in the output: the aggregator ID must be the same for both slaves, MII Status must be up on both, and Churn State must be none. A bond that is 'up' with churned slaves is a bond running on one link.

ifreload -a

cat /proc/net/bonding/bond0

# the three things to look for, extracted
grep -E 'MII Status|Aggregator ID|Churn State|Partner Mac' /proc/net/bonding/bond0

Read-only / Safe— Illustrative. Two slaves sharing Aggregator ID 1 with no churn is a working LACP bond. Different aggregator IDs means the switch put the ports in different bundles.

$ grep -E 'MII Status|Aggregator ID|Churn State' /proc/net/bonding/bond0

MII Status: up
Aggregator ID: 1
MII Status: up
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
MII Status: up
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none

Illustrative output

A bond that is 'up' with two aggregator IDs is running at half its bandwidth and nobody notices

ip link reports the bond as UP. Traffic flows. Throughput tests from a single client look fine, because a single flow uses one link anyway.

The tell is in /proc/net/bonding/bond0: each slave has an Aggregator ID, and in a working LACP bond they are the same. Two different IDs means the switch has placed the two ports in two separate bundles, so only one of them is carrying traffic and the other is standing by as its own single-port aggregation.

The cause is almost always on the switch: the ports were configured into different port-channels, or one was never added. The host side is correct and will keep insisting so.

This costs you half the bandwidth you paid for, and — more importantly — it costs you the failover you thought you had, because the standby aggregator is not carrying the same VLANs or MAC learning state.

VLAN-aware bridges

A VLAN-aware bridge allows multiple VLANs to share a single bridge + bond, with the bridge handling the 802.1Q tagging instead of the host creating a subinterface per VLAN.

auto vmbr0
iface vmbr0 inet manual
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 10 20 30 100

VMs attached to vmbr0 specify their VLAN tag in the GUI/CLI; the bridge handles the tag.

A VLAN missing from `bridge-vids` fails for guests and works for the host

bridge-vids is an allow-list. A VLAN not in it is dropped by the bridge, and the failure is selective in a way that misleads.

The host’s own interfaces — the management IP, a VLAN subinterface you created directly — do not traverse the bridge’s VLAN filter the way a guest’s tap device does. So the operator tests from the node, reaches the VLAN gateway, and concludes the trunk is fine. The guest on the same VLAN gets nothing, and the investigation goes to the guest.

bridge vlan show is the command that settles it: it lists, per port, which VLAN IDs are permitted. A tap device whose VLAN is absent is the answer.

Adding a VLAN means editing bridge-vids and applying — and note that ranges are written bridge-vids 2 4 100-200, so a range typed with a comma is silently a different set than intended.

Read-only / Safewhich VLANs does each bridge port actually allow?— Read-only. The per-port list is what the kernel is enforcing, as opposed to what /etc/network/interfaces says it should be. A running bridge that has drifted from its configuration shows up here and nowhere else.

bridge vlan show
bridge -d link show | grep -A2 vmbr0
grep -A8 'iface vmbr0' /etc/network/interfaces

MTU and jumbo frames

The default Ethernet MTU is 1500 bytes. Jumbo frames (9000) can improve storage throughput but only work if every device on the path agrees.

Path component	MTU setting
Physical NIC	Driver default or configured (`ip link set eno1 mtu 9000`)
Bond	Must support the higher MTU (most do)
Bridge	Inherits from lowest member
VM guest	Must be configured in the guest OS
Physical switch	Must allow jumbo frames on the port
Storage target	NFS, Ceph, iSCSI must all agree

TARGET_IP=192.0.2.30
ping -M do -s 8972 -c 3 "$TARGET_IP"

If the 8972-byte ping (which is 9000 minus IP+ICMP headers) succeeds, the path supports jumbo frames.

Switch configuration

The switch side must match:

Proxmox	Switch
Bond with LACP	Port-channel / EtherChannel with LACP active
VLAN trunk	Trunk port with allowed VLANs
Access VLAN	Access port with PVID
Jumbo frames	MTU 9216 on the port (9216 = 9000 + headers)

GUI walkthrough

Node → Network lists every interface, bond, and bridge. Create operations:

Create → Linux Bridge: choose ifup/ifdown, IP/CIDR/gateway, bridge ports, VLAN awareness, port for management.
Create → Bond: choose bond mode, slaves, monitoring interval, LACP rate.
Create → VLAN: choose parent interface (a bond or bridge), VLAN ID, optional IP.

Edits to a running network device are applied with ifreload -a. Some changes (like adding a slave to a bond) require a brief interruption.

CLI walkthrough

ip -br -d link show

bridge vlan show

ip -d link show eno1 | grep mtu

Production considerations

Common mistakes

Setting MTU 9000 on one side and 1500 on the other.
Using balance-tlb instead of LACP because “we don’t want to coordinate with the network team.” TLB has asymmetric performance and surprises during failover.
Forgetting that VLAN-aware bridges do not require per-VLAN subinterfaces but do require the bridge to be told which VLANs exist (bridge-vids).
Mixing tagged and untagged traffic on the same bridge port.

Key takeaways

LACP (bond-mode 802.3ad) is the production default for bonds.
A bond being UP is not evidence it aggregated. Check that both slaves share one Aggregator ID and that Churn State is none.
VLAN-aware bridges scale better than per-VLAN bridges, and bridge-vids is an allow-list — a missing VLAN breaks guests while the host keeps working.
MTU must match end-to-end. Mismatches cause silent failures.