Bonding and LACP - aggregating links for redundancy and throughput · Linux

Link bonding (also called NIC teaming or link aggregation) combines two or more physical interfaces into one logical interface. The benefits are:

Redundancy: if one link fails, traffic continues on the others.
Throughput: with the right mode, traffic is load-balanced across all links.

Bonding is configured at layer 2; the switch must be configured to match.

Bond modes

Mode	Name	Behaviour
0	balance-rr	Stripes packets of the same flow across links. Causes out-of-order delivery, which TCP reads as loss — a 2x1G balance-rr bond usually delivers less single-stream TCP throughput than one link on its own. Needs a static switch aggregation group. Do not use for TCP workloads.
1	active-backup	One link active, others standby. No switch config needed.
2	balance-xor	Hash-based selection. Needs switch support.
3	broadcast	All packets on all links. Needs switch support.
4	802.3ad (LACP)	Standard LACP. Needs switch configured as LACP.
5	balance-tlb	Adaptive transmit load balancing. No switch config.
6	balance-alb	Adaptive load balancing (TLB + receive). No switch config.

For most production use, mode 1 (active-backup) or mode 4 (LACP) are the right choices. Mode 1 needs no switch configuration and gives no throughput gain. Mode 4 gives redundancy plus aggregate throughput across many flows, but requires switch configuration.

balance-rr is the only mode that stripes a single flow, and that is precisely why it is not on the list. Consecutive frames of one connection go out different links with different queue depths, so they arrive at the receiver out of order. TCP does not distinguish reordering from loss: it fires duplicate ACKs, triggers fast retransmit, and collapses the congestion window. The reordering costs more than the striping gains, which is how a bond built to double the bandwidth ends up slower than the single link it replaced.

LACP does not double the speed of one connection

Mode 4 selects a member link by hashing header fields - the fields named by xmit_hash_policy (layer2, layer2+3, layer3+4, and so on). Every frame of a given flow hashes to the same value, so every frame of a single TCP connection leaves on the same member link. A 2x10G LACP bond does not give one stream 20 Gbps. It gives one stream 10 Gbps, and up to 20 Gbps only when many concurrent flows happen to hash evenly.

That matters because the heaviest traffic on a server is often a single flow: a backup stream, an iSCSI session, DRBD or Ceph replication, a database replica feed. If that is your bottleneck, buy a faster NIC - bonding will not help.

Verify real distribution under production load with per-member counters, not with one iperf3 stream:

ip -s link show eth0
ip -s link show eth1

If one member carries nearly all the bytes, your traffic is too few flows for the hash to spread, or the policy is too coarse (layer2 hashes on MAC only, so all traffic through one router lands on one link). layer3+4 adds ports and spreads better - but never spreads one connection.

Create a bond

ip link add bond0 type bond mode 802.3ad miimon 100 lacp_rate fast
ip link set bond0 up

ip link set eth0 down
ip link set eth1 down

ip link set eth0 master bond0
ip link set eth1 master bond0

ip link set eth0 up
ip link set eth1 up

ip addr add 10.0.0.10/24 dev bond0
ip route add default via 10.0.0.1 dev bond0

Without miimon the bond never notices a dead link

miimon is the MII link-monitoring interval in milliseconds, and the bonding module default is 0 - monitoring disabled. A bond created without miimon (or without the ARP-monitor pair arp_interval + arp_ip_target) does not detect carrier loss. Pull a cable, lose a transceiver, or have the switch port shut down, and the bond keeps queueing frames onto the dead member. That is the exact failure bonding exists to survive.

The declarative examples later in this lesson set it - mii-monitor-interval: 100 in Netplan, MIIMonitorSec=100ms in systemd-networkd. The imperative ip link form must set it too. Confirm it after creation:

cat /sys/class/net/bond0/bonding/miimon    # must not be 0
grep 'MII Polling Interval' /proc/net/bonding/bond0

Beware the false-confidence test. ip link set eth0 down is an administrative down: the kernel raises a NETDEV_DOWN event and bonding fails over regardless of miimon. It therefore passes on a bond that would fail a real cable pull. Only a physical unplug, a transceiver removal, or a switch-side shutdown exercises the carrier-loss path that miimon watches.

Inspect

ip -d link show bond0
cat /proc/net/bonding/bond0

The /proc/net/bonding/bond0 file is the most useful diagnostic. It shows:

Bond mode.
LACP rate (slow or fast).
MII status of each slave.
LACP partner information (when mode 4 is active).

Ethernet Channel Bonding Driver: v6.1.0

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: aa:bb:cc:00:00:01
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: aa:bb:cc:00:00:02
Slave queue ID: 0

If MII Status for any slave is down, the link is broken or the cable is wrong. If Aggregator ID differs between slaves, they are in different aggregators - usually a switch misconfiguration.

LACP considerations

LACP (Link Aggregation Control Protocol, 802.3ad / 802.1AX) negotiates with the switch:

LACP active: the host sends LACP frames. The switch responds.
LACP passive: the host only listens. Switch must be active.
LACP rate slow: LACPDU every 30 seconds.
LACP rate fast: LACPDU every 1 second. Required to detect a failure within ~3 seconds.

For most data centre use, fast LACP is correct. Slow is fine for non-critical links.

ip link set bond0 type bond lacp_rate fast
ip link set bond0 type bond ad_select bandwidth

Configure with Netplan

network:
  version: 2
  renderer: networkd
  bonds:
    bond0:
      interfaces:
        - eth0
        - eth1
      addresses:
        - 10.0.0.10/24
      routes:
        - to: default
          via: 10.0.0.1
      parameters:
        mode: 802.3ad
        lacp-rate: fast
        transmit-hash-policy: layer3+4
        mii-monitor-interval: 100

Configure with systemd-networkd

A .netdev file creates the bond:

# /etc/systemd/network/10-bond0.netdev
[NetDev]
Name=bond0
Kind=bond

[Bond]
Mode=802.3ad
LACPTransmitRate=fast
MIIMonitorSec=100ms
TransmitHashPolicy=layer3+4

A .network file attaches the slaves:

# /etc/systemd/network/11-eth0.network
[Match]
Name=eth0

[Network]
Bond=bond0

Switch configuration

Both member ports must be configured for LACP, and both must belong to the same channel group on the same LACP system. On a single Cisco switch:

interface range GigabitEthernet0/1 - 2
 channel-group 1 mode active

On a single Arista switch:

interface Ethernet1-2
 channel-group 100 mode active

Two independent switches cannot form one aggregation

An 802.3ad bond aggregates only links that terminate on switches presenting one LACP System ID. A single switch gives you that by definition. Two switches give you that only when they are stacked (Cisco StackWise) or run MLAG / vPC (Arista MLAG, Cisco vPC, and the Juniper and Nexus equivalents).

Two independent top-of-rack switches do not. The host receives two different System IDs, builds two aggregators, and selects only one. The second link carries nothing, so what looks like a 2x10G redundant bond is one active link with a slow and sometimes broken failover.

On an MLAG or vPC pair, each peer carries its own member port and both ports carry the same channel identity - on Nexus, the same vpc ID under the port-channel:

! Switch A (vPC peer 1)
interface Ethernet1/1
 channel-group 100 mode active
interface port-channel100
 vpc 100

! Switch B (vPC peer 2) - identical channel and vPC ID
interface Ethernet1/1
 channel-group 100 mode active
interface port-channel100
 vpc 100

If the two switches are independent, use active-backup with miimon and primary instead. It is not aggregation, but it is honest redundancy and needs no switch-side state.

Verify which one you actually built:

cat /proc/net/bonding/bond0

Both slaves must report the same Aggregator ID and the same partner system mac address. Two Aggregator IDs means two independent switches, and half your bandwidth is dark.

A host/switch mode mismatch is not benign

It is tempting to assume a mismatch simply falls back to a plain unaggregated link. It does not. There are three concrete outcomes, and none of them is “the bond stays up”:

Host 802.3ad against a static port-channel (channel-group ... mode on): the switch load-balances across both ports while the host has not aggregated at all. The result is MAC flapping and heavy packet loss.
Host 802.3ad against a switch running lacp suspend-individual (the default on several platforms): the switch never sees a matching partner state, suspends both ports, and the host goes completely offline.
Host active-backup against an LACP port-channel: the switch waits for LACPDUs that never arrive, so the ports stay individual or suspended depending on platform, and failover behaviour is undefined.

Confirm both sides before the change window, and stage the change so the switch side and the host side are never live apart for long. Check /proc/net/bonding/bond0 and the switch logs immediately after each step.