Bond failure modes — slave down, LACP flap, hash mismatch, MTU mismatch

A bond is a moving system. The kernel driver polls MII every 100ms, the LACP state machine exchanges LACPDU every second, the switch re-hashes flows across slaves, and the host reconfigures around failures. Anything that disturbs the steady state shows up either as a degraded link or as a flap. This lesson walks through the eight failure modes that produce the production pager incidents and the evidence each one leaves in the operational state.

The eight failure modes

flowchart TD
  A[Bond failure] --> B[Slave-layer]
  A --> C[Protocol-layer]
  A --> D[Configuration-layer]
  B --> B1[Slave down]
  B --> B2[Speed mismatch]
  B --> B3[MTU mismatch]
  B --> B4[Inter-switch cable]
  C --> C1[LACP flap]
  C --> C2[Single-side passive]
  C --> C3[Hash mismatch]
  D --> D1[Switch misconfiguration]

The layers are: the physical link (slave up or down), the protocol negotiation (LACP happy or flapping), and the static configuration (mode, MTU, hash policy all match on both sides).

Failure 1 — slave down

A slave dies. The cable is yanked, the SFP loses link, the remote switch port goes down. The bond driver detects the MII status change, removes the slave from the active set, and continues on the remaining slaves.

vyos@vyos:~$ cat /proc/net/bonding/bond0
...
Slave Interface: eth0
MII Status: down
Speed: Unknown
Duplex: Unknown
Link Failure Count: 1

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0

The evidence is in MII Status: down. The Link Failure Count increments by one each time the slave transitions from up to down. The aggregate bandwidth drops to the remaining slaves.

sequenceDiagram
  participant K as Kernel driver
  participant S1 as eth0
  participant S2 as eth1
  participant SW as Switch
  S1->>SW: Link up
  S2->>SW: Link up
  K->>S1: MII poll (every 100ms)
  S1->>K: link ok
  K->>S2: MII poll
  S2->>K: link ok
  Note over S1,SW: Cable yanked
  K->>S1: MII poll
  S1->>K: link down
  K->>K: Remove eth0 from active set
  K->>SW: LACPDU on eth1 only

The remediation:

1. Check the physical link. Is the cable seated? Is the SFP alive?
2. Check the switch port. Is it shutdown or err-disabled?
3. Check the switch side for the error-disable cause. Many switches auto-disable a port on BPDU guard violation or security violation.

Failure 2 — speed mismatch

The two slaves are at different speeds. A 1 Gbps slave and a 10 Gbps slave in the same bond run at the lowest common speed for LACP aggregation. The bond ends up at 1 Gbps aggregate, not 11 Gbps.

vyos@vyos:~$ cat /proc/net/bonding/bond0
...
Slave Interface: eth0
MII Status: up
Speed: 10000 Mbps
Duplex: full

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full

The evidence is in the speed field. The aggregation is the minimum of the slave speeds. The 10 Gbps link is wasted.

The remediation:

1. Check the switch port configuration. Is the 1 Gbps port configured for 10 Gbps? Or vice versa?
2. Check the cable. A wrong cable (Cat 5 instead of Cat 6) can force a port to 1 Gbps.
3. Reconfigure the port or replace the cable.

Failure 3 — MTU mismatch

The bond is configured for MTU 9000 (jumbo) but one slave is limited to MTU 1500. Jumbo frames sent to the MTU-limited slave are dropped at the slave level.

vyos@vyos:~$ ip link show eth0
4: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 ...
vyos@vyos:~$ ip link show eth1
5: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 ...
vyos@vyos:~$ ip link show bond0
3: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 ...

The evidence is the MTU of the slaves. The bond takes the minimum of the slave MTUs unless explicitly configured otherwise.

The remediation:

1. Set the MTU on the slaves explicitly with set interfaces ethernet ethN mtu '9000'.
2. Verify the switch port can handle jumbo frames.
3. Run ping -M do -s 8972 <peer> to confirm end-to-end jumbo connectivity.

Failure 4 — inter-switch cable

The two slaves are on different switches but the switches are not configured as an MLAG pair. The host sees both slaves up but the switches cannot coordinate. The result is unpredictable traffic distribution and a switch-side STP that blocks one port.

Switch A
  port 1: up, STP forwarding
Switch B
  port 1: up, STP blocking

The evidence is asymmetric: one switch sees the port as forwarding, the other sees it as blocking. The host is unaware because MII is fine on both.

The remediation:

1. Configure the switches as an MLAG pair (or stack).
2. Or move both slaves to the same switch.
3. Or accept the asymmetric state if the design tolerates it.

Failure 5 — LACP flap

The LACP state machine flaps. The aggregation comes up, drops, comes back up, drops again. The host bond log shows repeated transitions in and out of the aggregated state.

vyos@vyos:~$ dmesg | tail -20
bond0: link status definitely up for interface eth0
bond0: link status definitely down for interface eth0
bond0: link status definitely up for interface eth0
bond0: link status definitely down for interface eth0

The flapping is usually caused by:

• A duplex mismatch on the slave.
• A speed mismatch on the slave.
• A fibre cable that is starting to fail.
• A switch port that is going into error-disable and being auto-recovered.

The remediation:

1. Check the dmesg log for the exact sequence of events.
2. Check the slave with ethtool ethN for errors and negotiated parameters.
3. Replace the failing physical component.

Failure 6 — single-side passive

LACP is configured but both sides are passive. No one sends LACPDU. The bond never forms.

vyos@vyos:~$ cat /proc/net/bonding/bond0
...
LACP active: off

The evidence is LACP active: off on the host. The switch side will also show LACP not engaged. Both ends are waiting for the other to send first.

The remediation:

1. Set set interfaces bonding bond0 lacp-rate 'fast' (which implies active mode in VyOS).
2. Or set the switch side to mode active.
3. At least one side must be active.

Failure 7 — hash mismatch

The host and the switch compute the hash from different inputs. The host may use layer3+4 (IP + port); the switch may use layer2 (MAC). The two ends select different slaves for the same flow. Conversations break.

The evidence is asymmetric traffic distribution. Switch counters show all flow on one slave; host counters show the same flow on the other slave. The bond is “up” but flows are breaking.

The remediation:

1. Align the hash policy on both sides.
2. On VyOS: set interfaces bonding bond0 hash-policy 'layer3+4'.
3. On the switch: configure the matching port-channel load- balance.

Failure 8 — silent failure (the worst case)

The bond is “up” on both sides. The host sees two slaves up, the switch sees the channel up. But the aggregation is not working and traffic is silently dropped or double-sent.

This is the hardest failure mode to detect. There is no error log, no flap, no down state. The traffic is just wrong.

The evidence is:

• Sub-bandwidth utilisation on the bond aggregate.
• Switch port counters not matching the host counters.
• Spurious TCP retransmits or out-of-order packets.

The remediation:

1. Compare the per-slave counter on the host with the switch.
2. Run a throughput test (iperf3) and compare to the expected aggregate.
3. Reset the bond and the channel on both sides.

Detection playbook

The first command when a bond is misbehaving:

vyos@vyos:~$ cat /proc/net/bonding/bond0

This is the canonical source of truth. Look for:

• MII Status: on each slave — must be up.
• Aggregator ID: on each slave — must be the same non-zero number.
• Link Failure Count: — increments on slave transitions.
• LACP active: — must be on for 802.3ad.
• Speed: on each slave — must match.
• Partner MAC: — confirms the switch is on the other end.

The second command, on the switch:

Switch# show etherchannel summary

This shows the channel state, the protocol, and the bundle status. The P flag on each port means it is bundled.

The third command, on the host:

vyos@vyos:~$ ip -s link show bond0

This shows the per-slave counters. Throughput, errors, drops. The counters should match the switch-side counters within the normal asymmetric margin.

How it fails

The production failure modes the engineer must recognise:

• Slave down. A single slave failure can drop the aggregate bandwidth by half. The remaining slave continues but at reduced capacity.
• LACP flap. Repeated up/down on the aggregation. The bond may fail over entirely if all slaves flap.
• Hash mismatch. Sub-bandwidth utilisation; asymmetric traffic on the slaves.
• MTU mismatch. Dropped jumbo frames; the bond appears up but large packets fail.
• Speed mismatch. Aggregation runs at the lowest common speed; the high-speed slave is wasted.
• Single-side passive. Bond never forms. Both sides wait for the other to send.
• Inter-switch cable. Coordination failure between switches that are not configured as MLAG.
• Silent failure. Bond up but aggregation broken; no obvious error.

Rollback

The recovery from a bad bond failure:

• Verify the bond state with cat /proc/net/bonding/bond0.
• Identify which failure mode applies.
• Apply the remediation for that mode.
• Re-verify with show bonding and ip -s link show bond0.
• Re-verify the switch-side with show etherchannel summary.

For emergency rollback, change the bond to active-backup mode while the underlying issue is being debugged. The single-slave aggregation is less efficient but it is robust.

Production discipline

Cross-course references

The Linux course’s XIX-Linux-NetFoundations covers the kernel bonding driver. The OPNsense course’s VII-OPNsense-Interfaces covers the equivalent LAG failure modes. The Observability course’s XLIX-Observability-NetMetrics covers the monitoring side — counters, dashboards, and alerts.

Quiz