Redundant interfaces and switches - the cluster network resilience

A single network path is a single point of failure. For HA clusters, the network itself must be redundant: dual NICs, redundant switches, and dual uplinks. This lesson covers the design.

NIC bonding

Linux bonding aggregates multiple NICs into a single logical interface:

# /etc/network/interfaces (Debian)
auto bond0
iface bond0 inet static
    address 10.0.0.10/24
    gateway 10.0.0.1
    bond-slaves eth0 eth1
    bond-mode active-backup      # or 802.3ad (LACP)
    bond-miimon 100

Modes:

• active-backup: one NIC active, the other standby. Simple. No switch configuration needed.
• balance-xor: hash-based load balancing. Requires a static (non-LACP) EtherChannel / port-group on the switch. Without it the switch sees the host MAC on two ports and will flap.
• 802.3ad (LACP): link aggregation. Requires switch support; standard for production.
• balance-tlb / balance-alb: transmit / transmit-receive load balancing. No switch configuration needed.

The dividing line is simple. A mode is switch-free only if the host’s MAC address leaves by one port at a time. active-backup qualifies: one slave transmits, the rest are idle. balance-tlb and balance-alb qualify too, because they hand each peer a different slave MAC rather than spraying one MAC across ports.

balance-xor does not qualify. It hashes each frame to a slave, so the same source MAC egresses on both ports. To the switch that looks like a host that has physically moved, over and over. The switch logs MAC flapping, relearns the address on every frame, and on many platforms rate-limits or err-disables the port. A static port-group tells the switch those ports are one link, which is what makes the behaviour legal.

For most production, 802.3ad (LACP) with the switch configured for LACP is the right choice.

Switch redundancy

A single switch is a single point of failure. For HA:

• Stacking: multiple switches act as one logical switch. Example: Cisco StackWise, Arista MLAG.
• MLAG: Multi-Chassis Link Aggregation. Two switches act as one, with LACP across them.
• Dual uplinks: each host connects to two switches. LACP across both is only valid if those switches present a single LACP System ID - see below.

MLAG or stacking is the standard for production HA clusters.

Dual uplinks

Each host has two uplinks to two different switches:

Host eth0 ----- Switch A
Host eth1 ----- Switch B

Switch A and B are stacked or MLAG peers.

If one switch fails, the host traffic goes through the other. If one NIC fails, the other takes over (with bonding).

That last line only holds if Switch A and Switch B really are stacked or MLAG peers. If they are two independent switches, the same cabling with 802.3ad gives you one live link and one dark one, and the “switch failure” test passes or fails depending on which switch you happen to power off. Test both.

Test the design

After deploying, test:

• Pull a cable: traffic continues via the other path.
• Fail a switch: hosts continue to communicate. Fail each switch in turn, not just one.
• Fail a NIC: bonding takes over.
• Confirm both slaves share one Aggregator ID in /proc/net/bonding/bond0 before you call an 802.3ad bond redundant.

If any test fails, the design is not robust.

Knowledge check