Cluster network topology lab - design and validate

This lab designs and validates a 3-node cluster network with four separated roles: management, application, storage, and a dedicated Corosync heartbeat network.

Tasks

Task 1: Design the topology

For 3 nodes (node1, node2, node3):

• Management network: 10.0.0.0/24, on VLAN 10, for BMC and console.
• Application network: 10.1.0.0/24, on VLAN 20, for user traffic.
• Storage network: 10.2.0.0/24, on VLAN 30, for DRBD or Ceph.
• Heartbeat network: 10.3.0.0/24, on VLAN 40, for Corosync membership. Its own VLAN, not a slice of management or application.

Heartbeat gets a dedicated network because Corosync membership is latency-sensitive, not bandwidth-hungry. Share its path with application or backup traffic and a burst delays the token; the peers conclude the node is dead and fence it. A healthy node power-cycled because a backup job saturated a link is the failure this VLAN exists to prevent.

Corosync then takes a second link on an independent path — conventionally management — so that losing the heartbeat network costs you a link rather than your membership. See linux-cluster-network-design for the knet link configuration.

Each host has two NICs:

• eth0: trunk to switch A (VLAN 10, 20, 30, 40 tagged).
• eth1: trunk to switch B (VLAN 10, 20, 30, 40 tagged).

Switches A and B are stacked, or run MLAG/vPC as a single LACP system. That distinction decides the bond mode in Task 2, so record it in the design document now.

Each node gets its own last octet on every network. Write the address plan down before you touch a host:

Task 2: Build the bond first

Order matters. Build the bond, then build VLANs on top of the bond. Doing it the other way round throws the work away.

# Bond mode depends on the switch answer from Task 1.
#   Stacked / MLAG / vPC (one LACP system)  -> mode=802.3ad
#   Two independent switches                -> mode=active-backup
sudo nmcli con add type bond con-name bond0 ifname bond0 \
  bond.options "mode=802.3ad,miimon=100,lacp_rate=fast,xmit_hash_policy=layer3+4"

sudo nmcli con add type ethernet con-name bond0-p1 ifname eth0 master bond0
sudo nmcli con add type ethernet con-name bond0-p2 ifname eth1 master bond0
sudo nmcli con up bond0

Task 3: Configure the VLAN interfaces on the bond

Run this on each host, changing N to that node’s octet from the Task 1 table. Using the same address on all three nodes is the classic version of this mistake: three hosts claiming 10.0.0.10 produce ARP conflicts on every network at once, and the cluster loses membership in a way that looks like a switch fault.

N=10    # node1=10, node2=11, node3=12

sudo nmcli con add type vlan con-name mgmt    ifname bond0.10 \
  dev bond0 id 10 ip4 "10.0.0.${N}/24"
sudo nmcli con add type vlan con-name app     ifname bond0.20 \
  dev bond0 id 20 ip4 "10.1.0.${N}/24"
sudo nmcli con add type vlan con-name storage ifname bond0.30 \
  dev bond0 id 30 ip4 "10.2.0.${N}/24"
sudo nmcli con add type vlan con-name hb      ifname bond0.40 \
  dev bond0 id 40 ip4 "10.3.0.${N}/24"

sudo nmcli con up mgmt; sudo nmcli con up app
sudo nmcli con up storage; sudo nmcli con up hb
ip -br addr show | grep bond0

None of these VLANs gets a default route. Only the network that reaches the outside world needs one, and giving four interfaces a gateway each is how a cluster ends up sending Corosync traffic out of the application uplink.

Task 4: Configure the switch

One port-channel per server. A port-channel bundles links that terminate on the same peer device, so a channel-group number is per-neighbour, never shared between servers.

! ---- node1 ----
interface Port-channel1
  description node1 bond
  switchport trunk encapsulation dot1q
  switchport mode trunk
  switchport trunk allowed vlan 10,20,30,40
  spanning-tree portfast trunk

interface GigabitEthernet0/1
  description node1 eth0
  switchport trunk encapsulation dot1q
  switchport mode trunk
  switchport trunk allowed vlan 10,20,30,40
  channel-group 1 mode active        ! LACP to node1 only

! ---- node2 ----
interface Port-channel2
  description node2 bond
  switchport trunk encapsulation dot1q
  switchport mode trunk
  switchport trunk allowed vlan 10,20,30,40
  spanning-tree portfast trunk

interface GigabitEthernet0/2
  description node2 eth0
  switchport trunk encapsulation dot1q
  switchport mode trunk
  switchport trunk allowed vlan 10,20,30,40
  channel-group 2 mode active

! ---- node3 ----  Port-channel3 / Gi0/3, channel-group 3, same pattern.
!
! The second member of each host bond lands on switch B. For the
! bond to be 802.3ad, switch B must carry the SAME port-channel
! IDs under vPC/MLAG (for example `vpc 1` under Port-channel1).

Task 5: Validate connectivity

Ping peers, never your own address. A ping to an address the local host owns is answered by the local stack and succeeds with every cable unplugged, so it proves nothing.

# From node1 (octet 10), the peers are 11 and 12.
for peer in 11 12; do
  ping -c3 -I bond0.10 "10.0.0.${peer}"    # management
  ping -c3 -I bond0.20 "10.1.0.${peer}"    # application
  ping -c3 -I bond0.30 "10.2.0.${peer}"    # storage
  ping -c3 -I bond0.40 "10.3.0.${peer}"    # heartbeat
done

# Confirm both slaves are up and note which one is active.
cat /proc/net/bonding/bond0

If Corosync is already running on these nodes, confirm it is using the topology you just built rather than whichever address it happened to resolve first:

sudo corosync-cfgtool -s
# Expect two links per node:
#   LINK ID 0  -> 10.3.0.x  (heartbeat, VLAN 40)
#   LINK ID 1  -> 10.0.0.x  (management, VLAN 10)
# Both must report status "connected".

One link is not redundancy, and a link 0 on the application VLAN is the misconfiguration this whole topology exists to avoid. A silently down link 1 is worse than no link 1, because it gives you the configuration of redundancy without the behaviour.

Record the active slave. Task 6 needs it.

Task 6: Test redundancy

Pull the cable of the active slave on node1 - the one you recorded in Task 5. Pulling an idle slave changes nothing and passes trivially.

# Terminal 1, on node2: continuous peer ping to node1.
ping -D -i 0.2 -I bond0.10 10.0.0.10

# Terminal 2, on node1: watch the bond fail over.
watch -n1 'grep -E "Slave Interface|MII Status|Active Slave" /proc/net/bonding/bond0'

Pass criteria, all three required:

1. /proc/net/bonding/bond0 shows the pulled slave as MII Status: down and a different active slave.
2. The peer ping from node2 lost no more than one packet.
3. journalctl -u corosync shows no lost token.

Reconnect, wait for the slave to return to up, then repeat with the other cable. A test that cannot fail is not a test: if you see zero packet loss and zero change in /proc/net/bonding/bond0, you pulled the wrong cable.

Task 7: Document

CLUSTER NETWORK TOPOLOGY
========================
Nodes: node1, node2, node3
Switches: 2x, MLAG/vPC pair presenting ONE LACP system
NICs: 2x per host (eth0 -> switch A, eth1 -> switch B)
Bonding: bond0, 802.3ad, miimon=100, lacp_rate=fast
         (active-backup instead if the switches are independent)
VLANs:   built on bond0 (bond0.10/.20/.30/.40), never on the slaves

Switch: one port-channel PER SERVER
- Port-channel1 = node1 (Gi0/1 on A, Gi0/1 on B, vpc 1)
- Port-channel2 = node2
- Port-channel3 = node3

Networks and addresses:
- Management: VLAN 10, 10.0.0.0/24 - .10 .11 .12
- Application: VLAN 20, 10.1.0.0/24 - .10 .11 .12
- Storage:     VLAN 30, 10.2.0.0/24 - .10 .11 .12
- Heartbeat:   VLAN 40, 10.3.0.0/24 - .10 .11 .12  (dedicated)

Corosync links (corosync-cfgtool -s, both "connected"):
- link 0: 10.3.0.x  heartbeat  (VLAN 40)
- link 1: 10.0.0.x  management (VLAN 10, independent path)

Failure modes:
- Cable pull on active slave: bond fails over to the other slave
- NIC failure: bonding takes over
- Switch failure: MLAG peer continues; bond keeps one member

Tests (evidence required, not just PASS):
- Connectivity: peer-to-peer ping on all 4 VLANs, output attached
- Corosync: corosync-cfgtool -s showing 2 links, link 0 on 10.3.0.x
- Cable pull (active slave): /proc/net/bonding/bond0 before/after,
  peer ping loss <= 1 packet, corosync token not lost
- Switch failure: same evidence, switch B powered down