Cluster network design - management, application, and storage · Linux

A cluster network is more than “connect the nodes”. It is multiple networks with different purposes, redundancy, and failure domain separation. This lesson covers the design.

The four cluster networks

A production cluster typically has four separate networks:

Management: out-of-band (BMC, IPMI). Used for STONITH, firmware updates, console access.
Application: the network the application listens on. Load balancer, user traffic.
Storage: the SAN, Ceph, or DRBD replication network. High throughput, low latency.
Heartbeat: cluster membership (Corosync, Keepalived). Low latency, high reliability.

Each network serves a different purpose. Separating them avoids interference and provides failure domain isolation.

Why separate

If all four share one network:

A storage spike (e.g. backup) affects application traffic.
A broadcast storm blocks STONITH, leaving the cluster unable to fence.
An application misconfiguration floods the network, affecting storage replication.

Separation avoids this. Each network is dedicated.

VLAN separation is not redundancy

A VLAN tag separates traffic classes. It does not separate failure domains. One NIC, one cable and one switch port carry every VLAN configured on them, so a single failure takes all four networks with it. A VLAN tag does not partition bandwidth either, so the storage spike described above still starves application and heartbeat traffic on a shared link.

Minimum production baseline:

At least two physical NICs per host, cabled to two switches.
Bond them. Use 802.3ad only when the two switches are MLAG or vPC peers; otherwise use active-backup.
Put the VLANs on top of the bond, never on the individual slaves.
Give Corosync two independent links rather than one bonded link. See the next section for why.

Full physical separation is better still:

A dedicated NIC pair per traffic class.
Each pair on separate physical switches.
Each switch with its own uplink.

The cost is higher but the failure domain is cleaner. Where storage and heartbeat must share a link, apply egress shaping so a replication burst cannot delay Corosync tokens enough to drop membership.

Heartbeat network

The heartbeat network is for cluster membership. It must be:

Low latency (Corosync uses UDP, sub-second).
High availability (separate from other traffic).
Direct connectivity between all cluster nodes.

A separate VLAN or physical network is standard. Isolation alone is not enough: an isolated heartbeat network that exists only once is still a single point of failure. The redundancy comes from Corosync itself.

Corosync redundant links

Corosync 3 uses the knet transport and supports up to eight links per node. Configure at least two, on genuinely different physical paths, so losing one network costs you a link rather than your membership.

Declare both addresses per node when you build the cluster:

sudo pcs cluster setup mycluster \
  node1 addr=10.3.0.11 addr=10.0.0.11 \
  node2 addr=10.3.0.12 addr=10.0.0.12 \
  node3 addr=10.3.0.13 addr=10.0.0.13 \
  transport knet link linknumber=0 link linknumber=1

The first address on each node becomes link 0, the second becomes link 1. In corosync.conf the same thing is written as one ring0_addr and one ring1_addr per node:

nodelist {
  node { ring0_addr: 10.3.0.11  ring1_addr: 10.0.0.11  name: node1  nodeid: 1 }
  node { ring0_addr: 10.3.0.12  ring1_addr: 10.0.0.12  name: node2  nodeid: 2 }
  node { ring0_addr: 10.3.0.13  ring1_addr: 10.0.0.13  name: node3  nodeid: 3 }
}

totem {
  transport: knet
  link_mode: passive
}

Link 0 should be the dedicated heartbeat network. Link 1 should be a different physical path, commonly the management network. link_mode: passive sends on the highest-priority connected link and fails over when it goes down; link_mode: active sends on all links at once, which costs bandwidth but removes the failover delay.

Verify both links after every change. A silently down link 1 gives you the configuration of redundancy and none of the behaviour:

sudo corosync-cfgtool -s

Every link must report connected. Treat a link in status disconnected as an outage of the same severity as a failed disk in a mirror, because that is exactly what it is.

Link redundancy interacts with token timing. Corosync declares a node lost after token milliseconds plus token_retransmits_before_loss_const retransmissions. With link_mode: passive the failover to link 1 must complete inside that window, so do not shorten token below the default (1000 ms, extended automatically for larger clusters) without measuring the failover first. Shortening the token to “detect failures faster” is the classic way to turn a survivable link flap into a fencing storm.

Storage network

The storage network is for replication (DRBD, Ceph) and shared storage (SAN, NFS). It must be:

High throughput (Ceph: 10 Gbps+).
Low latency (SAN: FC or fast iSCSI).
Reliable (low packet loss).

Often a separate physical network with its own switches.

Management network

The management network is for out-of-band access:

IPMI / iDRAC / iLO.
Serial console.
Firmware updates.
STONITH (often).

Isolated from the application network. Often on a separate VLAN or physical network.

Failure domain considerations

Each network is a failure domain:

One switch fails: hosts on that switch are unreachable via that network. Other networks may be fine.
One cable is cut: similar.
One NIC fails: that network is down for the host.

To survive:

Dual NICs (bonded or separate paths).
Dual switches (MLAG, stacking, or independent).
Dual uplinks (to different routers or core switches).
Two Corosync links on independent paths, verified with corosync-cfgtool -s.

The last item is the one operators most often skip. Bonding protects the IP address; it does not protect membership when the failure is above the bond, such as a VLAN removed from a trunk or an MTU change that black-holes the larger Corosync packets. A second link on a different path does.