Multicast and unicast on the cluster network

Cluster membership is a broadcast problem: every node has to hear from every other node, continuously, with low latency. For years the obvious answer was IP multicast - one packet from a node, delivered by the switch to everyone who asked for it.

That answer is now the wrong one on almost every network, and the reason is not bandwidth. It is that multicast delivery depends on switch state that expires.

What Corosync uses today

Corosync 3 offers three transports, and knet is the default:

• knet - the modern transport. Unicast only. Up to eight links per node, with failover between them, plus encryption and compression.
• udpu - UDP unicast. One packet per peer, peers taken from the nodelist.
• udp - UDP multicast. The traditional transport, and the only one that uses bindnetaddr and mcastaddr.

If you are building a cluster today, the transport question is already answered: knet, because it is the only one that supports the redundant links the cluster network design lesson asks for. The rest of this lesson is about why multicast lost, and what unicast asks of the network in return - because plenty of running clusters are still on udp, and somebody has to migrate them.

The multicast failure that looks like nothing

A managed switch with IGMP snooping enabled watches for IGMP membership reports and forwards a multicast group only to the ports that asked for it. Without snooping, multicast floods every port like a broadcast - wasteful but functional.

Snooping needs a device to periodically send IGMP general queries so hosts re-announce their memberships. That device is the querier, normally a router. On a VLAN with no L3 gateway - which is exactly what a dedicated cluster heartbeat VLAN is - there is often no querier at all.

The sequence is the trap:

1. The cluster is built. Nodes send IGMP joins. The switch learns the group and forwards correctly.
2. Everything works. The cluster is tested, the change is closed.
3. Minutes later, with no query to refresh it, the switch’s group membership ages out.
4. The switch stops forwarding the group. Nodes stop hearing each other.
5. Every node forms its own single-node partition. Depending on quorum and fencing configuration, the result ranges from a frozen cluster to a fence race.

Multicast has two further constraints worth naming. It does not cross a router without multicast routing configured, so a stretched cluster on multicast needs PIM on the network team’s side. And most virtualised and cloud networks do not forward it at all, which is why cloud clusters have always been unicast.

What unicast asks for instead

udpu and knet send one copy of each message per peer. The traffic is therefore quadratic in node count rather than linear, and the CPU cost of encrypting each copy scales the same way.

For the cluster sizes Pacemaker supports - up to around 32 nodes - this is not a problem. Corosync frames are small and the interval is fixed. It is a real consideration only at the top of that range, and if you are there, an eight-node cluster with a stretched design is usually the better answer anyway.

What unicast asks for is much less than multicast: any IP path between the nodes. No switch state, no querier, no multicast routing. That is the whole reason it won.

The peer list is the nodelist. Under udpu and knet it is not documentation - it is how a node learns where the others are:

totem {
    version: 2
    cluster_name: mycluster
    transport: knet
    crypto_cipher: aes256
    crypto_hash: sha256
}

nodelist {
    node { ring0_addr: 192.0.2.11  ring1_addr: 198.51.100.11  name: node1  nodeid: 1 }
    node { ring0_addr: 192.0.2.12  ring1_addr: 198.51.100.12  name: node2  nodeid: 2 }
    node { ring0_addr: 192.0.2.13  ring1_addr: 198.51.100.13  name: node3  nodeid: 3 }
}

quorum {
    provider: corosync_votequorum
}

Testing the network before you build the cluster

omping sends both unicast and multicast probes between hosts and reports loss for each independently. Run it on every node at the same time, naming all the nodes:

$ omping -c 600 -i 1 -q node1 node2 node3

node2 :   unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.083/0.141/0.402/0.031
node2 : multicast, xmt/rcv/%loss = 600/599/0%, min/avg/max/std-dev = 0.091/0.157/0.437/0.034
node3 :   unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.079/0.138/0.386/0.029
node3 : multicast, xmt/rcv/%loss = 600/597/0%, min/avg/max/std-dev = 0.088/0.152/0.421/0.033

Two things make this test worth the ten minutes. It runs long enough to outlive the IGMP ageing window that a two-minute test sails past. And it separates unicast from multicast, so a result where unicast is clean and multicast degrades tells you exactly which transport the network will support.

The avg and std-dev figures are also the input to the next lesson: they are the latency and jitter the membership layer will have to live with.

Ports and firewalls

Under knet, Corosync uses UDP starting at the port configured as mcastport - 5405 by default - with an additional port per configured link. Distribution firewall definitions for high-availability clusters generally open a small range above it rather than a single port; on a firewalld system the high-availability service is the definition to inspect rather than to guess at:

sudo firewall-cmd --info-service=high-availability

The other ports a Pacemaker cluster typically needs are TCP - pcsd on 2224, Pacemaker Remote on 3121, and corosync-qnetd on 5403 if you run a quorum device. Check your distribution’s service definition rather than transcribing a list from a blog post; the ranges have changed across releases.

Verify what Corosync actually bound:

sudo ss -lunp | grep -i corosync
sudo corosync-cfgtool -s

Changing transport is not a reload

Corosync can reload parts of its configuration in place:

sudo corosync-cfgtool -R

The transport is not one of them. Moving a cluster from udp to knet changes how nodes address each other, and a partially migrated cluster is a partitioned cluster: the nodes still on multicast cannot hear the nodes on unicast, and both halves believe the other is dead.

Knowledge check