Corosync architecture - the cluster membership layer

Corosync is the cluster membership layer for Pacemaker. It provides node membership, quorum, and reliable messaging. This lesson covers how it works.

What Corosync does

Corosync is the foundation of a Pacemaker cluster:

• Membership: which nodes are in the cluster.
• Quorum: deciding who can make decisions.
• Messaging: reliable communication between nodes.
• Membership events: reporting node loss upwards.

Pacemaker sits on top of Corosync. Corosync provides the infrastructure; Pacemaker provides the resource management.

Note what is not in that list: Corosync does not fence. It reports that a node has left the membership; Pacemaker’s fencer (pacemaker-fenced) decides whether that warrants STONITH, picks the device, and executes it. The distinction matters during an incident, because it tells you where to look: a membership problem is a Corosync and network question (corosync-cfgtool -s, corosync-quorumtool), while a fence that did not happen or did not work is a Pacemaker and fence-agent question (pcs stonith config, the fencer’s log).

Architecture

Nodes:
  +-----+
  |  A  +----+
  |     |    |
  +-----+    +------+       +-----+
            | core |       |  D  |
  +-----+    |     |       +-----+
  |     |    +------+
  |  B  +----+
  |     |
  +-----+

Each node runs:
- corosync: membership, quorum, messaging
- pacemaker: resource management, decisions
- stonith: fencing

Transport

Corosync 3 offers three transports, and knet is the default:

• knet: the modern transport. Multiple links with failover, encryption and compression built in. Use this.
• udpu: UDP unicast. Peers come from the nodelist.
• udp: UDP multicast. The traditional transport, and the only one that uses bindnetaddr and mcastaddr.

The transport is configured in /etc/corosync/corosync.conf:

totem {
    version: 2
    cluster_name: mycluster
    transport: knet
    crypto_cipher: aes256
    crypto_hash: sha256
    token: 3000
}

nodelist {
    node { ring0_addr: 10.0.0.11  name: node1  nodeid: 1 }
    node { ring0_addr: 10.0.0.12  name: node2  nodeid: 2 }
    node { ring0_addr: 10.0.0.13  name: node3  nodeid: 3 }
}

quorum {
    provider: corosync_votequorum
}

logging {
    to_logfile: yes
    logfile: /var/log/corosync/corosync.log
    timestamp: on
}

Test it before restarting anything:

sudo corosync -t     # "Test configuration and then exit"

Quorum algorithm

Corosync uses a quorum algorithm (default: majority). For a 3-node cluster, quorum is 2; for a 5-node cluster, quorum is 3.

Quorum is calculated continuously as nodes join and leave. A node that cannot reach the others is marked as not-a-member by the partition that can still see each other.

Be precise about what happens next, because the loose version of this sentence teaches the wrong mental model. The isolated node does not fence itself. It notices it is in the minority, loses quorum, and applies no-quorum-policy — normally stop, so it stops its resources. The majority partition is what fences it, and it must, because “I stopped my resources” is a claim the majority cannot verify: a node wedged in D-state or a hung hypervisor makes no decisions at all while still holding a filesystem mounted.

The one case where a node does act against itself is SBD with a hardware watchdog: the watchdog resets the node when it fails to service it, which is a self-fence and does not need the majority to reach it. That is precisely why SBD exists for clusters with no usable power fencing.

Membership changes

When a node joins:

1. The new node sends a join message.
2. Existing members vote to accept.
3. If accepted, the new node is added to the membership.
4. All members see the new membership.

When a node leaves:

• Graceful: the node sends a leave message.
• Ungraceful: the node stops responding. Corosync detects the loss after a timeout.

The cluster adjusts membership and may trigger a transition (e.g. failover if the leaving node was the primary).

Common issues

• Network partition: nodes cannot reach each other. The minority is fenced.
• Slow network: token loss; retransmissions; performance issues.
• Misconfiguration: token lost because of an asymmetric network.

Verify Corosync health:

corosync-cmapctl
corosync-quorumtool
pcs status

Knowledge check