Latency, jitter and cluster membership

Every fence that begins with “the network had a blip” ends here. Corosync decides a node is dead using a timer, and the timer is measured against a network you share with other things. This lesson is about that timer: what it actually is, what delays it, and what happens when you try to make it shorter.

The token budget

Totem passes a token around the ring. A node that does not receive the token within the token interval assumes it was lost and asks for a retransmission. After token_retransmits_before_loss_const failed attempts - four by default - the ring is declared broken and a new membership is formed without the silent node.

The default token is 1000 ms, and it grows with cluster size. token_coefficient, 650 ms by default, is added for each node beyond the second:

effective token = token + (nodes - 2) * token_coefficient

So a three-node cluster budgets 1650 ms, a five-node cluster 2950 ms, and a nine-node cluster 5550 ms. Larger rings take longer to notice a failure, on purpose - more nodes means more chances for one of them to be briefly busy.

Read the running values rather than the file, because the file may not be what is loaded:

sudo corosync-cmapctl | grep -E 'runtime\.config\.totem\.(token|consensus)'

consensus, the time allowed to agree on a new membership, is derived from the token - the default is 1.2 times it - and Corosync refuses configurations where it is too small.

Note what the token budget is not: it is not the time from failure to service recovery. After the membership change, Pacemaker still has to run a transition, fence the lost node, wait for the fence to confirm, and start the resources elsewhere. The token is the first term in that sum and usually the smallest.

Why jitter beats latency

The token protocol does not care about your average round trip. It fires on a single interval being exceeded. What matters is the tail.

A link with a 0.15 ms mean and a 400 ms occasional spike is far more dangerous than a link with a steady 8 ms. The first one looks superb on every dashboard and will drop a token; the second one looks mediocre and never will.

This is why the std-dev and max columns of a latency test are the interesting ones, and why a test has to run long enough to catch the spike at all. A sixty-second ping is a measurement of the quiet minute you chose.

$ ping -c 3000 -i 0.2 -q 192.0.2.12

--- 192.0.2.12 ping statistics ---
3000 packets transmitted, 3000 received, 0% packet loss, time 599840ms
rtt min/avg/max/mdev = 0.081/0.147/612.394/11.208 ms

Where the jitter comes from

A shared link carrying storage. The commonest cause by a distance. A Ceph backfill, a DRBD resync, a backup window, or a single large rsync fills the switch’s egress queue, and the Corosync frame waits behind it. The link is not saturated on average; it is saturated for the 300 ms that matter.

Buffer bloat. Deep switch and NIC queues absorb a burst instead of dropping it, which is good for throughput and terrible for a latency-sensitive protocol. A queue that holds half a second of traffic adds half a second of delay to whatever is behind it.

Virtualisation. A cluster node that is itself a VM inherits its hypervisor’s scheduling. Snapshot and backup operations that briefly freeze a guest - seconds, in some products - look exactly like a dead node to its peers. So does heavy CPU steal on an oversubscribed host, and so does live-migrating the node.

Power management. Deep C-states and aggressive frequency scaling add hundreds of microseconds to interrupt handling, and more on some platforms. Rarely enough on its own; frequently enough to be the difference on a marginal link.

The node being too busy to run Corosync. Corosync normally runs at real-time priority for exactly this reason. Confirm it rather than assume it:

chrt -p "$(pidof corosync)"

Reading the evidence

Corosync says what happened, in order:

sudo journalctl -u corosync --since '2026-08-11 00:55' --until '2026-08-11 01:10'

The sequence to look for is a token-loss message naming the interval that was exceeded, followed by a processor-failure message, followed by a new membership being formed with fewer members. Pacemaker’s log then picks it up with a fence.

Check whether one link or all of them degraded:

sudo corosync-cfgtool -s

A cluster with two links where only link 0 flapped tells you the failure is specific to that path. A cluster where both flapped together tells you they are not as independent as the diagram says.

Fixing it in the right order

1. Separate the traffic. A dedicated physical path for membership is the fix that works without tuning anything. Everything below is a compromise for when that is not available.

2. Shape or prioritise. If the heartbeat must share a link, stop the bulk traffic from monopolising the queue. Replacing the default queueing discipline with fq_codel gives small, sparse flows - which is exactly what Corosync is - preferential treatment under load:

sudo tc qdisc replace dev eth1 root fq_codel
tc -s qdisc show dev eth1

For replication specifically, capping the sender is more direct: c-max-rate on a DRBD resource, or the recovery profile on a Ceph cluster, as covered in those lessons.

3. Fix the platform. Reschedule backups off the heartbeat path. Exclude cluster nodes from snapshot-based backup, or use a method that does not freeze the guest. Disable deep C-states on nodes that are marginal.

4. Only then, raise the token. In environments with genuinely high jitter - virtualised clusters, stretched clusters, anything crossing a WAN - a longer token is the correct configuration, not a workaround. Raising it trades detection speed for stability, and on a link with 600 ms spikes that is a trade worth making.

Changing the token safely

$ sudo pcs cluster sync
sudo pcs cluster reload corosync
sudo corosync-cmapctl | grep runtime.config.totem.token

runtime.config.totem.token (u32) = 5000

The verification step is not optional. Some totem parameters apply on reload and some require Corosync to restart, and the difference varies by version. If the running value has not changed, restart Corosync one node at a time, waiting for the cluster to be fully healthy between nodes:

sudo pcs cluster stop node2
sudo pcs cluster start node2
sudo pcs status
sudo corosync-cfgtool -s

Never restart Corosync on more than one node at a time. A two-node restart on a three-node cluster is a quorum loss you inflicted deliberately.

The numbers to know for your cluster

Write these down for each cluster you run, and re-measure after any network change:

• Effective token timeout, from runtime.config.totem.token.
• Observed p99 and maximum round-trip on each Corosync link, from a run of at least ten minutes.
• The ratio between them. A maximum observed latency above roughly a tenth of the token budget is a cluster living on its margin.
• Measured time from node power-off to service available elsewhere, broken down into token, transition, fence and start.

Knowledge check