Cluster networking deep dive: Corosync links, totem, and tuning · Proxmox VE

Corosync is the membership and messaging layer that holds a PVE cluster together. When it has problems, every cluster operation stalls: migrations freeze, the GUI hangs, /etc/pve goes read-only, HA decisions do not fire. A large proportion of Proxmox cluster outages trace back to it, and almost all of those are a network problem wearing a corosync costume.

This lesson is the operational anatomy: the configuration file, the editing procedure, multi-link redundancy, the totem parameters that actually exist, and diagnosis.

What corosync does, and what it does not

Corosync runs on every node and provides:

Membership — which nodes are in the cluster. A join or a departure is propagated within a token rotation.
Quorum — the majority-vote algorithm, implemented by corosync_votequorum. pvecm status reads its state from here.
Ordered messaging — reliable delivery to a closed process group, in the same order on every member.

It does not replicate configuration, manage services, or decide where guests run. pve-cluster (pmxcfs) subscribes to corosync for membership and uses the ordered messaging to keep /etc/pve identical everywhere; pve-ha-crm and pve-ha-lrm make the HA decisions. Proxmox does not use Pacemaker, so Pacemaker tools such as crm_mon and pcs are not part of this stack and will not be present.

The configuration file

/etc/pve/corosync.conf is the cluster’s master copy, and because it lives in pmxcfs it is replicated automatically. Each node also keeps a local copy at /etc/corosync/corosync.conf, which is what corosync actually reads at start-up.

That pair is the single most important structural fact in this lesson, and the recovery section at the end depends on it.

A real PVE 9 three-node configuration:

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve-01
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.0.0.11
  }
  node {
    name: pve-02
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.0.0.12
  }
  node {
    name: pve-03
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.0.0.13
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: prod-cluster
  config_version: 3
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

Four things to notice, because older documentation shows something quite different:

The interface block carries only linknumber. Addresses are per node in nodelist, as ring0_addr. There is no bindnetaddr, no mcastaddr and no mcastport — those belong to the pre-knet era.
There is no transport line. knet is the default in corosync 3 and PVE does not override it. pvecm status confirms it with Transport: knet.
secauth: on. Corosync traffic is authenticated and encrypted by default on a PVE cluster; pvecm status shows Secure auth: on. There is nothing to turn on here.
No totem timers are set. PVE ships the corosync defaults, which are correct for a LAN cluster. That is a deliberate choice, not an omission.

Editing it without losing the cluster

Cluster-wide riskthe edit procedure— Cluster-wide risk. A corosync.conf that will not parse takes down the node that reads it, and if you apply it everywhere at once it takes down the cluster. Keep the backup and keep a console open.

set -euo pipefail

# 1. Back up both copies. The local one is your recovery path.
cp /etc/pve/corosync.conf     /root/corosync.conf.bak
cp /etc/corosync/corosync.conf /root/corosync.conf.local.bak

# 2. Edit the copy in /etc/pve - it is the one that replicates.
#    Increase config_version by exactly one as part of the same edit.
nano /etc/pve/corosync.conf

# 3. Verify it parses before it is applied anywhere.
corosync-cfgtool -s >/dev/null && echo 'current instance still healthy'
grep -E 'config_version|linknumber|ring[0-9]_addr' /etc/pve/corosync.conf

# 4. Ask every corosync instance to reload, without restarting anything.
corosync-cfgtool -R

# 5. Confirm the new version is live on every node.
pvecm status | grep -E 'Config Version|Quorate|Total votes'
corosync-cfgtool -n

Adding a second link

A single corosync link is a single point of failure. If it drops for longer than the token timeout the node leaves the membership, and with HA armed it fences itself.

On PVE 9 a second link is two additions: an interface block with linknumber: 1, and a ring1_addr on every node.

nodelist {
  node {
    name: pve-01
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.0.0.11
    ring1_addr: 10.0.1.11
  }
  node {
    name: pve-02
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.0.0.12
    ring1_addr: 10.0.1.12
  }
  node {
    name: pve-03
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.0.0.13
    ring1_addr: 10.0.1.13
  }
}

totem {
  cluster_name: prod-cluster
  config_version: 4
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

Requirements that are not optional:

Every node must have every link. A nodelist where one node lacks ring1_addr is not a valid two-link cluster.
The links must be genuinely independent. Two VLANs on the same physical switch survive a VLAN misconfiguration and not a switch failure, which is the failure you were buying protection against.
Each link needs its own UDP port, allocated from the 5405–5412 range. A firewall permitting 5405 and not 5406 produces a link that is configured and dead.

link_mode controls how the links are used. passive — the PVE default — sends over the highest-priority working link and fails over. active sends over all links simultaneously, which lowers latency and raises load. For almost every cluster, passive is right.

Read-only / Safeproving both links are up, per node— Both links enabled and connected to every peer is the state you want. A link showing enabled but not connected is configured and not working - and it produces no other symptom until the day the other link fails.

# corosync-cfgtool -n

Local node ID 1, transport knet
nodeid: 2 reachable
 LINK: 0 udp (10.0.0.11->10.0.0.12) enabled connected mtu: 1397
 LINK: 1 udp (10.0.1.11->10.0.1.12) enabled connected mtu: 1397

nodeid: 3 reachable
 LINK: 0 udp (10.0.0.11->10.0.0.13) enabled connected mtu: 1397
 LINK: 1 udp (10.0.1.11->10.0.1.13) enabled connected mtu: 1397

Illustrative output

A redundant link is a hypothesis until you have induced the failure

Everything looks correct. pvecm status is green. The configuration is right. And you still do not know whether failover works, because link 0 has carried every packet since the day it was built.

The test is five minutes in a maintenance window:

Confirm quorum is healthy and no HA recovery is in progress.
On one node, administratively down the primary link interface.
From another node, run corosync-cfgtool -n and confirm that node is still reachable, now over link 1 only.
Confirm pvecm status still reports Quorate: Yes with full votes.
Bring the interface back up and confirm link 0 returns to connected.

Do it on one node at a time, and do not do it while the cluster is already degraded. A cluster that passes this test has redundancy; one that has not been tested has a second address in a configuration file.

Totem parameters: what exists, and the real defaults

Most clusters need none of these. They are here so that you can recognise them, and so that you do not copy a WAN-tuning snippet into a LAN cluster because it appeared authoritative on a forum.

Parameter	Default	What it does
`token`	3000 ms	Time without receiving the token before token loss is declared
`token_coefficient`	650 ms	Added to the effective token timeout per node beyond two, when the nodelist has three or more nodes
`token_retransmits_before_loss_const`	4	Retransmission attempts before a new configuration is formed
`consensus`	3600 ms	Time to wait for consensus on membership before starting a new round
`max_messages`	17	Maximum messages a member may send after receiving the token
`knet_ping_interval`	derived from the token timeout	knet link liveness probing interval
`link_mode`	`passive`	How multiple links are used: `passive`, `active` or `rr`
`ip_version`	`ipv4-6` on PVE	Address family preference

Do not tune totem to fix a network problem

The overwhelmingly common reason someone reaches for token is that nodes are dropping out of the membership. Raising the timeout makes the symptom go away, and it does so by making the cluster slower to notice a genuinely dead node — which lengthens every HA recovery, permanently, in exchange for masking a fault.

Work through the actual cause first:

Is corosync sharing a link with backup, migration or storage traffic?
Is there packet loss or duplex mismatch on the path?
Is an MTU mismatch fragmenting corosync packets?
Is a node’s CPU so saturated that the corosync process is not being scheduled?

All four are common, all four are fixable, and all four are cheaper than a permanently degraded failure-detection time. Change totem parameters only when you have a network you cannot change — a genuinely high-latency inter-site link — and understand that you are trading detection speed for tolerance.

Proxmox’s own guidance is that cluster latency should be under 5 ms. A configuration that needs a 30-second token is describing a topology the product does not support well, and a QDevice at a third site is usually the better structural answer.

Diagnosis

Symptoms that point at corosync:

Cluster operations stall — migrations, configuration changes, backups
Nodes appear and disappear from the GUI with no hardware fault
/etc/pve goes read-only intermittently
HA fires when nothing was wrong

Read-only / Safethe corosync diagnostic sequence— Read-only. Every command here is safe on a live cluster, including one that is currently misbehaving.

set -euo pipefail

# The PVE summary: quorum, votes, membership, config version.
pvecm status
pvecm nodes

# This node's links, with knet extended statistics.
corosync-cfgtool -s

# Every node with per-link status. Use -i with -s to narrow to one address.
corosync-cfgtool -n

# Live runtime counters. Retransmits and token loss show up here.
corosync-cmapctl -m stats | grep -Ei 'retrans|token|rx_error|tx_error' || true
corosync-cmapctl | grep -E 'runtime\.votequorum|members'

# What corosync and pmxcfs said as it happened.
journalctl -u corosync    --since '2 hours ago' --no-pager
journalctl -u pve-cluster --since '2 hours ago' --no-pager

Log pattern	Cause	Fix
Repeated `A processor joined or left the membership`	The link is dropping	Switch port, cable, NIC, MTU, or contention
`Token has not been received in N ms`	Latency exceeds the effective token timeout	Fix the network first; tune only as a last resort
`Retransmit List:` growing	Packet loss on the corosync path	Cable, switch, or a saturated shared link
`Configuration version mismatch`	Nodes are running different `corosync.conf`	Increase `config_version` and reload; confirm all nodes agree
`Invalid configuration` at start-up	The file does not parse	Repair the local copy at `/etc/corosync/corosync.conf`

Why corosync-cmapctl sees things pvecm status cannot

pvecm status presents a snapshot of the votequorum state. corosync-cmapctl exposes the configuration map, corosync’s in-memory key-value store, which includes live runtime counters that are cumulative since the process started.

The counters that matter for an intermittent fault are the retransmit and error counts per link. A cluster that looks perfectly healthy right now but has accumulated a large retransmit count over the last three weeks is telling you about a marginal path — a failing optic, an over-subscribed uplink, a duplex mismatch — long before it produces an outage anybody notices.

This is the corosync equivalent of reading SMART attributes on a disk that has not failed yet, and it is worth sampling into your monitoring rather than only looking at it during an incident. pvecm status will be green right up until the moment it is not; the counters will have been climbing for weeks.

Common operational scenarios

Changing a node’s corosync address

# Edit /etc/pve/corosync.conf: change that node's ring0_addr, and increase
# config_version. Then reload cluster-wide rather than restarting.
corosync-cfgtool -R
pvecm status | grep -E 'Config Version|Quorate'
corosync-cfgtool -n

Update /etc/hosts and DNS to match. A node whose corosync address and hostname resolution disagree will work until something resolves the name.

Removing a node

Do not edit the nodelist by hand. pvecm delnode removes the node from the nodelist, increases config_version and adjusts expected votes as one operation, and hand-editing gets one of those three wrong often enough to be worth avoiding.

# From a node that is staying, with the cluster quorate.
pvecm delnode pve-03
pvecm status

The removed node must be reinstalled before it can rejoin — it still holds a replica of the cluster’s secrets from /etc/pve/priv/, which is why the documented guidance is to reinstall rather than repurpose.

What happens if corosync will not start after an edit?

This is the situation the local copy exists for, and knowing the path out of it is the reason to read this section before you need it.

Corosync reads /etc/corosync/corosync.conf at start-up. If that file does not parse, corosync exits, the node has no membership, and pve-cluster mounts /etc/pve read-only — so the master copy you would like to fix is not writable, and on this node it may not be mounted at all.

The recovery is to repair the local file:

Edit /etc/corosync/corosync.conf directly. This is an ordinary file on the node’s root filesystem and it is writable regardless of quorum.
Make it identical on every affected node. The documentation is explicit that the content must match across nodes to avoid a split-brain.
Start corosync, then confirm membership from a peer.
Once the cluster is quorate again, fix /etc/pve/corosync.conf so the master copy matches, and increase config_version.

If the cluster is quorate on the surviving nodes, the master copy in /etc/pve is intact and the local file can simply be copied from a healthy node. If every node is down, you are in the case the pmxcfs lesson covers: bring pmxcfs up in local mode on one node with pmxcfs -l, repair the configuration there, and start the rest.

The thing that makes all of this survivable is having taken the backup in step 1 of the edit procedure. Restoring a known-good file is a thirty-second operation; reconstructing one from memory at 02:00 is not.

Production considerations

A dedicated network for corosync. Not shared with migration, backup or storage. Contention, not bandwidth, is what breaks it.
Two links on independent hardware, tested by inducing a failure.
Leave the totem defaults alone unless you have a measured reason.
Encryption is already on. secauth: on is the PVE default; there is no crypto to enable.
Do not make corosync changes during a rolling upgrade. One variable at a time.
Sample the retransmit counters into monitoring so a marginal link is a trend rather than a surprise.

Common mistakes

Editing without increasing config_version, so the change is ignored.
Restarting corosync on every node at once.
Copying a pre-knet configuration with bindnetaddr, mcastaddr and transport: udpu from old documentation.
Reaching for Pacemaker tools. PVE does not use Pacemaker.
Raising token to silence a flapping link, permanently slowing failure detection to mask a fixable fault.
Computing detection time from token alone, ignoring token_coefficient.
Two links on the same physical switch.
Hand-editing the nodelist to remove a node instead of using pvecm delnode.

Key takeaways

corosync.conf exists twice: the master in /etc/pve and the local copy in /etc/corosync that corosync actually reads. The second is your recovery path.
Increase config_version on every edit, then prefer corosync-cfgtool -R over a restart.
PVE 9 uses knet over unicast; addresses live per node as ringX_addr and the interface block carries only linknumber.
secauth: on is the default — corosync traffic is already authenticated and encrypted.
Real defaults: token 3000 ms, token_coefficient 650 ms per node beyond two, token_retransmits_before_loss_const 4, consensus 3600 ms.
A second link is unproven until you have downed the first one and watched the cluster stay together.

Knowledge check

Why this matters

corosync.conf is a short file that decides whether a set of servers is a cluster. It is also the only configuration file in Proxmox that lives inside the filesystem whose availability depends on it, and every careful thing in this lesson follows from that circularity.

Two habits are worth more than the rest of the content. Back up both copies before editing, because the recovery from a bad configuration is restoring a good one and that takes thirty seconds when you have it. And test the redundant link by breaking the primary, because a link that has never carried a packet is a line in a file rather than a piece of resilience, and the day you find out which one it is will be the day you needed it to be the second.

The rest is mostly restraint. PVE ships corosync defaults for a reason, the defaults are right for a LAN, and the strong instinct to tune timeouts when nodes flap almost always makes the cluster worse in a way that only shows up during the next real failure.