Corosync and pmxcfs

Why this matters in production

A Proxmox cluster is two independent pieces of machinery stacked on top of each other, and when something goes wrong the first useful question is which one broke.

• Corosync answers who is in the cluster right now, and do we have a majority.
• pmxcfs answers what does the cluster believe, and may I change it.

pmxcfs depends on corosync; corosync knows nothing about pmxcfs. That direction of dependency is the diagnostic shortcut: a corosync problem always produces pmxcfs symptoms, but a pmxcfs problem does not affect corosync. If pvecm status is healthy and /etc/pve is misbehaving, you are looking at the upper layer. If pvecm status is unhealthy, nothing you do to the upper layer will help.

Corosync — membership, quorum and messaging

Corosync provides three things and no more:

• Cluster membership. Which nodes are currently reachable and participating.
• Quorum. Whether the current membership constitutes a majority, computed by the corosync_votequorum provider.
• Ordered messaging. A guarantee that a message sent to the group is delivered to every member in the same order.

That third one is easy to skip past and it is the reason the whole design works. pmxcfs does not implement replication; it implements applying changes in the order corosync delivers them, which is a much smaller problem.

Since Proxmox VE 6.0, corosync uses Kronosnet (knet) as its transport, over UDP unicast. Multicast is no longer required — a detail worth knowing because a great deal of older Proxmox material online assumes multicast and tells you to configure switches accordingly.

Network requirements

# pvecm status

Cluster information
-------------------
Name:             prod-cluster
Config Version:   7
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Wed Aug 12 11:04:38 2026
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000001
Ring ID:          1.1a4
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Membership information
----------------------
  Nodeid      Votes Name
0x00000001          1 10.0.0.11 (local)
0x00000002          1 10.0.0.12
0x00000003          1 10.0.0.13

set -euo pipefail

# Per-link status for this node. On knet this includes extended statistics.
corosync-cfgtool -s

# Every node with the status of each of its links. This is the command that
# shows you a second link that is configured and not actually working.
corosync-cfgtool -n

# The runtime configuration map, including live counters.
corosync-cmapctl | grep -E 'members|totem\.|runtime\.votequorum'

# What corosync itself said when the membership last changed.
journalctl -u corosync --since '1 hour ago' --no-pager

pmxcfs — the cluster filesystem

pmxcfs is a FUSE filesystem mounted at /etc/pve. Every configuration file that describes the cluster lives there, and every one of them is the same on every node.

The documentation describes it as “a database-driven file system for storing configuration files, replicated in real time to all cluster nodes using corosync”, and each part of that sentence has an operational consequence:

• Database-driven. The backing store is a SQLite database at /var/lib/pve-cluster/config.db, with required permissions 0600. What you see under /etc/pve is a view of that database, not files on a disk.
• A copy resides in RAM, which imposes a hard ceiling: 128 MiB for the entire tree.
• Replicated in real time using corosync, which means writes require quorum and reads do not.

It is not a normal filesystem

The POSIX compatibility limitations are documented and they surprise people who try to treat /etc/pve as an ordinary directory:

The virtual status files

pmxcfs exposes several read-only JSON files that are generated rather than stored. They are the correct way to inspect cluster state programmatically, and they are frequently the fastest answer to a question.

# cat /etc/pve/.vmlist

{
"version": 41,
"ids": {
"100": { "node": "pve-01", "type": "qemu", "version": 12 },
"101": { "node": "pve-02", "type": "qemu", "version": 9 },
"200": { "node": "pve-03", "type": "lxc", "version": 4 }}
}

set -euo pipefail

# Is the daemon healthy? pmxcfs runs as the pve-cluster service.
systemctl status pve-cluster --no-pager

# Membership as pmxcfs sees it, which should agree with corosync.
cat /etc/pve/.members

# File versions, for detecting what changed.
cat /etc/pve/.version

# Recent cluster log entries.
cat /etc/pve/.clusterlog

# The backing database. Size it against the 128 MiB ceiling.
ls -lh /var/lib/pve-cluster/config.db
du -sh /etc/pve

# Confirm the mount is what you think it is.
findmnt /etc/pve

What lives in pmxcfs

Attributing a symptom to a layer

Configuration: corosync.conf

corosync.conf lives at /etc/pve/corosync.conf, which means it is inside the filesystem whose availability depends on it. That circularity is the single most important thing to understand about editing it, and it has its own lesson — the deep dive at the end of this part covers the editing procedure, multi-link configuration and the totem parameters in detail.

Two facts to carry forward from here:

• config_version must be increased on every edit. Corosync uses it to decide whether a configuration it has been handed is newer than the one it is running. An edit without a version bump is ignored.
• There is a local copy at /etc/corosync/corosync.conf. Corosync reads that one at start-up; the copy in /etc/pve is the cluster’s master and is distributed to the local copies. If corosync will not start, the local file is what you edit, because /etc/pve will not be available.

Production considerations

Common mistakes

• Editing corosync.conf without increasing config_version. The edit is silently ignored.
• Restarting corosync everywhere at once, risking a fence storm.
• Assuming multicast is required. Corosync has used knet over unicast since PVE 6.0.
• Blocking UDP 5405–5412 or TCP 22 between nodes with a host firewall.
• Using /etc/pve as a replicated file share, against a 128 MiB cluster-wide ceiling.
• Believing priv/ is node-local. It is replicated like everything else.
• Moving a guest configuration off a node that is partitioned rather than dead.
• Configuring a second corosync link and never testing failover.

Key takeaways

• Corosync provides membership, quorum and ordered messaging; pmxcfs uses that ordering to keep /etc/pve identical everywhere.
• The dependency runs one way, which makes pvecm status the first command for almost every cluster symptom.
• Corosync uses knet over UDP unicast on ports 5405–5412, plus TCP 22 for cluster actions. It needs low latency, not bandwidth.
• pmxcfs is a FUSE view of a SQLite database, capped at 128 MiB for the whole cluster, with real POSIX limitations.
• The virtual files .version, .members, .vmlist, .clusterlog and .rrd are the supported way to inspect state.
• Everything under /etc/pve is replicated, including priv/, and the directory a guest configuration sits in is its ownership record.

Knowledge check