When one cluster becomes two

Everything in this course up to here has assumed one cluster. One /etc/pve, one quorum, one HA manager, one storage configuration, one set of users. That assumption is doing a lot of work, and it holds for most estates for a long time.

This part is about what happens when it stops holding. Not because you ran out of a documented limit — you will not — but because a single cluster started being the wrong shape for the organisation around it.

What Proxmox actually documents about size

Start with the honest version, because the internet is full of confident numbers that are not in the documentation.

The Cluster Manager chapter states that there is no explicit limit for the number of nodes in a Proxmox VE cluster, and immediately qualifies it: the actual possible node count may be limited by host and network performance, with reports of clusters on high-end enterprise hardware running over 50 nodes in production.

That is the whole of it. There is no supported maximum, no sizing table, and no number a vendor will hold you to. Which means the number is not the interesting question.

The requirements that are specified are more useful, because they are the things that actually break:

The three real ceilings

Ceiling one: corosync and the latency budget

Corosync maintains a totem ring across every node. Every membership change, every /etc/pve write, every quorum decision goes through it. Its cost is not linear in guests or in storage; it is a function of node count and of the worst link in the ring.

The symptom of approaching this ceiling is not “the cluster is full”. It is intermittent: retransmit messages in the journal, nodes briefly leaving and rejoining membership, /etc/pve going read-only for a few seconds during an unrelated network event, and — if HA is enabled — a node fencing itself because its LRM could not renew its lock in time.

pvecm status

corosync-cfgtool -s

journalctl -u corosync --since "-24h" | grep -iE 'retransmit|token|forming' | tail -30

$ corosync-cfgtool -s

Local node ID 2, transport knet
LINK ID 0 udp
addr	= 192.0.2.12
status:
	nodeid:          1:	connected
	nodeid:          2:	localhost
	nodeid:          3:	connected
LINK ID 1 udp
addr	= 198.51.100.12
status:
	nodeid:          1:	connected
	nodeid:          2:	localhost
	nodeid:          3:	connected

Retransmits under load are the early warning. They mean the token is not making its round inside the budget, and they precede a fencing event by weeks — if somebody is looking.

Ceiling two: blast radius

This is the one that actually forces splits, and it has nothing to do with performance.

A Proxmox cluster is a single fault domain for a specific list of things:

• /etc/pve. pmxcfs is a cluster-wide filesystem backed by a shared database, replicated by corosync. A corruption, a bad write or a loss of quorum affects every node at once.
• Quorum. A network partition that leaves a minority without quorum makes /etc/pve read-only on those nodes. Guests keep running; you lose the ability to manage them.
• HA. The HA stack acts cluster-wide. A fencing decision is made by the CRM for the cluster, and a node that loses quorum with HA active will self-fence via its watchdog.
• The upgrade. More on that below.
• The security boundary. Every node in a cluster trusts every other node. Root on one node is root on /etc/pve, which is root on the cluster.

Part II asked you to draw failure domains. The uncomfortable consequence at scale is that a fault domain containing everything is not a fault domain. If your entire estate is one cluster, then a pmxcfs incident, a corosync incident, or a bad cluster-wide change is a total outage — and no amount of node redundancy inside that cluster changes it.

Ceiling three: the upgrade unit

A cluster is upgraded as a unit. Major version upgrades are documented as a node-by-node sequence, but the decision is cluster-wide: the cluster runs mixed versions only during the window, and HA deployments are documented as requiring all nodes on the same version.

That has an organisational consequence people underestimate. If production and the platform team’s staging environment are the same cluster, you cannot validate PVE 9.3 on staging before production sees it — staging is production, from the upgrade’s point of view. The same applies to a customer who has negotiated a change freeze and a customer who has not, when they share a cluster.

Splitting for the upgrade unit is one of the most defensible reasons to have a second cluster, and one of the least discussed.

What the second cluster costs

The hardware cost is usually visible and usually fine. The operational cost is neither.

A second cluster is a second copy of every one of these:

None of these are hard individually. Together they are the reason xxvii-multi-cluster-standards exists as a lesson: the second cluster’s real cost is drift, and drift is paid in incidents where the answer is “it works on the other cluster”.

Choosing the boundary

Split along a line that a failure will respect. The candidates, roughly in order of how well they hold up:

By site. The strongest boundary, because it matches the physical failure domain, the latency budget and usually the power and cooling domain. If two locations exist, the default is two clusters, and stretching one cluster across them needs a specific justification and a measured latency budget.

By upgrade unit and change policy. Production and non-production; a regulated workload with a change freeze and everything else. Defensible because it is the boundary the upgrade already imposes.

By tenancy, where tenancy is adversarial. A hosting provider separating customers who must not be able to reach each other’s control plane. Note the qualifier: for cooperative tenants inside one organisation, pools and SDN (lessons 5 and 6) are usually the right answer and a cluster per tenant is extravagant.

By storage architecture. A Ceph cluster and an iSCSI-SAN estate have different failure modes, different maintenance rhythms and different node shapes. Mixing them in one cluster is legal and often a mistake.

Boundaries that look tidy and are not:

• By department or cost centre. Chargeback is an accounting problem (Part XXII), not a clustering problem. Pools and tags solve it.
• By workload type. “The database cluster” and “the web cluster” usually share a site, a storage system and an upgrade window, so you have paid for a second control plane and bought no isolation.
• By node count, at a round number. Splitting at 16 nodes because 16 felt right produces two clusters with the same blast radius characteristics as one, plus drift.

Splitting is not a live operation

One thing to be explicit about, because it is the commonest wrong assumption in this subject: you cannot move a node from one cluster to another while it holds guests.

A node’s cluster membership is part of its identity. The documented path to change it is to remove the node from the cluster and rebuild it — and pvecm delnode is documented with a warning that a removed node must not be powered back on with its old configuration. Guests therefore travel by one of two routes:

• Backup and restore. Predictable, works offline, costs downtime equal to the restore, and gives you a rollback for free because the source is untouched.
• Remote migration. qm remote-migrate / pct remote-migrate, covered in the next lesson, with the important detail that it is labelled EXPERIMENTAL and copies every disk over the network.

Both are per-guest. There is no bulk “split this cluster in half” operation, and planning a split means planning a guest-by-guest movement schedule.

Key takeaways

• Proxmox VE documents no explicit node limit, with reports of 50+ node clusters in production. The number is not the constraint.
• The documented constraints are: under 5 ms RTT on the cluster network, UDP 5405–5412, up to 8 corosync links, three nodes or two plus a QDevice for quorum, and same-version nodes for HA.
• The three real ceilings are corosync latency and jitter, blast radius, and the upgrade unit. Only the first is a performance question.
• A fault domain that contains everything is not a fault domain. Redundancy inside a cluster does not shrink the cluster’s blast radius.
• The cost of cluster number two is a second copy of storage config, ACLs, tokens, firewall, SDN, backup jobs, HA rules, notifications and certificates — and the drift between them.
• Split by site, by upgrade unit, by adversarial tenancy, or by storage architecture. Do not split by department, workload type, or a round node count.
• Splitting is not live. Nodes cannot be re-parented; guests move by backup/restore or by remote migration, one at a time.

Knowledge check