Joining, removing, and replacing nodes · Proxmox VE

Why this matters in production

Adding a node is routine and forgiving. Removing one contains the irreversible steps, and the ordering matters more than the commands do: get it wrong and a node’s guest configurations are gone before you have finished reading the confirmation prompt.

Which command runs where

This is the first thing to get straight, because the two commands look similar and are not interchangeable.

Command	Run on	Purpose
`pvecm create <name>`	The first node	Creates the cluster
`pvecm add <cluster-node>`	The joining node	Joins this node to an existing cluster
`pvecm addnode <node>`	An existing cluster node	Internal — invoked by the join process, not by you
`pvecm delnode <node>`	An existing cluster node, not the one being removed	Removes a node
`pvecm status` / `pvecm nodes`	Any node	Read state
`pvecm qdevice setup` / `remove`	An existing cluster node	QDevice lifecycle

The pairing to remember: you add from the new node and you delete from an old one. Running either from the wrong place is where most of the confusion in this procedure comes from.

Before you join anything

Read-only / Safethe pre-join checklist, as commands— Read-only, and run on the new node. Every one of these has caused a failed join. Checking them takes two minutes; diagnosing a failed join takes considerably longer.

set -euo pipefail
PEER=pve-01.example.com
PEER_IP=192.0.2.11

# 1. Same PVE version as the cluster. A version mismatch is not supported.
pveversion -v | head -3

# 2. Forward and reverse resolution agree, for this node and the peer.
hostname -f
getent hosts "$(hostname -f)"
getent hosts "$PEER"
getent hosts "$PEER_IP"

# 3. Time is synchronised. Corosync will not tolerate meaningful skew.
chronyc tracking | grep -E 'Reference ID|System time|Leap status'

# 4. Reachability on the ports that matter: SSH for the join itself,
#    8006 for the API, and the corosync range for cluster traffic.
nc -z -w3 "$PEER_IP" 22   && echo 'ssh ok'
nc -z -w3 "$PEER_IP" 8006 && echo 'api ok'

# 5. THIS NODE MUST BE EMPTY. Joining destroys its local configuration.
qm list
pct list

A node joining a cluster loses its own configuration

This is the single most expensive mistake available in this lesson, and it is made by people who assume the join merges two configurations.

It does not. The joining node adopts the cluster’s /etc/pve wholesale. Guests configured on it beforehand, its storage definitions, its users and its firewall rules are replaced by the cluster’s.

So the rule is absolute: join empty nodes. A freshly installed node with its network configured and nothing else.

If you need to bring an existing standalone Proxmox host with guests on it into a cluster, the supported shape of that job is:

Back up every guest to a location that is not on that host — PBS, or vzdump to external storage.
Note the storage definitions and any local customisation you will need to recreate.
Join the now-expendable node to the cluster.
Recreate the storage definitions cluster-side.
Restore the guests.

There is no shortcut, and the step people skip is the first one.

Joining

The modern path is through the GUI: on an existing node, Datacenter → Cluster → Join Information, copy the blob, and paste it into Join Cluster on the new node. It carries the address, the fingerprint and the peer list, which removes the three things people mistype.

From the command line, on the new node:

Cluster-wide riskjoining from the CLI— Cluster-wide: the join changes expected votes for every node. Do it while the cluster is quorate and healthy, and one node at a time.

set -euo pipefail
PEER_IP=192.0.2.11

# Run on the NEW node. It will ask for the root password of the peer.
# --fingerprint comes from the join information on the existing cluster.
pvecm add "$PEER_IP" \
--fingerprint 'AA:BB:CC:DD:EE:FF:00:11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33:44:55:66:77:88:99'

# On a multi-link cluster, give this node an address on every link, or it
# joins with one link and the redundancy you think you have is incomplete.
#   pvecm add "$PEER_IP" --link0 10.0.0.14 --link1 10.0.1.14

# --use_ssh forces the older SSH-based join path, which is the fallback when
# the API join fails.

Read-only / Safeverifying the join, rather than assuming it— Read-only. Run all of it, on more than one node. A join that appears to work and left the new node with one link, or without its storage, is the kind of thing that surfaces during the next failure.

set -euo pipefail

# 1. The node is a member and the vote count has increased.
pvecm status | grep -E 'Quorate|Expected votes|Total votes'
pvecm nodes

# 2. Every link is up to every peer - not just link 0.
corosync-cfgtool -n

# 3. pmxcfs has synchronised: the new node can see the cluster's guests
#    and storage, which it could not before.
cat /etc/pve/.members
pvesm status

# 4. A write from the new node reaches the cluster.
PROBE=/etc/pve/.join-probe
date -Is > "$PROBE" && cat "$PROBE" && rm -f "$PROBE"

# 5. Certificates are consistent. If the GUI on one node cannot proxy to
#    another, this is usually why.
pvecm updatecerts

Removing a node

The order is the whole procedure. Three of these steps are hard to undo.

Cluster-wide riskremoving a node, in order— Cluster-wide and partly irreversible. Steps 1 to 4 are preparation and can be repeated; step 6 cannot be undone, and the node it removes has to be reinstalled before it can ever rejoin.

set -euo pipefail
DOOMED=pve-04

# --- On the node being removed ---
# 1. Move every guest off it. Nothing should be left running or configured.
#    Migrate what can migrate; back up and rebuild what cannot.
qm list
pct list

# 2. Take it out of HA management, so the CRM stops trying to place work on it.
ha-manager status

# --- On a node that is staying ---
# 3. If a QDevice is configured, remove it FIRST. Removing a node with a
#    QDevice in place is a documented way to end up unpicking state by hand.
pvecm status | grep -i qdevice && pvecm qdevice remove || true

# 4. Confirm the cluster is quorate and healthy before changing membership.
pvecm status

# 5. POWER THE DOOMED NODE OFF, and leave it off.
#    It must not be running when it is removed.

# 6. Remove it. This is the irreversible step.
pvecm delnode "$DOOMED"

# 7. Verify: expected votes have dropped, the node is gone from the nodelist.
pvecm status
pvecm nodes

# 8. Its directory under /etc/pve may remain. Check it is empty of guest
#    configurations BEFORE removing it - see the warning below.
ls -la "/etc/pve/nodes/$DOOMED/"

Power the node off before removing it, and never power it back on with its old configuration

The Proxmox documentation is unambiguous about this and the reason is worth understanding rather than memorising.

A removed node still holds a complete replica of /etc/pve from before its removal — including everything under priv/, which is the cluster’s secrets. If it is powered on again on the cluster network, it will attempt to participate with a stale view of membership, and the results range from confusing to destructive.

The documented guidance is to reinstall the node before reusing it. That is not a suggestion about tidiness; it is the mechanism by which the cluster keys and the stale configuration are actually destroyed.

The related consequence is that “temporarily remove a node and add it back later” is not a procedure. Removing is a one-way door. If you want a node out of the cluster for a while, that is what HA maintenance mode and guest evacuation are for — leave it a member.

What happens to the guest configurations on a removed node

Guest configuration files live at /etc/pve/nodes/<name>/qemu-server/ and /etc/pve/nodes/<name>/lxc/. That path is replicated, so every node holds a copy — but the directory belongs to the removed node, and the ownership implied by the path is exactly what makes those guests unmanageable once the node is gone.

This produces the failure that costs people an afternoon: a node dies unexpectedly, pvecm delnode is run to restore a sane cluster, and only then does somebody ask about the twelve guests that were on it. Their disks are on shared storage and perfectly intact. Their configurations are in a directory belonging to a node that no longer exists.

The recovery is a file move, and it is the same operation the pmxcfs documentation describes for recovering guests from a failed node:

mv /etc/pve/nodes/pve-04/qemu-server/100.conf /etc/pve/nodes/pve-02/qemu-server/

Which works, and carries the warning the documentation attaches to it: be absolutely certain the source node is powered off or fenced. A guest whose configuration exists on two nodes, with disks on shared storage, is a double-start away from filesystem corruption.

The ordering rule that avoids the whole situation: move the guest configurations before you run delnode, not after. Once the node directory is gone, so is the record of what was there.

And for guests whose disks were on the dead node’s local storage, no file move helps. Those come back from backup or they do not come back, which is the argument this course keeps making for not putting anything you care about on local storage in a cluster.

Replacing a failed node

The sequence for a node that has died and is not returning:

Confirm it is genuinely dead, through out-of-band management. A node that is partitioned rather than dead is a different problem entirely.
Move its guest configurations to a surviving node, for every guest on shared storage. Do this first.
Recover the rest from backup. Anything that was on local storage.
Remove it: pvecm delnode pve-04 from a surviving node.
Tidy /etc/pve/nodes/pve-04/ once you have confirmed it holds nothing you need.
Build the replacement. A fresh install. The hostname may be reused or not — Proxmox supports either, and reusing it keeps documentation simpler at the cost of some ambiguity in old logs.
Join it as a new node, and verify as above.

Common failure modes

The join fails

Read-only / Safethe checks that resolve most failed joins— The most common single cause, and it is a refusal rather than a failure: the joining node is not empty. The check exists precisely because the join would otherwise replace that configuration.

# pvecm add 192.0.2.11 --fingerprint AA:BB:...:99

detected the following error(s):
* this host already contains virtual guests
Check if node may join a cluster failed!

Illustrative output

Symptom	Cause	Fix
`this host already contains virtual guests`	The joining node is not empty	Back up and remove its guests, or rebuild it
Authentication failure	Wrong root password, or the fingerprint does not match	Re-copy the join information from the cluster
`cluster not ready - no quorum?`	The cluster you are joining is not quorate	Fix the cluster first
Version mismatch errors	Different PVE versions	Bring both to the same version
The join succeeds and the node shows only link 0	`--link1` was omitted	Add the address to `corosync.conf` and reload
Time-related failures	Clock skew	Fix NTP on both, then retry

After a join, quorum does not settle

Check that the addition actually took on every node — pvecm status on each should show the same Config Version and the same expected votes. A node that has not picked up the new corosync.conf will disagree, and the disagreement is visible in that one line.

A break/fix exercise

Break/Fixintermediate25 mincluster

A node joins successfully and the cluster silently loses its link redundancy

Symptoms

pvecm add on the new node completed without error
pvecm status shows the node as a member with the expected vote count
The GUI shows all four nodes green and guests can be migrated to the new node
corosync-cfgtool -n on an existing node shows the new node reachable on LINK 0 only

Available evidence

The three original nodes each have ring0_addr and ring1_addr in corosync.conf
The new node entry in the nodelist has ring0_addr only
The join was performed with pvecm add and the peer IP, with no --link options
The second corosync VLAN is trunked to the new node switch port and the interface is up

Show diagnosis & remediation

Root cause

pvecm add joins with the links it is given. With no --link options, the joining node received a single link derived from the address used to reach the peer, so its nodelist entry has ring0_addr and nothing else. Nothing about this produces an error, because a one-link node in a multi-link cluster is a valid configuration - it simply has no redundancy.

Safe remediation

Add the missing ring1_addr to the new node entry in /etc/pve/corosync.conf, increase config_version, and apply with corosync-cfgtool -R. Then confirm from an existing node that corosync-cfgtool -n reports the new node reachable on both LINK 0 and LINK 1. Prefer joining with explicit --link0 and --link1 arguments so this does not arise.

Verification

corosync-cfgtool -n on every node shows the new node connected on both links. Administratively downing the primary link interface on the new node in a maintenance window leaves it a member of the cluster, reachable over link 1, with pvecm status still quorate.

Prevention

Make explicit --link arguments part of the join procedure on any multi-link cluster, and add a post-join verification step that runs corosync-cfgtool -n from an existing node rather than from the one just joined. A node that joined with fewer links than the cluster uses produces no symptom at all until the day link 0 fails, which is precisely when the redundancy was supposed to matter.

What happens if you run pvecm delnode on a node that is alive but partitioned?

You remove a working node from the cluster, and the node does not find out.

From the surviving partition’s point of view the operation succeeds: the nodelist shrinks, expected votes drop, and the cluster is quorate and tidy. From the removed node’s point of view nothing happened at all — it never received the configuration change, because it was not reachable. It continues running with its old corosync.conf, its old membership view, and a full replica of /etc/pve as it stood at the moment of the partition.

When the network heals, you have two systems with conflicting ideas about what the cluster is. The removed node attempts to participate with a configuration version that is no longer valid. In the good case it simply fails to join and sits there. In the bad case, with shared storage and guests that were running on it, it is still running those guests — and if you have meanwhile recovered their configurations onto a surviving node and started them there, the same disks now have two writers.

This is why the ordering above puts “power the node off” before delnode, and why the first step of replacing a failed node is confirming through out-of-band management that it is genuinely dead rather than unreachable. Unreachable is a statement about the network. Dead is a statement about the node, and only one of them makes removal safe.

If it has already happened: keep the removed node powered off, verify what was running on it, reconcile the guests deliberately on the surviving side, and reinstall the node before it ever touches the cluster network again.

Production considerations

Common mistakes

Running pvecm add on the wrong node. It runs on the joiner.
Joining a node that has guests on it, and losing its configuration.
Joining without --link arguments on a multi-link cluster, silently ending up with one link.
Running delnode before moving the guest configurations off.
Removing a node that is partitioned rather than dead.
Powering a removed node back on without reinstalling it.
Removing a node while a QDevice is still configured.
Joining into a cluster that is already degraded.

Key takeaways

pvecm add runs on the joining node; pvecm delnode runs on a node that is staying. pvecm addnode is internal.
A joining node adopts the cluster’s configuration and loses its own. Join empty nodes.
On a multi-link cluster, join with explicit --link arguments and verify with corosync-cfgtool -n.
Move guest configurations off a dead node before delnode, and be certain the node is off first.
Removal is one-way. A removed node must be reinstalled before it can rejoin.
Removing a genuinely dead node promptly lowers the quorum threshold and restores failure tolerance.

Knowledge check

Why this matters

Cluster membership changes are infrequent enough that nobody is fluent in them and consequential enough that mistakes are expensive. That combination is exactly what a written procedure is for.

Two of the facts in this lesson are worth carrying even if the rest fades. A joining node loses its own configuration, which turns a routine-looking operation into data loss if the node was not empty. And removal is a one-way door: the node has to be reinstalled, so “take it out and put it back” is not available and the alternative — maintenance mode with the guests evacuated — is what you actually wanted.

The rest is ordering. Move the guest configurations before you remove the node, power it off before you remove it, and be certain it is dead rather than unreachable. Those three sentences are the difference between a clean decommission and an afternoon of reconstructing what used to be on a machine that no longer exists.