high riskcluster affecting~90 min

Permanently remove a node from a cluster without breaking quorum

1 · Prerequisites

Confirm every item is in place before any state change.

The decision to decommission is made and recorded; this procedure is not reversible in place
The remaining cluster can carry the workload: CPU, RAM and storage headroom checked against actual usage, not against nominal capacity
The quorum arithmetic after removal has been worked out and is acceptable, including the two-node case
Every guest on the node has a confirmed destination, or a decision that it is being retired
If the node runs Ceph daemons, the cluster can lose them: monitor count stays odd and at least three, and OSD capacity is sufficient without it
Out-of-band access to the departing node so it can be powered off and kept off

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· pvecm status reports Quorate: Yes and the expected vote count matches the current node count
· pvesh get /cluster/resources lists every guest and its node, so the departing node inventory is known and written down
· ceph -s is HEALTH_OK and ceph osd tree shows which OSDs live on the departing node
· ha-manager status shows no HA service assigned to the departing node once evacuation is complete
· The departing node appears in no storage restriction, no backup job, no replication job and no HA rule
· ls /etc/pve/nodes shows the exact directory name that will need removing later

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Record the full inventory of the departing node: guests, OSDs, monitors, storage restrictions, jobs, DNS entries, monitoring targets
2Migrate every guest off the node and confirm none remain assigned to it
3Remove the node from HA rules and groups so nothing tries to place a service back on it
4Remove any replication and backup jobs that name the node as source or target
5If Ceph is in use: mark the node OSDs out, wait for the cluster to become clean, then destroy them one at a time
6If Ceph is in use: destroy the manager, then the metadata server, then the monitor on this node, checking health between each
7Remove the node from storage definitions that restrict a storage to a node list
8Power the node off, and make sure it cannot boot back onto the cluster network
9From a surviving quorate node, run pvecm delnode with the departing node name
10Confirm the remaining cluster is quorate with the reduced expected vote count
11Remove the leftover /etc/pve/nodes directory for the departed node, after confirming it holds no guest configs
12Clean up SSH known hosts and regenerate cluster certificates
13Update DNS, monitoring, inventory, documentation and the firewall rules that named the node
14If the cluster is now two nodes, add a QDevice or accept and document the reduced fault tolerance

4 · Verification

Confirm the procedure actually fixed the problem.

✓pvecm status shows the reduced node count, Quorate: Yes, and expected votes equal to the new node count
✓pvecm nodes does not list the departed node
✓ls /etc/pve/nodes does not contain the departed node directory
✓ceph -s is HEALTH_OK, ceph mon stat shows an odd monitor count of at least three, and no PG is degraded or undersized
✓ha-manager status shows every service started on a surviving node
✓A live migration between two surviving nodes succeeds
✓A scheduled backup job runs to completion after the removal
✓The GUI shows no stale grey node entry after a browser reload

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Before pvecm delnode, rollback is straightforward: return the guests, re-enable the jobs, and the node is a normal member again
↶Ceph daemon destruction is not reversible in place - a destroyed monitor or OSD is recreated, not restored, and recreating an OSD means a full backfill
↶pvecm delnode is the point of no return. The removed node cannot rejoin under the same identity without a reinstall
↶NEVER power the removed node back on with its old corosync configuration. It will attempt to rejoin a cluster that has forgotten it, and the resulting state is a repair job, not a rollback
↶If the node must come back, it comes back as a rebuild: see the replace-failed-node-same-name runbook
↶If quorum is lost during the procedure, stop and restore quorum by returning a node to service - not by lowering the expected vote count

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the cluster owner before pvecm delnode, because that is the irreversible step
· Escalate to the storage owner before destroying any Ceph monitor or OSD, and stop if Ceph is not HEALTH_OK
· Escalate if the removal would leave an even number of nodes without a QDevice, or fewer than three Ceph monitors
· Escalate if delnode fails - a failure here usually means the cluster is not quorate or the node is still reachable, and forcing it makes both worse
· Escalate to the service owner if the remaining capacity headroom after removal is below the level that survives one more node failure

Verified against Proxmox VE 9.2.4 with Ceph Squid.

Removing a node is mechanically simple - one pvecm delnode - and that simplicity is the trap. The command takes seconds; the work is everything that happens before it, and the cleanup is everything that happens after.

Two facts shape the whole procedure. The node must be off, and stay off, before it is removed - a departed node that boots back onto the cluster network still believes it is a member, and that belief is expensive. And delnode changes the quorum arithmetic immediately, so the arithmetic has to be acceptable before you run it, not discovered after.

When to use this runbook

Hardware is being retired or returned at end of lease.
The cluster is being shrunk deliberately.
A node is being moved to a different cluster.
A node failed permanently and is being removed with no replacement. (If a replacement with the same name is coming, use replace-failed-node-same-name instead - the cleanup differs.)

Do not use it for a node that is temporarily unreachable. A node that might come back is a node-maintenance problem, not a delnode problem.

Do the quorum arithmetic first

Corosync gives one vote per node and needs a strict majority. After removal, expected votes drops to the new node count.

Before	After	What you are left with
5 nodes	4 nodes	Tolerates 1 failure. Even count - acceptable but a QDevice is better
4 nodes	3 nodes	Tolerates 1 failure. The healthiest small-cluster shape
3 nodes	2 nodes	Tolerates zero failures. Either node going down loses quorum. Add a QDevice
2 nodes	1 node	No cluster. Standalone. Say so explicitly and plan the HA and storage consequences

Step 1: Write down everything that names this node

Read-only / Safeinventory the departing node

NODE=pve04
OUT="/root/decom-$NODE-$(date +%Y%m%d).txt"

{
echo '== guests'
pvesh get /cluster/resources --type vm --output-format yaml
echo '== storage seen by this node'
pvesm status
echo '== ceph daemons'
ceph osd tree
ceph mon stat
ceph mgr stat
echo '== ha'
ha-manager status
echo '== replication'
pvesr status
echo '== node config dir'
ls -la /etc/pve/nodes/"$NODE"
} | tee "$OUT"

Also grep the cluster configuration for the hostname. A node is named in more places than the cluster tracks:

Read-only / Safeevery reference in /etc/pve

NODE=pve04
grep -rn "$NODE" /etc/pve/storage.cfg /etc/pve/ha /etc/pve/jobs.cfg \
/etc/pve/vzdump.cron /etc/pve/replication.cfg 2>/dev/null

# And outside the cluster
grep -rn "$NODE" /etc/hosts /etc/network/interfaces 2>/dev/null

Storage definitions with a nodes restriction are the classic leftover: the storage keeps working, but the restriction list names a node that no longer exists, and the next person to read it loses twenty minutes.

Step 2: Evacuate

Service impact possiblemove the guests

NODE=pve04
TARGET=pve01

ha-manager crm-command node-maintenance enable "$NODE"
sleep 30

# Online where possible
VMID=101
qm migrate "$VMID" "$TARGET" --online

# Containers require a brief restart
CTID=201
pct migrate "$CTID" "$TARGET" --restart

Read-only / Safeprove nothing is left

NODE=pve04
pvesh get /cluster/resources --type vm --output-format json \
| grep -o "\"node\":\"$NODE\"" | wc -l
# Must print 0 before continuing.

That count is a check that can fail, and it is the one people skip because the GUI “looked empty”. A stopped guest still has a config file on the node, and a config file on a node you are about to delete is a guest you are about to lose the definition of.

Step 3: Dismantle the Ceph roles, in order

Skip this step entirely if the node runs no Ceph daemons. If it does, the order matters: OSDs first, and each stage waits for health.

Cluster-wide riskOSDs out, then destroyed, one at a time

OSD=7

ceph osd out "$OSD"

# Wait for the data to move. This is not instant and must not be rushed.
watch -n 30 ceph -s
# Continue only when: HEALTH_OK, all PGs active+clean, recovery finished.

systemctl stop "ceph-osd@$OSD"
pveceph osd destroy "$OSD" --cleanup 1
ceph -s

Cluster-wide riskthen mgr, mds, mon

NODE=pve04

pveceph mgr destroy "$NODE"
ceph -s

# Only if this node runs a metadata server
pveceph mds destroy "$NODE"
ceph -s

# The monitor goes last
pveceph mon destroy "$NODE"
ceph mon stat
ceph -s

Check the monitor count before and after. Ceph monitors need a majority too, and they need an odd count of at least three for a production cluster. Going from three monitors to two leaves a monitor quorum that tolerates zero failures - the same trap as corosync, in a different system. If removing this node takes you below three, add a monitor on a surviving node before destroying this one.

Step 4: Remove the jobs and rules that name it

Configuration changeha, replication, backup, storage

NODE=pve04

# HA rules and groups
ha-manager config
# Edit or remove any rule that names the node, via GUI or:
#   pvesh set /cluster/ha/rules/RULENAME ...

# Replication jobs with this node as target
pvesr list

# Storage restricted to a node list
grep -A5 -n 'nodes' /etc/pve/storage.cfg

A backup job that targets a node that no longer exists fails silently at 02:00 and is discovered when a restore is needed. Removing it now is thirty seconds; finding it later is an incident.

Step 5: Power it off, and keep it off

Service impact possibleshut down for good

NODE=pve04
ssh "root@$NODE" 'systemctl poweroff'
sleep 60
ping -c 3 "$NODE" || echo 'node is down'

The node must be off before delnode, and must never boot onto this network again

pvecm delnode removes the node from the cluster’s view. It does not tell the node anything - the node is gone by then. If the departing node is still running, or is later powered on with its old /etc/corosync configuration, it will try to participate in a cluster that has removed it.

What follows is not a clean rejection. The old member can disrupt corosync membership, and if it still has access to shared storage it can start guests that are already running elsewhere - the same disk image mounted twice, which destroys the filesystem inside it.

Power it off first. If the hardware is being repurposed, wipe or reinstall it before it ever touches the cluster network again. If it is being stored as a spare, physically disconnect the cluster NIC.

Step 6: delnode

Cluster-wide riskfrom a surviving, quorate node

NODE=pve04

pvecm status
# Confirm: Quorate: Yes, and the departing node shows as offline.

pvecm delnode "$NODE"

pvecm status
pvecm nodes

Expected votes should now equal the new node count, and the cluster should still report Quorate: Yes. If delnode refuses, read the error rather than working around it - the usual causes are that the cluster is not quorate, or that the node is still online.

If delnode fails because the cluster is not quorate, do not run `pvecm expected 1`

pvecm expected 1 tells a node it constitutes a majority by itself. It makes /etc/pve writable again on an inquorate node, which is why it gets suggested - and it removes the only mechanism preventing two nodes from independently deciding they own the same guest.

In a planned decommission there is no situation where it is the right first move. If the cluster is not quorate, something has gone wrong that is unrelated to the removal, and the correct action is to restore quorum by getting a node back - or to stop and escalate.

It is a last-resort tool for a cluster that has permanently lost a majority of its nodes, used with every remaining node confirmed destroyed and every shared storage path confirmed unreachable from them. Even then it is a step in a recovery plan, not a command you type at a prompt because /etc/pve is read-only.

Step 7: Clean up what delnode leaves behind

delnode removes the node from corosync membership. It does not remove the node’s directory in the cluster filesystem.

Destructiveremove the leftover node directory

NODE=pve04

# Confirm it holds no guest configs. This must print nothing.
ls -la /etc/pve/nodes/"$NODE"/qemu-server/ 2>/dev/null
ls -la /etc/pve/nodes/"$NODE"/lxc/ 2>/dev/null

# Only then:
rm -rf /etc/pve/nodes/"$NODE"
ls /etc/pve/nodes/

If those listings are not empty, stop. Those files are the definitions of guests the cluster still thinks exist. Either the evacuation missed something, or these are stale configs for guests that were deleted. Copy the directory somewhere safe and work out which before removing it.

Configuration changessh identities and certificates

NODE=pve04

ssh-keygen -R "$NODE"
ssh-keygen -R "$(getent hosts "$NODE" | awk '{print $1}')" 2>/dev/null

pvecm updatecerts --force
systemctl restart pveproxy

Step 8: If you are now at two nodes

Configuration changeadd a QDevice

QNET=192.0.2.60

# On the external witness host (a small always-on Debian machine, not a cluster node):
#   apt install corosync-qnetd

# On one cluster node:
pvecm qdevice setup "$QNET"
pvecm status

The QDevice is an external vote holder. It does not run guests and it does not need to be powerful - it needs to be independent of both nodes and of the failure domain that could take one of them out. Putting it on a VM inside the same cluster defeats the entire point.

If a QDevice is not being added, record that decision and its consequence in the change record: this cluster now has no fault tolerance for a node loss.

Step 9: The world outside the cluster

DNS: remove or repoint the A and PTR records.
Monitoring: remove the host, or the next maintenance window is spent acknowledging alerts for a machine that no longer exists.
Firewall and access lists: remove the node IP.
Backup infrastructure: remove any PBS entry or ACL scoped to it.
Inventory and documentation: the cluster diagram is now wrong.
Physical: label the hardware so nobody racks it back and powers it on.

Rollback

Stage	Can you go back?
Inventory taken, nothing changed	Yes, trivially
Guests migrated off	Yes - migrate them back
HA and jobs removed	Yes - recreate them
Ceph OSDs destroyed	No. Recreating means a full backfill, not a restore
Ceph monitor destroyed	No. Recreate it, which is a new monitor with new state
Node powered off	Yes, until `delnode`
`pvecm delnode` run	No. The node cannot rejoin under the same identity without a reinstall
Node directory removed	No, unless you kept the copy

Common patterns

Symptom	Likely cause	Resolution
`delnode` refuses	Cluster not quorate, or the node is still online	Fix the cause; do not force it
Removed node still shows in the GUI	Stale `/etc/pve/nodes` directory, or a cached browser page	Remove the directory, reload the browser
Ceph `HEALTH_WARN` after removal	PGs undersized because failure domain is host and a host went away	Add capacity, or adjust the CRUSH rule deliberately
Backup job fails at 02:00 after the window	The job named the removed node	Remove or repoint the job
Cluster loses quorum a week later	Shrunk to two nodes with no QDevice	Add a QDevice; this was foreseeable
Migration fails to a surviving node	Storage `nodes` restriction still lists the departed node	Edit `storage.cfg`
The old node reappears in `pvecm nodes`	It was powered back on with its old corosync config	Power it off immediately; then reinstall it before reuse

Escalation

Escalate when:

delnode fails for any reason.
Ceph is not HEALTH_OK before or after any daemon destruction.
The removal takes the cluster below three Ceph monitors.
The remaining capacity does not survive one more node failure.
The departed node has been powered back on by accident.

Why this matters

A decommission is the change with the longest tail. The command that removes the node is the shortest part of it, and almost nothing that goes wrong afterwards is caused by that command - it is caused by the references nobody removed and the arithmetic nobody did.

The arithmetic is the part worth being rigid about. Corosync quorum and Ceph monitor quorum are two independent majority systems, and a removal can be fine for one and fatal for the other. Three nodes to two, or three monitors to two, both leave you with a system that tolerates zero failures - a state that looks completely healthy in every dashboard right up until the moment it isn’t.

And the rule that has no exceptions: the node goes off before it goes away, and it never comes back on this network with its old identity. A forgotten member that reboots into a cluster which has removed it does not politely fail - it can start guests that are already running somewhere else, on the same disks. That is the failure this procedure’s ordering exists to prevent.