Decommissioning and refreshing a node · Proxmox VE

Adding a node to a Proxmox cluster is one command and is nearly impossible to get wrong. Removing one is a sequence, most of the steps are irreversible, and doing them out of order can cost you quorum, Ceph data, or both.

The asymmetry is not an oversight. pvecm delnode removes an identity from a distributed system that other nodes have been coordinating with, and that identity cannot come back — which puts a lot of weight on getting everything off the node before you run it.

The order, and why it is this order

Take the node out of HA scheduling so nothing is placed on it while you work.
Move every guest off, HA-managed and not.
Remove its Ceph daemons — OSDs first and one at a time, then the manager, then the monitor.
Remove any storage that only exists on this node from the cluster storage configuration.
Shut the node down and confirm the cluster is quorate without it.
Run pvecm delnode from a surviving node.
Ensure the removed node never boots onto the cluster network again.
Sanitise the disks, then dispose of or repurpose the hardware.

Each step assumes the previous ones. Steps 3 and 6 are the two that cause real damage when reordered.

Step 1–2: evacuate

Service impact possiblestop new placement, then move what is there— Maintenance mode migrates HA-managed guests off in a controlled order and stops the CRM placing anything new. Non-HA guests are not covered and must be migrated explicitly.

NODE=pve03

ha-manager crm-command node-maintenance enable "$NODE"
ha-manager status

# Anything still on the node, HA-managed or not:
pvesh get /nodes/"$NODE"/qemu --output-format json | grep -o '"vmid":[0-9]*'
pvesh get /nodes/"$NODE"/lxc  --output-format json | grep -o '"vmid":[0-9]*'

Service impact possiblemigrate the remainder— Live-migrates a VM to another node. Guests on node-local storage cannot live-migrate and need an offline migration with a storage move, which copies the disk and takes proportionally longer.

VMID=142
TARGET=pve01

qm migrate "$VMID" "$TARGET" --online
# For a guest on local storage:
# qm migrate "$VMID" "$TARGET" --with-local-disks

The check that matters before continuing is that the node holds nothing at all — not just nothing running. A stopped VM, an ISO nobody remembers, a container template, a backup written to local are all data that disappears with the node.

Read-only / Safewhat is still on this node's local storage?— Read-only. Run this on the node being decommissioned. The first command lists its storages; the loop prints the contents of each, which is where a forgotten ISO, template or local backup shows up.

pvesm status

for STORE in $(pvesm status --enabled 1 | awk 'NR>1 {print $1}'); do
echo "== $STORE"
pvesm list "$STORE"
done

Step 3: Ceph, if the node runs it

This is the step where impatience destroys data.

A Proxmox node running Ceph typically hosts OSDs, and often a monitor and a manager. Each has to be removed in a way that lets the cluster re-establish its redundancy first.

Read-only / Safewhat does this node run?— Read-only. Establishes the daemons on the node and the current health. Do not start removing anything while health is not HEALTH_OK for reasons unrelated to this work.

NODE=pve03

ceph -s
ceph osd tree
ceph mon dump
ceph mgr dump | head -20

Data-loss riskremove one OSD, then wait— Marking an OSD out triggers rebalancing of its data onto the remaining OSDs. Destroying it before the rebalance completes removes a copy the cluster is still relying on. One OSD at a time, and wait for HEALTH_OK between each.

OSD_ID=7

ceph osd out "$OSD_ID"

# Wait for the rebalance. Do not proceed until this reports HEALTH_OK
# and no PGs are degraded or backfilling.
watch -n 10 ceph -s

pveceph osd destroy "$OSD_ID"

Once the OSDs are gone, remove the manager and then the monitor:

Cluster-wide riskremove the manager, then the monitor— Monitors form their own quorum. Removing one changes the monitor count, so check that an odd number remains and that quorum survives before and after.

NODE=pve03

pveceph mgr destroy "$NODE"

ceph mon stat
pveceph mon destroy "$NODE"
ceph mon stat

Monitors need their own majority. Going from three monitors to two leaves a configuration that cannot tolerate a single failure — so a decommission that reduces the monitor count should be paired with promoting a monitor elsewhere, ideally before the removal rather than after.

Step 4–5: storage, shutdown, quorum

Remove storage definitions that only pointed at this node. A definition restricted to a removed node is a stale entry that produces confusing errors later; a shared definition should keep working and is worth verifying rather than assuming.

Configuration changeclean up node-restricted storage— Removes a storage definition, or narrows its node list. This changes cluster configuration, so it is replicated immediately; it does not delete any data.

cat /etc/pve/storage.cfg

# Drop a node from a storage's node restriction:
pvesm set local-zfs-pve03 --nodes pve01,pve02

# Or remove the definition entirely:
pvesm remove local-zfs-pve03

Then shut the node down and confirm the cluster is healthy without it:

Read-only / Safeis the cluster fine without this node?— Read-only. Run from a surviving node after the target is powered off. Expected votes drops by one only after delnode; before it, the cluster still expects this node to vote.

pvecm status
pvecm nodes
ha-manager status

Step 6: `pvecm delnode`

Cluster-wide riskremove the node from the cluster— Removes the node from corosync and from the cluster configuration, cluster-wide and immediately. Not reversible. The target node must be powered off when this runs.

NODE=pve03

pvecm delnode "$NODE"

pvecm status
pvecm nodes

The node must be off when this runs. Removing a node that is still running leaves it believing it is a cluster member while the cluster has forgotten it — and what happens next is the subject of the failure section below.

After removal, the node’s directory under /etc/pve/nodes/ remains. It holds that node’s guest configurations and its host.fw, and Proxmox keeps it deliberately so a mistake is recoverable. Once you are certain the node is gone for good, it can be removed:

Destructiveclean up the leftover configuration directory— Deletes the removed node's configuration from pmxcfs, cluster-wide. Verify the directory contains no guest configuration you still need: these are the last copies.

NODE=pve03

ls -la /etc/pve/nodes/"$NODE"/
ls -la /etc/pve/nodes/"$NODE"/qemu-server/ /etc/pve/nodes/"$NODE"/lxc/ 2>/dev/null

# Only after confirming the above is empty of anything you need:
rm -r /etc/pve/nodes/"$NODE"

Step 7–8: keep it gone, then sanitise

Make sure it cannot come back. Disconnect it from the cluster network, or wipe it, before it is powered on again for any reason.

Then sanitise the disks. The right method depends on the media, and the wrong method is a common finding.

Media	Method	Notes
SATA/SAS SSD	ATA Secure Erase, or the drive’s sanitize command	Overwriting is unreliable on flash: wear levelling means the blocks you write are not the blocks that held the data
NVMe SSD	`nvme format` with a secure-erase setting	The controller erases including over-provisioned blocks
Self-encrypting drive	Crypto-erase — destroy the key	Instant, and complete, because the ciphertext becomes unrecoverable
Spinning disk	Multi-pass overwrite, or degauss	Overwriting genuinely works here
Any, high-assurance	Physical destruction	The only method that needs no trust in the firmware
ZFS or LUKS encrypted at rest	Destroy the key material	The simplest correct answer, if it was encrypted from the start

The strongest position is to decide this at build time rather than at disposal time. A node whose pool was created with ZFS native encryption is sanitised by destroying the key, which takes seconds, works regardless of media, and is verifiable.

The mixed-generation cluster

A refresh means new nodes and old nodes coexisting for weeks. Three things need attention.

CPU model and live migration. A guest with cpu: host gets the host’s full feature set, and cannot migrate to a node whose CPU lacks a feature it is using. In a mixed cluster that means new guests started on new hardware cannot move to the old nodes — which is fine until you need to evacuate a new node. Set a common CPU model that all generations support for anything that must migrate freely, and accept host only where the guest is pinned to a hardware class deliberately.

Uneven capacity changes placement. HA and CRS place guests according to available resources, so new larger nodes attract more guests. That is usually desirable and it concentrates blast radius: losing one new node now takes out more workload than losing one old node ever did. Check the failure arithmetic after the first new node arrives rather than after the last one.

Ceph across generations. Mixing OSD sizes changes CRUSH weighting, and mixing device classes — spinning disks and NVMe in one pool — makes the pool as slow as its slowest members for the placement groups that land there. Plan the CRUSH rules alongside the hardware, not after it.

Read-only / Safewhat will and will not migrate?— Read-only. Lists the CPU type configured for each VM. Anything set to host is pinned to hardware compatible with where it started.

for VMID in $(qm list | awk 'NR>1 {print $1}'); do
printf '%s ' "$VMID"
qm config "$VMID" | grep -E '^cpu:' || echo 'cpu: (default kvm64)'
done

Under the hood

Why delnode is one-way, and what the node itself still believes.

Cluster membership is corosync configuration in /etc/pve/corosync.conf, plus the node’s entry under /etc/pve/nodes/, plus the cryptographic material that lets it join pmxcfs. pvecm delnode removes the corosync entry and lowers the expected vote count, and it does this on the surviving nodes — the removed node is not consulted, which is exactly why it must be powered off.

The removed node’s own copy of /etc/pve is a stale snapshot of the cluster as it was. It still contains a corosync.conf naming the cluster, still holds the cluster’s authentication key, and still believes it is a member. Nothing in the removal reaches out and tells it otherwise.

That is the mechanism behind the strongest rule in this lesson: a removed node must never rejoin the cluster network. If it powers on with its old configuration, it attempts to participate in a corosync ring using a node ID the cluster has reassigned or retired, and the result ranges from harmless-looking log noise to a genuine split-brain, in which two sets of nodes each believe they are authoritative for /etc/pve. Proxmox’s documented position is that a removed node must be reinstalled before it can join again — not reconfigured, reinstalled.

Two related details.

Expected votes change on removal, not on shutdown. A powered-off node still counts toward the expected total until delnode runs, which is why the cluster can be short of quorum during the window between shutting down the last node of a decommission batch and completing the removal. Removing nodes one at a time, completing each removal before shutting the next one down, avoids the situation entirely.

Ceph identity is separate from PVE identity. A monitor removed from Ceph is still a PVE cluster member, and a node removed from the PVE cluster with its OSDs still defined leaves the Ceph cluster referencing daemons on a host that no longer exists. The two removals are independent and both are required, which is why Ceph comes first in the ordering: it needs a working cluster and a working node to do it cleanly.

/etc/pve/nodes/$NODE surviving the removal is deliberate. It holds the guest configuration files for anything that was on the node, and those are the last copies once the node is gone. Proxmox leaves them so that a premature removal is recoverable — the configurations can be moved to another node’s directory and the guests started there, provided the storage they reference is still reachable.

What happens if a decommissioned node is powered back on with its old configuration?

The realistic version of this is not malice. It is a node put on a shelf, then reused six months later for something unrelated, plugged into the same rack switch, and powered on — with /etc/pve still describing a cluster it was removed from.

What happens next depends on what the cluster has done since.

If the node ID was never reused, the node broadcasts on the corosync ring using the cluster’s authentication key and a node ID the cluster no longer expects. Surviving nodes log the unexpected member. Usually nothing breaks, and you get an alarming set of corosync log entries and a few minutes of confusion.

If the node ID has been reused — because you added a new node after the removal and corosync assigned the next free ID — two machines now claim the same identity in the ring. That is the bad case. Corosync membership becomes unstable, quorum calculations disagree between nodes, and pmxcfs can end up with two sets of nodes each believing they hold the authoritative /etc/pve. Writes on both sides diverge, and reconciling diverged cluster configuration is manual and unpleasant.

If HA is enabled, this is worse still. The revived node runs its own LRM, reads a stale view of what it is supposed to be running, and may attempt to start guests that are running on other nodes right now — the same disk, opened twice, by two hypervisors. On shared storage that is filesystem corruption rather than an availability problem.

The immediate response is to disconnect it from the cluster network physically, not to try to fix it while it is participating. Then assess whether the surviving cluster’s configuration diverged during the window, and whether any guest was started twice.

Prevention is entirely procedural and costs nothing:

Wipe or physically disconnect the node as part of the decommission, in the same window, before it becomes something on a shelf. Step 7 exists for this reason and it is the step most often deferred.

Label the hardware. A physical tag reading “removed from cluster prod-a, requires reinstall before reuse” is a low-technology control that works on the person who was not involved in the decommission.

Reinstall before reuse, always. Proxmox’s documented position is that a removed node must be reinstalled to join a cluster again. Treat that as applying to any reuse at all, not only to rejoining, because the failure mode is the node reaching the network with its old configuration regardless of what you intended to use it for.

Common mistakes

pvecm delnode while the node is still running. The node keeps believing it is a member. Power it off first.
Removing several OSDs at once. Concurrent rebalances can take placement groups below min_size and stop I/O, or lose data on a size=2 pool.
Not waiting for HEALTH_OK between OSD removals. The wait is the procedure.
Decommissioning to two nodes without adding a QDevice. A two-node cluster cannot survive a single failure.
Reducing monitors to two. Promote a replacement monitor before the removal, not after.
Assuming the node is empty because nothing is running. Stopped guests, ISOs, templates and local backups all vanish with it.
dd if=/dev/zero on an SSD as sanitisation. Wear levelling means the data is still there.
Leaving the node on a shelf, configured. It gets powered on six months later by someone who was not involved.
Forgetting cpu: host guests during a refresh. They cannot migrate back to older hardware, which you discover while evacuating a new node.

Key takeaways

The order is evacuate, remove Ceph daemons, clean up storage, shut down, delnode, keep it off the network, sanitise. Most steps are irreversible.
OSDs come out one at a time with a wait for HEALTH_OK between each. Concurrent rebalances are how a decommission becomes an outage.
The node must be powered off before pvecm delnode, and a removed node must be reinstalled before it joins anything again.
A powered-on removed node can cause split-brain, and with HA enabled can start a guest that is already running elsewhere.
Expected votes drop at delnode, not at shutdown. Complete each removal before shutting down the next node.
/etc/pve/nodes/$NODE survives on purpose and holds the last copies of that node’s guest configurations.
Sanitise according to the media. Encrypting at rest from day one turns disposal into key destruction.
In a mixed-generation cluster, watch cpu: host guests, concentrated blast radius on larger nodes, and CRUSH weighting across device classes.

Knowledge check

Why this matters

Decommissioning is the operation with the worst ratio of frequency to consequence. Most operators do it a handful of times, always under some pressure — a lease ending, a refresh deadline, hardware that has already failed once — and every irreversible step in it is one they have performed rarely enough not to have a feel for.

The consequences are not evenly distributed either. Getting the guest evacuation wrong costs you a restore. Getting the Ceph ordering wrong can cost you data. And the step people treat as trivial — what happens to the machine afterwards — is the one that produces a split-brain six months later, initiated by someone who was not involved and had no way to know.

That last case is worth dwelling on, because it is the clearest example in Proxmox operations of a risk that outlives the change window. The cluster does nothing wrong. The person who powers on a spare server does nothing unreasonable. The failure is entirely in what the decommission left behind: a machine holding a cluster’s authentication key and a stale belief about its own membership, sitting on a shelf where nothing records what it is.

Which makes the two cheapest steps in the whole procedure the ones worth insisting on. Wipe it in the same window rather than later, and put a physical label on it. Neither takes five minutes, and both work on the person who was not in the room.