high riskcluster affecting~240 min

Replace a failed node with a rebuilt one of the same name

1 · Prerequisites

Confirm every item is in place before any state change.

The failed node is confirmed dead and will not be repaired in place: hardware diagnosis done, not assumed from a failed ping
The surviving cluster is quorate; if it is not, restore quorum first, because this procedure needs a writable /etc/pve
Replacement hardware is available and its network ports are cabled the same way as the original
Installation media for the exact Proxmox VE version the rest of the cluster runs
The original node network configuration is known: hostname, IP, netmask, gateway, bond members, VLANs, corosync link addresses
It is established which guests lived on shared storage - and are therefore recoverable from config - and which lived on node-local storage and must come from PBS
A PBS restore has been tested recently for at least one of the guests that lived on local storage

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· pvecm status on a surviving node reports Quorate: Yes
· ls /etc/pve/nodes/NODE/qemu-server and .../lxc show the guest configs the cluster still holds for the dead node
· pvesm status shows which storages are shared and which are node-local
· ceph -s shows the health impact of the lost node, and ceph osd tree shows its OSDs as down
· The intended hostname and IP are not currently answering on the network
· The replacement hardware boot media and out-of-band console both work

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Copy the dead node guest configs out of /etc/pve to a location outside the cluster filesystem, before anything is removed
2Record the cluster identity details the rebuild must reproduce: hostname, IPs, corosync link addresses, storage layout
3Decide per guest: recoverable from config plus shared storage, restore from PBS, or retired
4Start the recoverable guests on surviving nodes now, by moving their config files, so the service is restored before the rebuild begins
5Destroy the dead node Ceph daemons from a surviving node so Ceph stops waiting for them
6Remove the old node identity: pvecm delnode, then remove the /etc/pve/nodes directory for it
7Install Proxmox VE on the replacement hardware at the same version, with the same hostname and IP
8Bring the network configuration to match the original, including bonds, VLANs and the corosync link addresses
9Join the rebuilt node to the cluster
10Recreate the node Ceph daemons: monitor, manager, and each OSD
11Return workload to the node gradually, starting with one guest that can be lost
12Restore from PBS any guest whose data lived only on the failed node local storage
13Re-add the node to backup jobs, replication jobs, HA rules, monitoring and DNS

4 · Verification

Confirm the procedure actually fixed the problem.

✓pvecm status shows the rebuilt node as a member, with expected votes equal to the full node count and Quorate: Yes
✓pveversion -v on the rebuilt node matches the other nodes exactly
✓pvesm status on the rebuilt node lists every shared storage as active
✓ceph -s is HEALTH_OK, the new OSDs are up and in, and all PGs are active+clean
✓A live migration succeeds both onto and off the rebuilt node
✓Every guest that lived on the failed node is running somewhere, and each one has been checked at the application level
✓Guests restored from PBS have been checked for data currency against the expected RPO, not merely for booting
✓A scheduled backup job that includes the rebuilt node completes successfully

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The failed node is already gone; this procedure builds a replacement, so there is nothing to restore to
↶Copying the guest configs out is the only step that protects anything. Do it first and do not skip it
↶Removing /etc/pve/nodes for the dead node is irreversible and destroys the last copy of its guest configs inside the cluster - which is why the copy comes first
↶If the rebuild goes wrong, wipe and reinstall the replacement; nothing on the surviving cluster is affected until the join
↶If the join fails, do NOT lower expected votes to make it work. Diagnose why, because a node that joins a cluster it does not fully agree with is worse than a node that stays out
↶A guest restored from PBS replaces whatever is there. If the guest was partially recovered by other means first, restoring over it discards that work - decide which source wins before restoring

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the cluster owner before pvecm delnode and again before the join
· Escalate to the storage owner before recreating Ceph daemons, and stop if Ceph is not at least HEALTH_WARN with a understood cause
· Escalate immediately if the surviving cluster is not quorate - that is a different and more urgent problem than the rebuild
· Escalate if any guest config is missing from the copy taken in step one, because that guest may exist only as a PBS snapshot
· Escalate to the service owner with the measured data loss for every guest restored from PBS, expressed as the gap between the snapshot time and the failure time

Verified against Proxmox VE 9.2.4 with Ceph Squid and Proxmox Backup Server 4.2.5.

A node has died. The cluster is still running, still holds that node’s guest configuration in /etc/pve/nodes/NODE, and the replacement is going to take the same hostname and the same IP because everything from DNS to firewall rules to the monitoring system already knows that name.

The complication is that a cluster will not accept a join from a node whose name it already knows. So the rebuild is not “install and join” - it is rescue, forget, rebuild, rejoin, restore, and the order is not negotiable because step two destroys what step one saves.

When to use this runbook

A node suffered an unrecoverable hardware failure: motherboard, boot media, both PSUs, a fire.
A node’s operating system is unrecoverable and reinstalling is faster than repairing.
A node was compromised and must be rebuilt rather than cleaned.

If the node might come back, this is the wrong runbook - a temporarily unreachable node is a node-maintenance situation. If no replacement is coming, use remove-node-from-cluster instead.

What survives the failure, and what does not

This is the assessment that determines the whole recovery, and it is worth doing on paper before touching anything:

What	Where it lived	State now
Guest configuration	`/etc/pve`, replicated to every node	Survives. The cluster still has it
Guest disks on Ceph or NFS	Shared storage	Survives. Untouched by the node loss
Guest disks on local ZFS or LVM	The dead node’s disks	Gone, unless the disks are physically recoverable or a replica exists
ZFS replication targets	The dead node’s pool	Gone. The replica on the surviving side is what you have
Ceph OSD data	The dead node’s disks	Redundant elsewhere, if `size` was 3. The cluster has already recovered or is recovering
The node’s `/etc` outside `/etc/pve`	Local root filesystem	Gone. Network config, custom units, local scripts
Local backups written to the node	Local storage	Gone. This is why backups go somewhere else

The row that decides your day is the third one. A guest on shared storage is back in minutes because its config still exists. A guest on local storage is a PBS restore, and its recovery point is whenever the last snapshot ran.

Step 1: Rescue the configs before anything else

Read-only / Safecopy the dead node's guest configs off the cluster filesystem

NODE=pve03
DEST="/root/rescue-$NODE-$(date +%Y%m%d-%H%M)"

mkdir -p "$DEST"
cp -a /etc/pve/nodes/"$NODE" "$DEST/"
ls -R "$DEST"

# Also record what the cluster believes about those guests
pvesh get /cluster/resources --type vm --output-format yaml > "$DEST/cluster-resources.yaml"
cat /etc/pve/storage.cfg > "$DEST/storage.cfg"

This copy is the only thing standing between you and losing guest definitions

/etc/pve/nodes/pve03/qemu-server/*.conf is where the cluster keeps the definition of every VM that node owned: disks, network, CPU, boot order, the lot. Those files are replicated, so they survived the node’s death.

Step 4 of this runbook deletes that directory, because the cluster will not accept a rejoin from a node whose name it still knows. Once it is deleted, a guest whose config you did not copy exists only inside a PBS snapshot - recoverable, but as a restore rather than a start.

Copy it to /root on a surviving node and somewhere off the cluster. Then check the copy is complete against the resource list before you proceed. It takes a minute and it is the difference between an inconvenient evening and a bad one.

Step 2: Restore service before rebuilding hardware

Guests whose disks are on shared storage can run right now, on a surviving node, without waiting for the replacement. Moving a guest config between node directories in /etc/pve is how you reassign it.

Service impact possiblereassign a shared-storage guest to a surviving node

DEAD=pve03
ALIVE=pve01
VMID=104

# Confirm the guest disks are on shared storage, not on the dead node
grep -E '^(scsi|virtio|sata|ide|efidisk|tpmstate)' /etc/pve/nodes/"$DEAD"/qemu-server/"$VMID".conf

# Confirm it is not running anywhere
qm status "$VMID" 2>/dev/null || echo 'not running on this node'

mv /etc/pve/nodes/"$DEAD"/qemu-server/"$VMID".conf \
 /etc/pve/nodes/"$ALIVE"/qemu-server/"$VMID".conf

qm start "$VMID"
qm status "$VMID"

For guests whose disks were local to the dead node, do not move the config. They will be restored from PBS in step 9, and the restore creates its own config.

Step 3: Tell Ceph the node is gone

Skip if this cluster does not run Ceph. If it does, Ceph is currently waiting for daemons that will never answer.

Cluster-wide riskfrom a surviving node

DEAD=pve03

ceph -s
ceph osd tree

# For each OSD that lived on the dead node:
OSD=5
ceph osd out "$OSD"
ceph osd purge "$OSD" --yes-i-really-mean-it

# Then the monitor and manager, if it ran them
ceph mon remove "$DEAD"
ceph mon stat
ceph -s

Watch the monitor count. If the cluster had three monitors and one is now gone, you have two - a monitor quorum that tolerates no further failure. Add a monitor on another surviving node before you continue if the rebuild will take more than a short window.

Step 4: Remove the old identity completely

Cluster-wide riskdelnode and clear the node directory

DEAD=pve03

pvecm status
# Confirm Quorate: Yes before doing anything.

pvecm delnode "$DEAD"
pvecm nodes

# The directory survives delnode. It must go, or the rejoin will refuse.
# You copied it in step 1. Confirm that copy exists first.
ls -R /root/rescue-"$DEAD"-*/

rm -rf /etc/pve/nodes/"$DEAD"
ls /etc/pve/nodes/

Configuration changeforget the old SSH identity

DEAD=pve03
DEADIP=192.0.2.13

ssh-keygen -R "$DEAD"
ssh-keygen -R "$DEADIP"
sed -i "/$DEAD/d" /etc/pve/priv/known_hosts 2>/dev/null
pvecm updatecerts --force

The rebuilt machine will present a new SSH host key. Without clearing the old one, the join fails with a host key verification error that reads like a security problem and is actually bookkeeping.

Step 5: Install the replacement

Install Proxmox VE on the new hardware. Three things must match the original exactly:

Version. Check pveversion on a surviving node and install the same release. A node that joins at a different major version puts the cluster into the mixed state described in the 8-to-9 upgrade runbook, without any of the planning that state requires.
Hostname. Exactly as before, including the domain if the original used an FQDN.
Primary IP. The corosync link address in particular. Corosync configuration on the surviving nodes is keyed to addresses; a rebuilt node on a different address is a different node.

Read-only / Safewhat the survivors expect

pveversion
cat /etc/pve/corosync.conf
grep -E 'ring[0-9]_addr|name:' /etc/pve/corosync.conf
cat /etc/hosts

Configuration changeon the rebuilt node: bring the network to match

# Recreate bonds, bridges and VLANs to match the original layout.
# Verify before joining - the join runs over this network.
ip -br addr
ip -br link
cat /etc/network/interfaces

MASTER=192.0.2.11
ping -c 3 "$MASTER"
ping -c 3 -M do -s 1472 "$MASTER"   # MTU sanity on the corosync path

Also match the repository configuration and bring the node fully up to date before joining. A node joining several point releases behind the cluster is a variable you do not need.

Step 6: Join

Cluster-wide riskon the rebuilt node

MASTER=192.0.2.11

pvecm add "$MASTER"
# It will ask for the root password of the target node and show a fingerprint.

sleep 30
pvecm status
pvecm nodes
ls /etc/pve/nodes/

After a successful join the node has the whole cluster filesystem: every storage definition, every other node’s guest configs, the HA configuration. If /etc/pve on the rebuilt node looks empty or partial, pve-cluster did not sync - stop and see the pmxcfs recovery runbook before continuing.

Step 7: Rebuild the Ceph daemons

Cluster-wide riskmonitor, manager, then OSDs

NODE=pve03

pveceph mon create
ceph mon stat

pveceph mgr create
ceph -s

# One OSD at a time, letting each backfill settle
DISK=/dev/nvme0n1
pveceph osd create "$DISK"
ceph -s
ceph osd tree

Creating an OSD triggers backfill: Ceph moves data back onto the new disks. This takes as long as it takes and it competes with guest I/O. Throttle it if the cluster is busy:

Configuration changeslow the backfill if guests are suffering

ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 1
ceph -s

# Restore the defaults once the cluster is clean again
ceph config rm osd osd_max_backfills
ceph config rm osd osd_recovery_max_active

Step 8: Return workload, one guest first

Service impact possibletest the rebuilt node before trusting it

NODE=pve03
VMID=9001

qm migrate "$VMID" "$NODE" --online
sleep 20
pvesh get /nodes/"$NODE"/qemu/"$VMID"/status/current --output-format yaml

# And back again - both directions must work
qm migrate "$VMID" pve01 --online

Migration in both directions is the check that catches a mismatched CPU model, an unreachable migration network, or a storage the rebuilt node cannot see. Run it before returning anything that matters.

Step 9: Restore what only the dead node had

For guests whose disks lived on the failed node’s local storage:

Destructiverestore from PBS onto the rebuilt node

STORE=pbs-main
VMID=112
TARGET=local-zfs

# What snapshots exist, and how old is the newest?
pvesm list "$STORE" | grep "vm/$VMID"

SNAP="backup/vm/112/2026-08-11T23:00:00Z"
qmrestore "$STORE:$SNAP" "$VMID" --storage "$TARGET"

qm start "$VMID"
qm status "$VMID"

Step 10: Close out

Re-add the node to backup jobs, replication jobs and HA rules.
Confirm monitoring sees the rebuilt node and its checks pass.
Confirm DNS still resolves correctly, forward and reverse.
Re-run one scheduled job of each kind by hand rather than waiting for the schedule.
Record: what was lost, what was restored and from when, how long the cluster ran degraded, and what existed only on that node.

That last item is the valuable one. A node that held something irreplaceable is a finding about your architecture, not about the hardware that failed.

Common patterns

Symptom	Likely cause	Resolution
`pvecm add` refuses: node already exists	The old `/etc/pve/nodes/NAME` directory is still there	Remove it on a surviving node (after copying it)
Join fails on host key verification	The rebuilt node has a new SSH host key	`ssh-keygen -R`, clear `/etc/pve/priv/known_hosts`, `pvecm updatecerts`
`/etc/pve` empty on the rebuilt node	`pve-cluster` did not sync after the join	Remove and rejoin from clean; see the pmxcfs runbook
Node joins but shows grey in the GUI	Certificates not regenerated	`pvecm updatecerts --force`, restart `pveproxy`
Guest config missing after the rebuild	The node directory was removed without the copy	Restore the guest from PBS instead
New OSDs stay `down`	Disk not zapped, or a stale LVM signature	`ceph-volume lvm zap DISK --destroy`, then recreate
Migration to the rebuilt node fails	CPU model mismatch, or a storage it cannot see	Compare `pvesm status` and the guest `cpu` setting across nodes
Corosync unstable after the join	The rebuilt node is on a different link address or MTU	Compare against `corosync.conf`; fix the network, not the config

Escalation

Escalate when:

The surviving cluster is not quorate. That outranks the rebuild.
Any guest config is missing from the rescue copy.
Ceph cannot return to HEALTH_OK after the OSDs are recreated.
The join fails twice from a clean install.
The measured data loss for any restored guest exceeds the agreed RPO.

Why this matters

The instinct when a node dies is to get the hardware back. The instinct is wrong by about two steps: the guests that can run right now are the ones whose configuration the cluster still holds, and getting them running on a surviving node takes minutes and does not depend on the replacement arriving at all.

Which makes the first real decision an inventory question, not a hardware one: for each guest, did its data live somewhere the failure could not reach? Shared storage means the guest is a config move away from running. Local storage means it is a restore, and its recovery point is whenever the backup last ran - a number that was decided months ago and is now non-negotiable.

The mechanical trap is smaller and easier to fall into. A cluster refuses to accept a rejoin from a name it already knows, so the rebuild requires deleting /etc/pve/nodes/NAME - the very directory that holds the dead node’s guest definitions. Copy it first. People who have done this once copy it first for the rest of their careers.