Major upgrade rolling lab

This lab walks through upgrading a 3-node PVE 9.2 cluster to 9.3 with rolling node-by-node upgrades, keeping the cluster in service throughout.

Steps

1. Pre-flight: confirm cluster health

pvecm status
# Expected: 3 nodes, quorum, all online

ceph -s
# Expected: HEALTH_OK, all PGs active+clean

# Take fresh backups of every VM (defence in depth)
vzdump --all --storage pbs-lab --mode snapshot

2. Identify the upgrade order

For a rolling upgrade, drain one node at a time:

# Start with node 3 (lowest priority VMs there)
# Then node 2
# Then node 1 (typically the most VMs)

Document the order in your change log.

3. Drain node 3

# On node 3, set it to maintenance mode (HA-managed VMs will migrate)
pveum node modify pve-03 --maintenance 1

# Manually migrate any non-HA VMs
for vmid in $(pvesh get /nodes/pve-03/qemu --output-format json | jq -r '.[].vmid'); do
  echo "Migrating VM $vmid to pve-01 or pve-02..."
  qm migrate $vmid pve-01 --online 2>&1 || true
done

Verify node 3 is empty:

pvesh get /cluster/resources --type vm | jq '.[] | select(.node=="pve-03")'
# Expected: empty

4. Upgrade node 3

ssh root@pve-03

apt update
apt full-upgrade -y
# Or for a major version bump:
# sed -i 's/bookworm/trixie/' /etc/apt/sources.list
# apt update && apt full-upgrade -y

reboot

After reboot:

pveversion -v
# Expected: 9.3.x or later
pvecm status
# Expected: 3 nodes, quorum, this node "online"

5. Restore node 3 to service

pveum node modify pve-03 --maintenance 0

Watch HA rebalance VMs if HA groups favour node 3.

6. Repeat for node 2 and node 1

Follow the same procedure for the other two nodes. After each:

pvecm status
ceph -s
# Verify cluster and storage remain healthy

7. Post-upgrade verification

# Check the cluster for any pending changes
pvesh get /cluster/tasks --typefilter all --since "1 hour ago"

# Verify all VMs started successfully
pvesh get /cluster/resources --type vm | jq '.[] | {vmid, node, status, running: .status}'
# Expected: all VMs running on their expected nodes

# Test live migration across all node pairs
qm migrate <test-vmid> pve-02 --online
qm migrate <test-vmid> pve-03 --online
qm migrate <test-vmid> pve-01 --online

8. Clean up

# Remove old kernels on each node
apt autoremove --purge -y

# Verify nothing else needs upgrading
apt list --upgradable

Verification

• All three nodes report the same PVE version
• All VMs survived the upgrade cycle
• Live migration works in all directions
• Ceph (if used) reports HEALTH_OK
• No VM data loss

Notes

• For major version bumps (9.x → 10.x), follow the official upgrade guide exactly. Don’t skip reading the release notes.
• Do NOT use pvecm delnode during the upgrade — it changes quorum math in a way that can lock you out.
• If something goes wrong, PBS backups are the safety net. Restore a single VM as a test before committing to the full upgrade.

Cleanup

Restore any VMs that were moved out of their original node:

qm migrate <vmid> <original-node> --online