Upgrade a cluster from PVE 8 to PVE 9, node by node

Verified against Proxmox VE 9.2.4 upgrading from 8.4, with Ceph Squid and Proxmox Backup Server 4.2.5.

A major PVE upgrade is not a patch. It moves the node across a Debian release boundary, replaces the kernel, QEMU, corosync and Ceph packages, and rewrites parts of the on-disk configuration. The cluster survives it because you do one node at a time and stop at the first thing you cannot explain - not because the upgrade is safe.

The single most useful framing: each node has exactly one moment where rollback stops being cheap. Before apt dist-upgrade runs, reverting is two file edits. After it, the node is a reinstall. Everything in this runbook exists to make sure you arrive at that moment having already answered every question.

When to use this runbook

• Proxmox VE 8.4 is approaching or past end of support.
• A feature, driver or hardware platform requires PVE 9.
• The rest of the estate has moved and this cluster is the outlier.

Do not use it to upgrade a cluster that is currently unhealthy. An upgrade is not a repair, and a degraded Ceph cluster or a flapping corosync link becomes considerably harder to diagnose once the versions differ between nodes.

Blast radius

One node at a time, and the guests on it. But the honest blast radius is larger than that, because a mixed-version cluster has properties the homogeneous one did not:

during the window     guests can migrate 8 -> 9, but NOT 9 -> 8
                      an upgraded node is a one-way destination
                      corosync speaks across versions, but do not linger
                      Ceph daemons must not be mid-upgrade on two nodes at once
after the last node   normal again, once the deferred cluster-wide steps are done

Write the node order down before you start, and the guest placement for each step. Improvising guest placement during a mixed-version window is how a critical VM ends up on a node that cannot hand it back.

Step 1: Get to the starting line

The upgrade guide assumes a fully updated 8.4. That is not a formality - the 8.4 point releases contain the compatibility shims the upgrade path depends on.

# Run on each node and compare the output side by side
pveversion -v
cat /etc/debian_version
pvecm status
ha-manager status

apt update
apt dist-upgrade
pveversion
# Reboot if a new kernel was installed; do that node-by-node too.

A node that is on a different 8.4 point release from its peers is a variable you do not want in the upgrade. Level them first.

Step 2: Run the checker, and treat WARN as work

pve8to9 --full 2>&1 | tee "/root/pve8to9-$(hostname -s).txt"
grep -cE '^FAIL' "/root/pve8to9-$(hostname -s).txt"
grep -E '^(WARN|FAIL)' "/root/pve8to9-$(hostname -s).txt"

Zero FAIL is the bar to proceed. WARN is the interesting category: the checker cannot decide for you, so each one needs a person to read it and write down why it is acceptable. The ones that most often matter:

Step 3: Empty the node, properly

NODE=pve02

# HA moves services away instead of treating the reboot as a failure
ha-manager crm-command node-maintenance enable "$NODE"
sleep 30
ha-manager status

# Anything left behind?
pvesh get /nodes/"$NODE"/qemu --output-format json | head -40
pvesh get /nodes/"$NODE"/lxc  --output-format json | head -40

Migrate what can migrate, shut down what cannot. A guest with PCI passthrough, a host CD-ROM or a local-only disk will not live-migrate; decide its fate before the window rather than at 02:40.

NODE=pve02
RUNNING=$(pvesh get /cluster/resources --type vm --output-format json \
| grep -c "\"node\":\"$NODE\"")
echo "guests still assigned to $NODE: $RUNNING"

Step 4: Keep what you will want if this goes wrong

NODE=$(hostname -s)
STAMP=$(date +%Y%m%d-%H%M)
mkdir -p "/root/pre-upgrade-$STAMP"
cp -a /etc/apt/sources.list /etc/apt/sources.list.d "/root/pre-upgrade-$STAMP/"
cp -a /etc/network/interfaces "/root/pre-upgrade-$STAMP/"
cp -a /etc/pve/nodes/"$NODE" "/root/pre-upgrade-$STAMP/pve-node-config"
pveversion -v > "/root/pre-upgrade-$STAMP/pveversion.txt"
ls -la "/root/pre-upgrade-$STAMP"

Note that /root is local to this node. If the node is unrecoverable the copy goes with it - so also put the pve8to9 output and the version list somewhere off the node.

Step 5: Move the repositories to Trixie

grep -rhE '^(deb|Suites:|URIs:)' /etc/apt/sources.list /etc/apt/sources.list.d/ \
| sort -u

Every Proxmox-related and Debian-related suite moves from bookworm to trixie: the Debian base, pve-no-subscription or pve-enterprise, and the Ceph repository. PVE 9 uses the deb822 .sources format; the upgrade guide’s conversion is the authoritative sequence and it is short. Follow it rather than hand-editing, and then read the result back:

apt update
grep -rl bookworm /etc/apt/sources.list /etc/apt/sources.list.d/ || echo 'no bookworm references remain'
apt policy | head -30

A single forgotten third-party repository still pointing at Bookworm will either fail apt update or, worse, succeed and pull a package built against the old libraries. Resolve it before continuing.

Step 6: The dist-upgrade

apt update
apt dist-upgrade
# Read the summary. Look specifically for packages being REMOVED.
# A removal of a pve-* or ceph-* package is a stop condition, not a detail.

Two things demand attention while it runs:

Configuration file prompts. apt will ask about files you or your automation modified. The default - keep the local version - is usually wrong for Proxmox-managed files and usually right for files you deliberately customised. If you cannot say which category a file is in, choose to keep the local version and note it, because a kept file is recoverable and an overwritten one is not.

Anything about /etc/network/interfaces. Answer conservatively. A node that boots with the wrong network configuration is a console recovery, and the copy from Step 4 is what you will restore from.

Step 7: Reboot on the console

sync
systemctl reboot

Watch it on the out-of-band console. Ping coming back proves the NIC is up; it does not prove the root filesystem mounted read-write, that pve-cluster started, or that the ZFS pool imported. The console shows you a failure two minutes before SSH would have.

Step 8: Verify this node before touching the next

pveversion -v
cat /etc/debian_version
uname -r
systemctl --failed --no-pager
journalctl -b -p err --no-pager | tail -40

pvecm status
pvecm nodes
corosync-cfgtool -s
pvesm status
ha-manager status

ceph -s
ceph versions
ceph osd tree
ceph health detail

ceph versions during the window will legitimately show two releases - that is the mixed state. What it must never show is a daemon that failed to start. ceph -s must be HEALTH_OK or a warning you have read and understood before the next node is touched.

Step 9: Return workload gradually

NODE=pve02
VMID=9001

ha-manager crm-command node-maintenance disable "$NODE"
qm migrate "$VMID" "$NODE" --online
sleep 20
pvesh get /nodes/"$NODE"/qemu/"$VMID"/status/current --output-format yaml

Move one guest you can afford to break, confirm it runs and serves, and only then return the rest. The first guest on a freshly upgraded node is a test, and it should be chosen as one.

Step 10: Repeat, then close the mixed state

Work through the remaining nodes in the recorded order. After the last one:

for N in $(pvecm nodes | awk '/^ *[0-9]/ {print $3}'); do
echo "== $N"
ssh "$N" 'pveversion; cat /etc/debian_version; apt list --upgradable 2>/dev/null | tail -n +2 | wc -l'
done

Then re-enable what you froze in Step 1 - backup jobs, replication jobs, automation - and run one of each by hand before trusting the schedule.

VMID=9001
SRC=pve01
DST=pve02

qm migrate "$VMID" "$DST" --online && sleep 15 && qm migrate "$VMID" "$SRC" --online
vzdump "$VMID" --storage pbs-main --mode snapshot --notes-template 'post-upgrade smoke test'

Migration in both directions and one successful backup is the smallest proof that the cluster is whole. A cluster where every node reports 9.x but live migration fails is not upgraded, it is broken quietly.

Rollback in practice

Common patterns

Escalation

Escalate when:

• A node does not return from the reboot.
• Ceph reports anything other than HEALTH_OK between nodes.
• Quorum is lost at any point.
• A pve8to9 WARN was accepted and the thing it warned about has now happened.
• The estate is stuck in mixed-version state at the end of the window.