Machine types, QEMU versions and migration compatibility

The sentence that brings people to this subject is always some version of:

I am halfway through upgrading the cluster. Two nodes are done. I cannot migrate VM 118 onto either of them and I cannot work out why.

Machine versions are the answer often enough to be worth learning before the upgrade rather than during it. Proxmox VE 9.2 ships QEMU 11.0, which means every cluster that upgrades to it crosses a QEMU version boundary, which means this is live for every reader right now.

What a machine type is

A machine type is the virtual motherboard. It fixes which chipset the guest sees, which buses exist, where devices sit on them, and dozens of smaller behaviours that an operating system inspects at boot and then assumes will not change.

Proxmox offers two chipsets for x86 guests:

The machine option carries more than the chipset:

--machine [[type=]<machine type>] [,aw-bits=<number>]
          [,enable-s3=<1|0>] [,enable-s4=<1|0>]
          [,viommu=<intel|virtio>]

Versioning is the part that matters

From the documentation:

Each machine type is versioned in QEMU and a given QEMU binary supports many machine versions. New versions might bring support for new features, fixes or general improvements.

So pc-i440fx-11.0 is a specific revision of the virtual motherboard, and a QEMU 11.0 binary can also present pc-i440fx-10.0, pc-i440fx-9.2 and a long tail of older ones. That backwards range is what makes upgrades survivable.

Proxmox additionally appends its own revision when it changes something independently of a QEMU release, producing types such as pc-i440fx-9.2+pve1.

# qm config 118 | grep -E '^(machine|ostype|name):'

name: app-frontend-01
machine: pc-i440fx-9.2+pve1
ostype: l26

The rule, from the wiki: a version suffix such as pc-i440fx-9.2 means the version is pinned; no suffix means the VM automatically uses the latest available version.

set -euo pipefail

for vmid in $(qm list | awk 'NR>1 {print $1}'); do
m=$(qm config "$vmid" | awk -F': ' '/^machine:/ {print $2}')
name=$(qm config "$vmid" | awk -F': ' '/^name:/ {print $2}')
os=$(qm config "$vmid" | awk -F': ' '/^ostype:/ {print $2}')
case "$m" in
  *-[0-9]*.[0-9]*) printf '%-6s %-28s %-8s pinned   %s\n' "$vmid" "$name" "$os" "$m" ;;
  '')              printf '%-6s %-28s %-8s FLOATING (default type)\n' "$vmid" "$name" "$os" ;;
  *)               printf '%-6s %-28s %-8s FLOATING %s\n' "$vmid" "$name" "$os" "$m" ;;
esac
done

# What the running QEMU on this node actually offers:
qemu-system-x86_64 -machine help | head -20

Windows is pinned, Linux is not

This asymmetry is deliberate and it is documented:

For Windows guests, the machine version is pinned during creation, because Windows is sensitive to changes in the virtual hardware.

For everything else:

the Latest machine version is used by default. This means that after a fresh start, the newest machine version supported by the QEMU binary is used.

Windows inspects hardware at install time, binds drivers to specific device paths, and treats significant changes as a different computer — which affects driver binding and, in some editions, activation. Linux enumerates hardware at every boot and mostly does not care.

Live migration and the mixed-version cluster

The documented behaviour:

For operations on a running VM, such as live migrations, the running machine version is saved to ensure that the VM can be recovered exactly as it was.

A live migration hands the guest’s exact hardware state to the destination. If the destination QEMU cannot present the same machine version, the migration cannot happen — refusing is the correct behaviour, because the alternative is changing hardware under a running operating system.

Since a newer QEMU supports older machine versions and an older QEMU cannot support versions that did not exist when it was built, the rule is directional:

set -euo pipefail

# Pin one VM to a version every node in the cluster can present.
qm set 118 --machine pc-i440fx-9.2+pve1

# Pin every currently floating Linux guest on this node before an upgrade.
for vmid in $(qm list | awk 'NR>1 {print $1}'); do
cfg=$(qm config "$vmid")
printf '%s' "$cfg" | grep -q '^machine:.*-[0-9]' && continue
printf '%s' "$cfg" | grep -q '^ostype: win' && continue
echo "pinning $vmid"
qm set "$vmid" --machine pc-i440fx-9.2+pve1
done

Upgrading a machine version deliberately

Pinning is not free forever. A VM pinned to a five-year-old machine version misses fixes and features and eventually reaches a QEMU that no longer carries that version. The wiki’s procedure, in order:

1. Decide the target version. Usually the latest the cluster supports.
2. Review known issues for every intermediate version, not only the target. You are crossing all of them at once.
3. Edit the Machine property to the target version.
4. Reboot the VM through the interface — rebooting from within the guest is not enough.

Two additional cautions from the same page: schedule it in a maintenance window, because there is downtime, and take a working backup first.

set -euo pipefail
VMID=118

# 1. Record the current value. This is the rollback.
OLD=$(qm config "$VMID" | awk -F': ' '/^machine:/ {print $2}')
[ -n "$OLD" ] || OLD='(unset - was following latest)'
printf 'VM %s current machine: %s\n' "$VMID" "$OLD"

# 2. Confirm a recent backup exists before touching anything.
qm config "$VMID" > "/root/vm-$VMID-preupgrade.conf"

# 3. Apply and restart through PVE, not from inside the guest.
qm set "$VMID" --machine pc-i440fx-11.0
qm shutdown "$VMID" --timeout 300
qm start "$VMID"

# 4. Verify from inside the guest that devices came back, then:
#    rollback is  qm set $VMID --machine "$OLD"  followed by another restart.

Knowledge check