low riskservice affecting~60 min

Diagnose and recover a VM that will not start

1 · Prerequisites

Confirm every item is in place before any state change.

Root shell on the node the guest is assigned to
The guest configuration file is present, and if it is not, that is a different problem - see the pmxcfs recovery runbook
A copy of the guest configuration is taken before any change, so every edit can be reverted
It is known whether this guest ever started successfully on this node, because a guest that has never run here is a compatibility question rather than a fault
The change history is available: what was modified on the guest, the node or the storage since it last started

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· qm status VMID reports the current state, and it is stopped rather than running or locked
· qm start VMID run in the foreground produces an actual error message, which is the primary evidence
· pvesm status shows every storage referenced by the guest configuration as active
· df -h and pvesm status show free space, because a full storage produces misleading start failures
· free -g on the node shows enough memory for the guest
· journalctl -u pvedaemon -n 100 covers the start attempt

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Run qm start in the foreground and capture the exact error - do not work from the GUI summary
2Copy the guest configuration file before making any change to it
3Classify the error against the known categories: lock, storage, missing volume, memory, passthrough, CPU, firmware, or configuration
4Confirm the storages the configuration references are active and the referenced volumes exist
5If the guest is locked, establish why before clearing the lock, and never clear it while the owning task may be alive
6If a volume is missing, find out whether it exists elsewhere on the storage before assuming it is gone
7Reproduce the failure with the generated QEMU command line, which reports errors the wrapper hides
8Apply the smallest change that addresses the identified cause, one change at a time
9Start the guest and verify it reaches its operating system, not merely that the process exists
10If the configuration cannot be repaired, restore the guest from backup rather than accumulating edits
11Record the cause and revert any diagnostic change that is not part of the fix

4 · Verification

Confirm the procedure actually fixed the problem.

✓qm start VMID exits successfully with no error output
✓qm status VMID reports running, and the process is still present sixty seconds later - a guest that starts and immediately exits is not fixed
✓The guest reaches its operating system: the guest agent answers, or the console shows a login prompt
✓An application-level check inside the guest passes
✓qm config VMID contains no leftover diagnostic change: no reduced memory, no removed device, no disabled passthrough that was only meant to be temporary
✓The guest survives a deliberate stop and start cycle, proving the fix is not a one-off
✓If the guest is HA-managed, ha-manager status shows it started rather than in error

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The copy of the configuration file taken before any edit is the rollback for every configuration change in this runbook
↶Restore it with a plain copy back into place and re-read the config to confirm
↶Diagnostic changes - reducing memory, detaching a passthrough device, switching the CPU type - must be reverted once the cause is identified, or the guest silently runs degraded forever
↶qm unlock is not reversible in a meaningful sense: if the lock was legitimate, clearing it allows a second operation on a guest that is already the subject of one
↶Restoring from backup overwrites the current disk. If any data was written since the backup - including by a partially started guest - it is lost
↶If a disk was detached during diagnosis, reattach it to the same bus and index; a disk that returns as a different device may change the boot order or the in-guest device names

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the storage owner if a referenced volume genuinely does not exist on the storage, because that is data loss and not a configuration fault
· Escalate to the service owner before restoring from backup, with the snapshot timestamp so the data loss is a decision rather than a surprise
· Escalate to the hardware owner if a passthrough device has disappeared from the host or changed its PCI address
· Escalate if several unrelated guests fail to start on the same node, because the fault is the node or the storage and not the guests
· Escalate if the guest starts and immediately exits with no logged reason, since that usually needs the QEMU command line reproduced with full output

Verified against Proxmox VE 9.2.4.

A VM that will not start is one of the few Proxmox failures that tells you exactly what is wrong, if you ask it in the right place. qm start run in a shell prints the real error. The GUI shows a task that failed. Those are not the same amount of information, and most wasted time in this incident comes from working off the second one.

So the discipline is short: get the error, classify it, change one thing. Almost every start failure is one of eight causes, and the error text names which.

When to use this runbook

qm start fails for a specific guest.
A guest failed to start after a node reboot, migration or upgrade.
A guest starts and exits immediately.
HA reports a service in error state because it cannot start it.

If several guests on the same node fail, this is the wrong runbook - the fault is the node or its storage, and that is where to look.

Step 1: Get the real error

Read-only / Safeforeground start, full output

VMID=104

qm status "$VMID"
qm config "$VMID"

qm start "$VMID" 2>&1 | tee "/tmp/start-$VMID.log"

Read-only / Safeand the surrounding logs

VMID=104

journalctl -u pvedaemon -n 100 --no-pager | tail -40
journalctl -u pvestatd -n 50 --no-pager | tail -20
ls -t /var/log/pve/tasks/active 2>/dev/null
grep -rl "qmstart:$VMID" /var/log/pve/tasks/ 2>/dev/null | head -3

Step 2: Copy the config before you change it

Read-only / Safethe rollback for everything below

VMID=104
NODE=$(hostname -s)

cp -a "/etc/pve/nodes/$NODE/qemu-server/$VMID.conf" \
    "/root/$VMID.conf.$(date +%Y%m%d-%H%M%S)"
ls -la /root/"$VMID".conf.*

Every fix below edits the configuration. Diagnosis that involves removing a device to see if it helps needs a way back, and “I will remember what it said” is not one.

Step 3: Classify the error

Error text contains	Cause	Go to
`VM is locked (backup)` / `(migrate)` / `(snapshot)`	Another operation owns it	Step 4
`storage 'X' is not online` / `does not exist`	Storage unavailable	Step 5
`volume 'X' does not exist` / `no such file`	Missing disk image	Step 6
`cannot allocate memory` / `hugepages`	Memory unavailable on the host	Step 7
`no IOMMU detected` / `device does not exist` / `vfio`	Passthrough	Step 8
`host doesn't support requested feature`	CPU model	Step 9
`unable to find EFI vars` / `efidisk` / `tpmstate`	Firmware volumes	Step 10
`unknown setting` / `property is not defined`	Bad config edit	Step 11
Starts, then exits within seconds, no clear error	Needs the raw command line	Step 12

Step 4: Locked

Read-only / Safefind out what holds the lock

VMID=104

qm config "$VMID" | grep '^lock:'

# Is the owning task still alive?
pvesh get /nodes/"$(hostname -s)"/tasks --output-format json | head -c 2000
echo
ps -ef | grep -E "[v]zdump|[q]m migrate|[q]mrestore" | head

Service impact possibleclear a genuinely stale lock

VMID=104

qm unlock "$VMID"
qm config "$VMID" | grep '^lock:' || echo 'lock cleared'
qm start "$VMID"

A backup lock left by a killed vzdump is the common legitimate case, and it is safe once you have confirmed no vzdump process exists for that VMID on any node.

Step 5: Storage not available

Read-only / Safewhich storage, and why is it down

VMID=104

qm config "$VMID" | grep -E '^(scsi|virtio|sata|ide|efidisk|tpmstate)[0-9]'
pvesm status
grep -A8 -n 'nfs\|cifs\|rbd\|iscsi' /etc/pve/storage.cfg | head -40

Read-only / Safethe usual reasons a storage is inactive

# NFS or CIFS: the server or the mount
showmount -e 192.0.2.30 2>/dev/null
findmnt -t nfs,nfs4,cifs

# Ceph
ceph -s 2>/dev/null | head

# ZFS
zpool status -x
zpool list

# The storage may simply be restricted to other nodes
grep -B2 -A6 'nodes' /etc/pve/storage.cfg

A storage restricted to a nodes list that does not include this node produces a start failure that looks like an outage and is a configuration line. It is worth checking early because it is free to check.

Step 6: The volume does not exist

Read-only / Safelook before concluding it is gone

VMID=104
STORE=local-zfs

qm config "$VMID" | grep -E '^(scsi|virtio|sata|ide)[0-9]'
pvesm list "$STORE" | grep "vm-$VMID-"

# And every other storage - it may have been moved
for S in $(pvesm status | awk 'NR>1 && $3=="active" {print $1}'); do
pvesm list "$S" 2>/dev/null | grep "vm-$VMID-" && echo "  ^ found on $S"
done

If the volume exists on a different storage than the config names, the fix is to correct the config - a previous move probably failed partway.

Configuration changereattach a volume the config lost track of

VMID=104

# Attach unreferenced volumes belonging to this VMID as unusedN entries
qm rescan --vmid "$VMID"
qm config "$VMID" | grep '^unused'

# Then attach the right one to the right bus
qm set "$VMID" --scsi0 local-zfs:vm-104-disk-0
qm config "$VMID" | grep scsi0

If the volume genuinely does not exist on any storage, this is data loss, not a configuration fault. Stop editing and go to the restore path in Step 13.

Step 7: Memory

Read-only / Safewhat the host has versus what the guest wants

VMID=104

free -g
qm config "$VMID" | grep -E '^(memory|balloon|hugepages|numa)'

# What is already committed on this node
pvesh get /nodes/"$(hostname -s)"/qemu --output-format json | tr ',' '\n' | grep -c vmid

# Hugepages, if the guest requests them
grep -i huge /proc/meminfo

A guest configured with hugepages will not start if the host has not reserved enough of them, and the reservation does not survive a reboot unless it is in the kernel command line. That is the version of this failure that appears only after a node restart.

Step 8: Passthrough

Read-only / Safedoes the device still exist, at the same address

VMID=104

qm config "$VMID" | grep -E '^(hostpci|usb)'
lspci -nnk | grep -A3 -iE 'nvidia|amd/ati|ethernet|nvme' | head -30
ls -la /sys/kernel/iommu_groups/ | head
dmesg | grep -iE 'iommu|vfio' | tail -20

PCI addresses change when hardware is added, removed or moved between slots, and after some firmware updates. A hostpci0: 0000:41:00 that worked last month may now point at nothing.

Configuration changediagnostic only: start without the device

VMID=104

qm set "$VMID" --delete hostpci0
qm start "$VMID"
# If it starts, the passthrough is the cause.
# Then RESTORE the config from the Step 2 copy and fix the address properly.

That is a test, not a fix. A guest left running without the GPU it exists to use is a resolved ticket and an unresolved problem.

Step 9: CPU model

Read-only / Safewhat the guest asks for, what the host has

VMID=104

qm config "$VMID" | grep -E '^(cpu|args)'
lscpu | grep -E 'Model name|Flags' | cut -c1-200
qm cpu 2>/dev/null | head -20

This is the failure that appears after migrating a guest to a node with an older CPU, or after a fleet with mixed generations gets a new machine. A guest with cpu: host sees the source CPU’s full feature set and demands it on arrival.

The correct fix is a named CPU model that every node in the cluster supports, applied to the guest permanently - not a one-off edit to get it running today.

Step 10: Firmware volumes

Read-only / Safeefidisk and tpmstate

VMID=104

qm config "$VMID" | grep -E '^(bios|efidisk|tpmstate|machine)'

STORE=local-zfs
pvesm list "$STORE" | grep -E "vm-$VMID-(disk|cloudinit)"

An OVMF guest without its efidisk0 has nowhere to keep boot variables. Depending on the exact state it either refuses to start or boots to an EFI shell. This is the classic leftover from a partial storage migration where only the data disk was moved.

Step 11: A bad configuration edit

Read-only / Safecompare against the last known good

VMID=104
NODE=$(hostname -s)

diff -u /root/"$VMID".conf.* "/etc/pve/nodes/$NODE/qemu-server/$VMID.conf" | head -40

If the config was edited by hand or by automation and the guest stopped starting, restore the copy and reintroduce the change one line at a time.

Step 12: Reproduce with the real command line

When the error is unhelpful or the guest exits immediately, take the wrapper out of the picture.

Read-only / Safethe QEMU command PVE would run

VMID=104

qm showcmd "$VMID" --pretty | tee "/tmp/qemu-$VMID.txt"
wc -l "/tmp/qemu-$VMID.txt"

Service impact possiblerun it by hand and read what QEMU says

VMID=104

# Run the printed command directly. QEMU's own stderr is far more
# specific than the task wrapper's summary - it names the file it could
# not open, the feature the host lacks, or the device that is missing.
bash "/tmp/qemu-$VMID.txt"

Running the guest this way starts it outside PVE’s supervision. Stop it and start it properly with qm start once the cause is found - a guest running from a hand-issued command line is invisible to HA and to the task system.

Step 13: When the configuration cannot be repaired

Read-only / Safewhat backups exist and how old

VMID=104
STORE=pbs-main

pvesm list "$STORE" | grep "vm/$VMID"

Destructiverestore, overwriting the current guest

VMID=104
STORE=pbs-main
TARGET=local-zfs
SNAP="backup/vm/104/2026-08-11T23:00:00Z"

# This DESTROYS the current disks for this VMID.
qmrestore "$STORE:$SNAP" "$VMID" --storage "$TARGET" --force 1
qm start "$VMID"
qm status "$VMID"

Verify properly

Read-only / Safestarted is not the same as working

VMID=104
GUESTIP=192.0.2.40

qm start "$VMID"
sleep 60
qm status "$VMID"

# Still running a minute later? A guest that exits at t+5s looks fine at t+2s.
qm agent "$VMID" ping
qm agent "$VMID" get-osinfo | head

curl -sS -o /dev/null -w '%{http_code}\n' "http://$GUESTIP:8080/healthz"

# And no diagnostic leftovers
qm config "$VMID" | grep -E '^(memory|cpu|hostpci|balloon)'

Then stop and start it once more. A guest that starts because of a transient condition will fail again, and finding that out now is better than finding it out during the next node reboot.

Common patterns

Symptom	Likely cause	Resolution
`VM is locked (backup)` after a killed backup	Stale lock	Confirm no `vzdump` process, then `qm unlock`
Fails only after a node reboot	Hugepages not reserved, or a storage not mounted at boot	Check `/proc/meminfo` and `findmnt`
Fails after a migration	CPU model, or a storage the new node cannot see	Named CPU type; `pvesm status` on that node
Boots to EFI shell	`efidisk0` missing or on a different storage	Move or recreate it, then fix boot order
Starts and exits in seconds	Passthrough, memory, or a QEMU-level error	`qm showcmd` and run it by hand
`volume does not exist` but the disk is there	Config points at the wrong storage after a failed move	`qm rescan`, reattach correctly
Several guests fail on one node	Node or storage fault	Stop looking at the guests
HA service in `error`, guest starts by hand	HA state is stale	`ha-manager set SERVICE --state started` after the fix

Escalation

Escalate when:

A referenced volume does not exist on any storage.
A restore from backup is required.
A passthrough device has vanished or moved.
Several unrelated guests fail on the same node.
The guest exits with no logged reason after the command line is reproduced.

Why this matters

This is the failure where the pressure to guess is highest and the payoff for reading is largest. The error message names the cause in the great majority of cases, and the single habit that separates a ten-minute fix from a two-hour one is running qm start in a shell instead of clicking Start and reading a task summary.

The second habit is copying the config before editing it. Diagnosis here is inherently a sequence of small mutilations - detach the GPU, drop the memory, change the CPU type - and without a baseline you end up with a guest that boots, a config nobody can explain, and a passthrough device that quietly stopped being used three months ago.

The one genuinely dangerous command in the whole procedure is qm unlock. Every other step is recoverable from the config copy. That one, used on a guest whose migration or restore is still in flight, gives you two hypervisors writing the same disk - the failure this platform spends considerable effort making impossible.