VM startup failures: debugging the most common boot problems

A VM that won’t start is the most visible symptom of any cluster problem. Users see it immediately; on-call gets paged. This lesson walks through the systematic diagnosis.

The startup sequence

When you run qm start 100, the following happens:

1. PVE checks the VM’s configuration for validity
2. PVE acquires the necessary locks on shared resources
3. PVE asks the configured storage backend to attach the disk
4. PVE asks the network bridge to attach the VM’s NIC
5. PVE launches qemu with the configured arguments
6. qemu initialises the VM’s virtual hardware
7. The VM’s BIOS / bootloader runs
8. The guest kernel boots
9. Services start

Failure at any step shows different symptoms.

Where to look first

# Task log (most useful)
cat /var/log/pve/tasks/active
# Or in the GUI: Datacenter → Task History → filter for VM ID

# qemu process output (only available if you start it manually)
qm start 100 --debug
# Or watch the qemu console from the GUI: VM → Console

# System log
journalctl -u qemu-server --since '5 minutes ago'

The task log shows the exact failure point. Match the error message against the common causes below.

Common failure modes

1. Storage backend not available

TASK ERROR: storage 'ceph-pool' is not online

The storage backend can’t be reached. Check:

# Ceph pool status
ceph -s
# If HEALTH_OK, Ceph is up

# NFS mount
mount | grep nfs
# If empty, the NFS export isn't mounted

# iSCSI session
iscsiadm -m session
# If empty, no iSCSI sessions

# Storage status in PVE
pvesm status

For each storage type, the fix is “make the storage available”:

• Ceph: bring up the cluster (check MONs)
• NFS: re-mount with mount -a
• iSCSI: re-login with iscsiadm -m node -l

2. Storage full

TASK ERROR: no space left on device

The disk is full. Either grow it or clean up:

# Resize the disk
qm resize 100 scsi0 +50G

# Or clean up old snapshots / backups
# Check current usage
pvesh get /nodes/pve-01/storage/local-zfs/content

3. Network bridge missing

TASK ERROR: bridge 'vmbr99' does not exist

The VM’s configured bridge doesn’t exist. Either create it or change the VM’s config:

# Create the bridge
# Edit /etc/network/interfaces to add vmbr99

# Or change the VM\'s bridge
qm set 100 --net0 virtio,bridge=vmbr0

4. Corrupted disk image

TASK ERROR: qemu-img: Could not open '/var/lib/vz/images/100/vm-100-disk-0.qcow2': Input/output error

The disk file is corrupted. Try:

# Check the file system on the underlying storage
zpool status tank
# Or
mdadm --detail /dev/md0

# Try qemu-img check (read-only)
qemu-img check -r /var/lib/vz/images/100/vm-100-disk-0.qcow2

# If qcow2 is corrupted, qemu-img check might find and report errors
# Restore from backup if the corruption is severe

5. Memory overcommit

TASK ERROR: cannot allocate memory

The host doesn’t have enough free memory for the VM. Options:

# Check host memory
free -h
# And over-commit state
cat /proc/meminfo | grep -E 'Commit|MemFree'

# Reduce the VM\'s memory
qm set 100 --memory 2048

# Or migrate other VMs off this host
qm migrate 101 pve-02

# Or add RAM to the host

6. CPU overcommit (less common)

PVE allows CPU overcommit (sum of VM cores > host cores) but has limit checks for some operations. If the start fails:

# Check current CPU allocation
pvesh get /cluster/resources --type node

# Reduce VM CPU count
qm set 100 --cores 2

# Or migrate VMs off

7. BIOS / firmware issue

The VM’s BIOS can’t find a bootable disk. Symptoms:

• qemu starts but the console shows “No bootable device”
• PVE shows TASK OK but the VM never reaches the OS

# Verify the disk is in the boot order
qm config 100
# Look at "boot: order=scsi0" or similar

# If wrong, set it
qm set 100 --boot order=scsi0

# For UEFI VMs, verify the EFI disk is configured
qm set 100 --bios ovmf
qm set 100 --efidisk0 local-zfs:1,efitype=4m,pre-enrolled-keys=0

8. Lock contention

TASK ERROR: VM is locked (clone)

The VM is locked by another operation (backup, migration, etc.). Wait for it to finish, or force-unlock:

# Check the lock
pvesh get /cluster/resources --type vm | jq '.[] | select(.vmid == 100) | .lock'

# Force-unlock (DANGEROUS — only if the locking operation is hung)
qm unlock 100

The PVE rescue boot

For VMs that won’t boot due to filesystem corruption or bootloader issues, PVE has a rescue mode:

1. Stop the VM if running
2. Get the rescue ISO or use a Linux live ISO
3. Boot the VM from the rescue media
4. Mount the VM’s disk and repair

# 1. Download a Linux live ISO
# Use Ubuntu Server or Debian netinst

# 2. Attach the ISO to the VM
qm set 100 --ide2 local:iso/ubuntu-22.04-live-server-amd64.iso,media=cdrom
qm set 100 --boot order=ide2

# 3. Start the VM (will boot from ISO)
qm start 100

# 4. Open the console and use the rescue shell

# 5. Once repaired, restore the boot order
qm set 100 --boot order=scsi0
qm set 100 --delete ide2

For ZFS root disks, you can also:

# On the host, import the VM's zvol as a block device
# Then use fsck or testdisk inside a rescue VM
zfs send tank/vm-storage/vm-100-disk-0@pre-fix | \
  zfs receive -o mountpoint=/mnt/vm100-fix tank/vm-storage-fix

A startup-failure triage checklist

When a VM won’t start, work through this in order:

• Check the task log for the exact error
• Storage: is the backend online? Is there space?
• Network: does the configured bridge exist?
• Memory: does the host have enough free?
• Disk: is the disk file readable?
• Boot order: is the boot disk correctly configured?
• Lock: is the VM locked by another operation?
• Config: does the VM config validate?

For each step, the fix is “make the dependency available” or “adjust the VM to use what’s available”.

Production considerations

• Document the fix path. When a VM fails to start, write down what failed and how you fixed it. The same VM class will fail the same way in the future.
• Test your fixes. Don’t just reboot; verify the fix actually worked. A “fix” that doesn’t address the root cause is a temporary measure.
• Have backups. If the disk is corrupted beyond repair, the only path forward is restoring from PBS. Without recent backups, the VM is permanently broken.

Common mistakes

• Forcing the issue. qm start --force skips validation and often makes things worse.
• Not reading the error. The task log error tells you exactly what’s wrong. Read it.
• Randomly restarting things. Restart the host, restart services, restart the VM. Most of these don’t fix the problem and obscure the diagnostic trail.
• Ignoring recurring failures. If a VM fails to start weekly, there’s a root cause (bad disk, full storage, recurring lock). Fix the cause, not the symptom.

Key takeaways

• Read the task log first; the error is specific.
• Triage: storage → network → memory → disk → boot order → lock.
• Use rescue boot for filesystem corruption.
• Have backups; without them, a corrupted disk is a lost VM.

Knowledge check