Windows guest troubleshooting

Windows failures on a hypervisor have a characteristic shape: the symptom is inside the guest, the cause is in the VM configuration, and the guest has no way to tell you that. A blue screen says INACCESSIBLE_BOOT_DEVICE, which is accurate and says nothing about the scsihw line somebody changed forty minutes ago.

The general method from Part XIX applies here and this lesson does not repeat it. What follows is the Windows-specific part: the four failures that come up most, what each actually is, and the recovery order for each — because on several of these the obvious first move makes the problem harder to undo.

The diagnostic order for a Windows guest that will not boot

Before touching the guest, place the failure in one of three categories by watching the boot. This takes thirty seconds and determines everything after.

Category one never needs the guest touched. Category two is where INACCESSIBLE_BOOT_DEVICE lives.

And the first question in every category is the same one: what changed?

VMID=9100

qm config "$VMID"
qm config "$VMID" --current 1

# Snapshots carry their own configuration; this shows what the VM looked like then
qm listsnapshot "$VMID"

INACCESSIBLE_BOOT_DEVICE

Stop code 0x7B. Windows started, loaded the drivers marked boot-critical, and found that none of them can reach the volume holding the rest of the operating system.

Three configuration changes produce it, and they are not equally common:

1. The disk controller changed. scsihw moved to or from virtio-scsi-single, or a disk moved between the sata, ide, scsi and virtio buses. This is the overwhelming majority.
2. The machine type changed. Moving between i440fx and q35, or a large jump in pinned machine version, relocates devices on the virtual bus. Lesson 1 explained why Windows pins the version at creation.
3. The disk is a restored or imported image whose original controller does not exist on this VM.

Recovery, in order of cost

First: revert the configuration change. This is free, it is instant, and it is skipped astonishingly often because the operator is already thinking about Windows repair options.

VMID=9100

qm stop "$VMID"
qm set "$VMID" --scsihw lsi
qm start "$VMID"

If that boots the guest, you have a working machine and a planned change to redo properly. Do not proceed to the repair procedures below; proceed to the pre-staging procedure.

Second: pre-stage the driver, then make the change. This is the correct way to change a Windows guest’s storage controller, and it works because Windows marks a storage driver boot-critical when it enumerates the device and installs the driver while the system is running normally.

VMID=9100
STORE=local-zfs

# 1. While the guest still boots on its old controller, give it a small disk
#    on the target controller so Windows enumerates and installs the driver.
qm set "$VMID" --scsihw virtio-scsi-single --scsi1 "$STORE:1"
qm start "$VMID"

# 2. Inside the guest: confirm the Red Hat VirtIO SCSI controller appears in
#    Device Manager with no warning, and that the 1 GiB disk is visible in
#    Disk Management. Then shut down cleanly.

# 3. Move the system disk to the new controller and remove the scratch disk.
qm set "$VMID" --delete scsi1
qm start "$VMID"

Step 2 is the verification that can fail, and it is the whole point of the procedure. If the controller shows a warning triangle or the scratch disk does not appear, stop: the driver is not loading, and moving the system disk now will produce the blue screen you were avoiding.

Third: inject the driver into the offline image. This is the recovery for a guest that is already showing the blue screen and whose configuration you cannot or should not revert — an imported image, or a change that has to stand.

Inside the recovery command prompt, find the real drive letters first. In Windows RE the installed system is very often not C: — the recovery environment claims that letter for its own scratch space, and running the next command against the wrong drive does nothing useful while appearing to succeed.

diskpart
list volume
exit

rem then confirm by looking for the expected directories
dir D:\Windows
dir E:\vioscsi

Then inject the driver. DISM’s offline driver servicing is documented for exactly this:

/Add-Driver — Adds third-party driver packages to an offline Windows image.

/Add-Driver {/Driver:<folder_containing_INF> | /Driver:<path_to_driver.inf>} [/recurse] [/ForceUnsigned]

If you point to a path and use the /Recurse option, all subfolders are queried for drivers to add.

Dism /Image:D:\ /Add-Driver /Driver:E:\vioscsi\2k22\amd64\vioscsi.inf

rem confirm it landed
Dism /Image:D:\ /Get-Drivers

/Get-Drivers is the verification: the documentation notes that pointing at an image lets you “determine what drivers are in the image, in addition to the state of the drivers (installed or staged)”. A driver that is not listed was not added, whatever the previous command appeared to say.

Then shut down, detach the ISOs, restore the boot order, and start.

The guest boots to a UEFI shell

Category one from the table. Windows is not involved; the firmware could not find a bootloader. Three causes, in order:

1. The boot order lost the disk, or still names a CD-ROM that is now empty. Check boot: order= in qm config.
2. efidisk0 is missing, empty or was replaced. The UEFI variables include the boot entries, so a fresh EFI variable store has no idea where Windows is. This is the failure mode behind Lesson 1’s warning that the efidisk is guest data: a disaster recovery procedure that copied “the disk” and not the EFI variable store produces exactly this.
3. The disk is attached but on a bus the firmware cannot boot from, or is not attached at all after a storage migration that half-completed.

For cause two, the recovery from the UEFI shell or the boot menu is to browse the EFI system partition and launch \EFI\Microsoft\Boot\bootmgfw.efi manually, which proves the installation is intact, then recreate the boot entry properly from Windows RE with bcdedit. Proving it first matters: if the manual launch boots Windows, the problem is a lost boot entry and nothing more, and you have converted an unknown outage into a five-minute fix.

Activation after a hardware change

Windows activation binds to a hardware fingerprint. Change enough of the virtual hardware and Windows concludes it is running on a different computer and asks to be reactivated.

What tends to trigger it on Proxmox:

• Changing the machine type family or making a large jump in pinned machine version.
• Changing the CPU model, particularly to or from host.
• Changing the firmware between SeaBIOS and OVMF.
• Adding or removing a TPM.
• Restoring to a different VM, or cloning without generalisation.

How much this hurts depends entirely on your licensing model, and this is the part that gets overlooked at design time:

Two consequences worth acting on:

If you are running MAK on a virtual estate, you have an availability dependency on your activation count. Any bulk change — a cluster-wide machine version bump, a CPU model standardisation — is also a bulk reactivation, and the key can run out. Plan the change in batches, or move to KMS or AD-based activation first.

If you are running KMS, the KMS host is a production dependency for every Windows guest. A KMS host that is unreachable does not break running machines immediately, because activation has a grace period — but it converts a routine hardware change into a countdown. Monitor it like any other dependency.

The prevention is Lesson 1 again: choose the firmware, machine type and CPU model before installing, and the estate never has to find out how its activation model behaves under change.

Clock drift

A Windows guest whose clock is wrong does not report a clock problem. It reports authentication failures, certificate validation errors, scheduled tasks that run at the wrong time, and log entries that are impossible to correlate with anything else in the estate. Kerberos rejects tickets outside a tolerance window — five minutes by default — so a domain-joined guest that drifts past that stops working, comprehensively, with error messages that point at the network.

Lesson 5 covered the hierarchy. The diagnostics:

# for VMID in $(qm list | awk '$3 == "running" { print $1 }'); do
GT=$(qm agent "$VMID" get-time 2>/dev/null) || continue
echo "$VMID $GT"
done

120 1786037640123456000
121 1786037640198765000
151 1786037518004321000

get-time returns nanoseconds since the epoch as the guest sees it. Compare against the host’s own clock: guest 151 above is about 122 seconds behind, which is well past the Kerberos tolerance and is an incident that has not been reported yet.

Inside the guest, w32tm /query /status and w32tm /query /source answer the two questions that matter: how far off is it, and who does it think it is synchronising with. A domain member whose source is anything other than a domain controller is misconfigured; a domain controller whose source is Local CMOS Clock has fallen out of the hierarchy entirely and is now an authoritative source of the wrong time for everything below it.

The Proxmox-side contributors to drift are worth knowing:

• A guest that was paused or suspended resumes with a stale clock and needs a moment to correct. After any maintenance that pauses guests, check the ones that matter.
• ostype not set to a Windows value gives a UTC hardware clock to an operating system that expects local time — an offset equal to your timezone, permanently, until something authoritative overrides it.
• Heavy host contention can starve a guest of the CPU time it needs to keep its own timekeeping accurate, which is one more reason the performance work in Lesson 4 is not only about performance.

Key takeaways

• Categorise the failure by what you see before touching anything: firmware, driver, system, or not-a-boot-problem. Category one never needs the guest opened.
• INACCESSIBLE_BOOT_DEVICE is a boot-critical driver that cannot reach the system volume. Reverting the configuration is free and should be tried first.
• Change a controller by pre-staging: attach a scratch disk on the target controller while the guest still boots, confirm the driver binds, then move the system disk.
• For a guest already down, Dism /Image: /Add-Driver injects the driver into the offline image, and /Get-Drivers verifies it landed. Never remove a boot-critical driver during a recovery.
• A UEFI shell means the firmware found no bootloader — check the boot order and efidisk0 before assuming Windows is damaged.
• Activation reacts to machine type, CPU model, firmware and TPM changes. KMS and AD-based activation absorb it; MAK consumes a finite pool.
• Clock drift presents as authentication failure. qm agent get-time measures it from the host across the whole estate, cheaply.

Knowledge check