medium riskservice affecting~45 min

Live-migrate a running VM, and recover when it stalls

1 · Prerequisites

Confirm every item is in place before any state change.

Source and target nodes are members of the same quorate cluster
The guest disks are on storage both nodes can see, or the migration is explicitly planned as a local-disk migration with the extra time that implies
Source and target run the same Proxmox VE major version; migration to an older version is not supported
The target node has enough free memory for the guest, checked against actual free memory rather than nominal capacity
The migration network is known, and its bandwidth is known well enough to estimate how long the transfer takes
A maintenance note exists for the guest, because a live migration causes a brief pause even when it works perfectly

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· pvecm status reports Quorate: Yes
· qm config VMID shows no host-bound resources: no hostpci, no usb passthrough, no local CD-ROM iso, no host serial or parallel device
· qm config VMID disk lines reference a storage that pvesm status shows as active on the target node
· The target node free memory exceeds the guest memory plus overhead
· qm status VMID shows the guest running and unlocked
· The guest current memory dirty rate is understood, at least approximately, for anything write-heavy such as a database

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Confirm the guest is migratable at all: check for passthrough devices, local ISOs and unshared disks
2Confirm the target can host it: storage visible, memory available, CPU model compatible
3Record the pre-migration state: node, uptime, boot time, and an application-level health check that currently passes
4Start the migration, with a bandwidth limit if the migration network is shared
5Watch the task output for the memory transfer to converge, not for the percentage to move
6If it converges: confirm the guest is running on the target and the health check still passes
7If it stalls: decide between raising the allowed downtime, throttling the guest, or cancelling
8To cancel: issue migrate_cancel through the QEMU monitor while the source copy is still authoritative
9After a failed or cancelled migration, confirm the guest is running on exactly one node and clear any leftover lock only once that is proven
10Verify at the guest level: the guest did not reboot, the application answers, and no service restarted

4 · Verification

Confirm the procedure actually fixed the problem.

✓qm status VMID on the target reports running, and the same command on the source reports the guest does not exist there
✓The guest uptime is continuous across the migration - a reset uptime means it restarted rather than migrated
✓The config file has moved: /etc/pve/nodes/TARGET/qemu-server/VMID.conf exists and the source path does not
✓An application-level check that passed before the migration passes after it
✓qm config VMID shows no lock line left behind
✓The guest network is reachable from a client, not merely up inside the guest - a migration can land a guest on a node whose bridge lacks the VLAN
✓No error entries for this VMID in the task log after the migration completed

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Before the cutover, rollback is clean and complete: cancelling returns the guest to the source node still running, with nothing changed
↶The cutover is the point of no return. Once the guest is executing on the target, going back means another migration in the opposite direction
↶If the migration failed partway, the guest normally remains running on the source. Confirm that before doing anything else
↶A leftover lock is cleared with qm unlock, but ONLY after proving the guest is running on exactly one node - unlocking a guest that is genuinely mid-migration invites two copies
↶Never start the guest on the target to "recover" from a failed migration without first proving the source copy is stopped. Two running copies on shared storage destroys the guest filesystem
↶If the guest was migrated to fix a problem and the problem followed it, migrate it back rather than accumulating attempts on the new node

6 · Escalation

When the runbook isn't enough, contact:

· Escalate if the guest appears to be running on both nodes - stop everything else and deal with that first
· Escalate to the network team if migrations stall consistently on a specific link or between a specific pair of nodes
· Escalate to the service owner before raising migrate_downtime on a latency-sensitive guest, because the pause becomes user-visible
· Escalate to the storage owner if the migration fails because the target cannot see a storage that it should
· Escalate if a guest cannot be migrated at all and the reason is architectural - passthrough or local disks - because that is a design decision, not an incident

Verified against Proxmox VE 9.2.4.

A live migration copies a running guest’s memory to another node while the guest keeps executing, repeatedly re-sending the pages that changed since the last pass, until the remaining set is small enough to transfer inside an acceptable pause. Then it stops the guest on the source, sends the last pages, and starts it on the target.

That description contains the entire failure mode. If the guest dirties memory faster than the link can carry it, the remaining set never gets small enough, and the migration runs forever without failing. It is not stuck; it is losing a race. Knowing that changes what you do about it.

When to use this runbook

Evacuating a node for maintenance or an upgrade.
Rebalancing load across the cluster.
Moving a guest away from failing hardware while it is still serving.
A migration is already running and not finishing.

Step 1: Is this guest migratable at all?

Some guests cannot live-migrate, and finding that out at 02:00 during a node evacuation is avoidable.

Read-only / Safethe blockers, all in one config

VMID=104

qm config "$VMID"
qm config "$VMID" | grep -E '^(hostpci|usb|serial|parallel)' && echo 'HOST-BOUND DEVICE: cannot live-migrate'
qm config "$VMID" | grep -E 'cdrom|\.iso' && echo 'ISO attached: target must see the same storage'
qm config "$VMID" | grep -E '^(scsi|virtio|sata|ide)[0-9]'

Blocker	Why	What to do
`hostpci` - PCI passthrough (GPU, HBA, NIC)	The device exists only on this node	Offline migration only, or shut down and move
`usb` passthrough	Same	Same
Local CD-ROM or an ISO on local storage	The target cannot open the file	Detach the ISO, or move it to shared storage
Disks on local ZFS or LVM	The data is on this node	`--with-local-disks`, which copies the disk too and takes far longer
`cpu: host` with different CPU generations	The guest sees instructions the target lacks	Use a named model both nodes support
Guest is locked (backup, snapshot)	Another operation owns it	Wait, or resolve that operation

Read-only / Safecan the target actually take it

VMID=104
TARGET=pve02

# Storage visible on the target?
pvesh get /nodes/"$TARGET"/storage --output-format yaml | grep -E 'storage:|active:'

# Free memory on the target, in MB
pvesh get /nodes/"$TARGET"/status --output-format json | tr ',' '\n' | grep -E 'memory|free' | head

# What the guest wants
qm config "$VMID" | grep -E '^(memory|balloon|cores|cpu)'

Step 2: Record what “working” looks like now

Read-only / Safebaseline, so verification can fail

VMID=104

qm status "$VMID"
qm agent "$VMID" ping && echo 'guest agent responds'
qm agent "$VMID" get-osinfo 2>/dev/null | head

# An application-level check that passes right now
GUESTIP=192.0.2.40
curl -sS -o /dev/null -w '%{http_code}\n' "http://$GUESTIP:8080/healthz"

Without a check that passes before, “it is running” after the migration proves only that QEMU is executing. A guest can migrate perfectly and land on a node whose bridge does not carry its VLAN, and qm status will happily report running while nothing can reach it.

Step 3: Migrate

Service impact possiblethe migration itself

VMID=104
TARGET=pve02

qm migrate "$VMID" "$TARGET" --online

With a bandwidth cap, if the migration network is shared with anything that matters:

Service impact possiblecapped, so corosync survives the transfer

VMID=104
TARGET=pve02

# KiB/s. Leave headroom on a shared link.
qm migrate "$VMID" "$TARGET" --online --bwlimit 200000

Step 4: Read the progress properly

Read-only / Safewatch convergence, not percentage

VMID=104

# From another shell, follow the running task
pvesh get /nodes/"$(hostname -s)"/tasks --output-format json | head -c 2000
echo

# The QEMU monitor is the authoritative view
qm monitor "$VMID"
# then at the (qemu) prompt:
#   info migrate
#   quit

info migrate reports remaining RAM, transferred RAM and dirty sync count. The number that tells you whether this will finish is remaining RAM across successive polls:

Pattern	Meaning	Action
Remaining falling steadily	Converging normally	Wait
Remaining flat at a few hundred MB, dirty sync count climbing	Losing the race	Step 5
Remaining rising	The guest is dirtying faster than the link carries	Step 5, and cap expectations
Transferred far exceeds guest RAM	Many passes have been sent - a long-running non-convergence	Step 5

Step 5: A migration that will not converge

Three levers, in increasing order of user impact.

Raise the permitted downtime. The migration completes when the remaining memory can be sent within migrate_downtime seconds. The default is deliberately small. Raising it trades a longer pause for a migration that actually finishes.

Configuration changeallow a longer cutover pause

VMID=104

qm config "$VMID" | grep migrate_downtime
qm set "$VMID" --migrate_downtime 2
# Seconds. 2 finishes most stubborn guests. Anything above a few seconds
# is a visible outage to users - ask the service owner first.

Cap the guest instead of the link. If the guest is dirtying memory because of a batch job, pausing that job for two minutes converts an impossible migration into a trivial one. This is often the cheapest answer and it is the one nobody tries.

Accept an offline move. A short scheduled stop, an offline migration and a start is frequently less disruptive than forty minutes of a non-converging live migration followed by a multi-second stall anyway.

Service impact possiblethe honest alternative

VMID=104
TARGET=pve02

qm shutdown "$VMID" --timeout 300
qm migrate "$VMID" "$TARGET"
qm start "$VMID"
qm status "$VMID"

Step 6: Cancelling safely

Service impact possiblecancel while the source is still authoritative

VMID=104

qm monitor "$VMID"
# at the (qemu) prompt:
#   info migrate
#   migrate_cancel
#   info status
#   quit

Step 7: After a failure, find out where the guest actually is

Read-only / Safeexactly one node, verified on every node

VMID=104

# On EVERY node in the cluster:
hostname -s
ps -eo pid,cmd | grep "[k]vm -id $VMID" | head -1 || echo 'not running here'
ls -la /etc/pve/nodes/*/qemu-server/"$VMID".conf

The config file exists on exactly one node. If a failed migration left the config on the target while the process runs on the source, that mismatch is the thing to resolve - and it is resolved by establishing where the guest is genuinely executing, then moving the config to match.

Service impact possibleclear a stale lock, only after proving one copy

VMID=104

qm config "$VMID" | grep '^lock:'
# Only if you have confirmed on every node that exactly one QEMU process
# exists for this VMID:
qm unlock "$VMID"
qm status "$VMID"

Step 8: Verify at the guest level

Read-only / Safedid it migrate, or did it restart

VMID=104
TARGET=pve02
GUESTIP=192.0.2.40

ssh "root@$TARGET" "qm status $VMID"
ssh "root@$TARGET" "qm agent $VMID get-osinfo" 2>/dev/null | head

# Uptime must be continuous. A reset uptime means it rebooted.
qm agent "$VMID" exec -- /usr/bin/uptime -s 2>/dev/null

curl -sS -o /dev/null -w '%{http_code}\n' "http://$GUESTIP:8080/healthz"

Guest uptime is the check that separates a migration from a restart. A guest that quietly failed over and rebooted reports running and answers its health check - and has lost every in-memory session it had. Compare the boot time against the value from Step 2.

Common patterns

Symptom	Likely cause	Resolution
`can't migrate VM with local resources`	`hostpci`, `usb`, or a local ISO	Detach, or migrate offline
Migration never converges	Guest dirties memory faster than the link	Raise `migrate_downtime`, pause the workload, or migrate offline
Migration fails at start: storage not available	Target cannot see the storage	Check `pvesm status` on the target and the storage `nodes` restriction
Guest starts on target then immediately fails	CPU model mismatch, or missing CPU flags	Use a named CPU type supported by both nodes
Node fences during a migration	Migration traffic starved corosync	Cap `--bwlimit`; configure a dedicated migration network
Guest unreachable on the target, `running` reported	Target bridge lacks the VLAN or the bridge name differs	Compare `/etc/network/interfaces` between nodes
Guest left locked after a failed migration	The task died without releasing the lock	Prove one copy exists, then `qm unlock`
Migration is slow but converging	Bandwidth limit set too low, or a 1G link	Expected. Let it finish, or plan the link

Escalation

Escalate when:

The guest appears to be running on two nodes.
Migrations stall consistently between the same pair of nodes.
migrate_downtime needs raising on a latency-sensitive service.
The target cannot see a storage it should.
A guest is architecturally unmigratable and that is a problem for maintenance planning.

Why this matters

Live migration looks like a file copy and behaves like a race. The guest is writing to memory while you are copying that memory, so the question is never “how big is the guest” but “how fast does it dirty pages compared to how fast the link carries them”. A 64 GB idle VM migrates in a minute; a 4 GB busy database can run for an hour and never finish.

That reframing is what makes the stall actionable. A migration that is not converging is not stuck and will not error - it is losing, and your options are to speed up the link, slow down the guest, or accept a longer pause. Waiting is not one of them.

The safety property underneath is worth holding onto too. Before the cutover the source copy is authoritative and cancelling costs nothing; after it, the target is. Everything dangerous in this runbook - qm unlock, starting the guest manually, “recovering” a failed migration - comes from acting as if you know which side of that line you are on without having checked.