medium riskdata loss risk~90 min

Move a VM disk to different storage while it runs, and roll back a stalled move

1 · Prerequisites

Confirm every item is in place before any state change.

The target storage is visible to this node and has free space greater than the allocated size of the disk, not merely its used size
The target storage supports the disk format required: a thin-provisioned qcow2 cannot be moved to a raw-only block storage without conversion
The performance of the target storage is known to be adequate for the guest, because after the move the guest runs on it permanently
A current backup of the guest exists and was taken before the move began
The guest workload during the move window is understood - a heavy writer extends the mirror indefinitely
It is decided in advance whether the source volume is deleted automatically or retained for manual removal

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· qm config VMID identifies the exact disk key being moved and its current storage
· pvesm status shows the target storage as active with sufficient available space
· qm status VMID shows the guest running and unlocked, with no other task in progress for it
· The guest current write rate is measured or estimated, because the mirror must converge against it
· No backup, snapshot or replication job for this guest is scheduled to start during the move window
· The task log shows no previous failed move for this VMID leaving an orphaned volume behind

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Record the current disk configuration and the free space on both source and target storage
2Confirm no other operation owns the guest and no scheduled job will start during the window
3Start the move with deletion of the source disabled, so the source volume survives the operation
4Watch the mirror job progress through the QEMU monitor rather than only the task percentage
5If the mirror reaches ready and completes, the guest is now running on the target disk
6If the mirror will not converge, cancel it - which discards the target copy and leaves the guest on the source untouched
7Verify the guest is running from the new storage and the application is healthy
8Let the guest run on the new storage long enough to be confident, then remove the retained source volume deliberately
9Reconcile storage: confirm no orphaned volume is left on either storage

4 · Verification

Confirm the procedure actually fixed the problem.

✓qm config VMID shows the disk on the target storage with the expected size
✓The guest uptime is unchanged across the operation - a reset uptime means it restarted rather than mirrored
✓An application-level check that passed before the move passes after it
✓Guest I/O latency after the move is within the expected range for the target storage, measured rather than assumed
✓pvesm list on the source storage either still shows the retained volume, deliberately, or shows it gone - and which one is the case is known
✓pvesm list on the target storage shows the new volume, and its size matches the configured disk size
✓No leftover block job is reported by the QEMU monitor for this guest

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶While the mirror job is running or in ready state, cancelling it discards the target copy and leaves the guest running on the source. That is a complete rollback with no downtime
↶The completion of the mirror is the point of no return: after it, the guest reads and writes the target volume, and the source is a stale point-in-time copy
↶If the source volume was retained, rollback after completion is possible but is NOT a simple config edit - the source is stale by however long the guest has been running on the target
↶Rolling back to a stale source volume loses every write since the mirror completed. Treat it as a restore decision, with the same conversation, not as an undo
↶Never run the move with automatic deletion of the source on a first attempt against unfamiliar storage. The retained volume is the only cheap rollback that exists
↶If the guest crashes during the mirror, it restarts from the source volume; the partial target copy is garbage and must be removed

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the storage owner if the target storage fills during the move, because a full storage affects every guest on it, not just this one
· Escalate to the service owner before cancelling a move that has already completed, since that is a data-loss decision rather than a rollback
· Escalate if the mirror will not converge against the guest write rate, because the realistic options are a maintenance window or a different target
· Escalate if guest I/O latency after the move is materially worse - the move succeeded and the outcome is still wrong
· Escalate if an orphaned volume is found that predates this operation, because something failed silently earlier

Verified against Proxmox VE 9.2.4.

Moving a running guest’s disk to another storage uses a QEMU drive mirror: the hypervisor copies the existing blocks to the target while duplicating every new write to both sides. When the copy catches up, the job reports ready, and completing it switches the guest to the target volume.

The structure to hold in your head is that there are two very different moments. While the mirror runs, the source is the real disk and the target is a copy in progress - cancelling costs nothing. After completion, the target is the real disk and the source is a snapshot of the past that gets staler every second.

Which is why --delete 0 on the first pass is not caution, it is the entire rollback plan.

When to use this runbook

Migrating guests off a storage that is being decommissioned.
Moving a workload from spinning disks to NVMe, or from local to shared.
Rebalancing a storage that is filling up.
Converting a guest from local storage so it can live-migrate.

Step 1: Know exactly what you are moving

Read-only / Safethe disk, its storage, and its real size

VMID=104

qm config "$VMID"
qm config "$VMID" | grep -E '^(scsi|virtio|sata|ide|efidisk|tpmstate)[0-9]'
qm status "$VMID"

A guest usually has more than one volume, and the ones people forget are efidisk0 and tpmstate0. They are tiny and the guest will not boot without them. Moving only scsi0 to new storage and decommissioning the old one leaves a guest whose EFI variables live on a storage that no longer exists.

Read-only / Safespace on both sides, allocated not used

SRC=local-zfs
DST=ceph-rbd

pvesm status
pvesm list "$SRC" | head -20
pvesm list "$DST" | head -20

Step 2: Make sure nothing else owns the guest

Read-only / Safeno competing operations

VMID=104

qm config "$VMID" | grep '^lock:' || echo 'no lock'
cat /etc/pve/jobs.cfg 2>/dev/null | grep -A5 "$VMID"
pvesr status 2>/dev/null

# Anything running right now on this node
pvesh get /nodes/"$(hostname -s)"/tasks --output-format json | head -c 1500
echo

A backup job starting mid-mirror will either fail or block, and a replication job for a disk that is moving underneath it is worse. Suspend the schedule for the window rather than hoping the timing works out.

Step 3: Baseline the guest

Read-only / Safeso verification can fail afterwards

VMID=104
GUESTIP=192.0.2.40

qm agent "$VMID" exec -- /usr/bin/uptime -s 2>/dev/null
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' "http://$GUESTIP:8080/healthz"

# Current I/O latency, for comparison after the move
pvesh get /nodes/"$(hostname -s)"/qemu/"$VMID"/rrddata --timeframe hour --output-format json \
| head -c 800
echo

The latency baseline matters more than it sounds. A move that completes successfully onto a storage that is slower than the old one is a successful operation with a bad outcome, and without a before-number nobody can prove it.

Step 4: Start the move, keeping the source

Service impact possiblemove the disk, do not delete the source

VMID=104
DISK=scsi0
DST=ceph-rbd

qm move-disk "$VMID" "$DISK" "$DST" --delete 0

With a bandwidth limit, if the storage network is shared:

Service impact possiblecapped

VMID=104
DISK=scsi0
DST=ceph-rbd

# KiB/s
qm move-disk "$VMID" "$DISK" "$DST" --delete 0 --bwlimit 100000

Step 5: Watch the mirror, not the percentage

Read-only / Safethe block job's own view

VMID=104

qm monitor "$VMID"
# at the (qemu) prompt:
#   info block-jobs
#   quit

info block-jobs reports offset, length and whether the job is ready. The states in order:

State	Meaning	Can you cancel?
Running, offset climbing	Bulk copy in progress	Yes. Target copy is discarded
Running, offset near length, not `ready`	Catching up with new writes	Yes
`ready: true`	Fully mirrored; both copies live; awaiting completion	Yes - and this is the last cheap moment
Completed	Guest switched to the target	No. Source is now stale

Step 6: A mirror that will not converge

The same race as a live migration, with disk writes instead of memory writes. If the guest writes faster than the target absorbs, the offset approaches the length and never reaches ready.

Options, in order of preference:

Cap the guest’s I/O rather than the copy. A batch job or a log rotation is often the whole cause. Pausing it for ten minutes fixes the move.
Check the target is not the bottleneck. A target storage that is itself rebalancing or degraded will never keep up. ceph -s, or the ZFS pool status, before blaming the guest.
Do it offline. Stop the guest, move the disk cold, start it. For a heavy writer this is frequently faster in wall-clock terms than a mirror that never finishes.

Service impact possiblethe offline alternative

VMID=104
DISK=scsi0
DST=ceph-rbd

qm shutdown "$VMID" --timeout 300
qm move-disk "$VMID" "$DISK" "$DST" --delete 0
qm start "$VMID"
qm status "$VMID"

Step 7: Cancelling

Service impact possibleabort the mirror; the guest stays on the source

VMID=104

qm monitor "$VMID"
# at the (qemu) prompt:
#   info block-jobs
#   block-job-cancel drive-scsi0
#   info block-jobs
#   quit

The job name is drive- plus the disk key: drive-scsi0, drive-virtio1. Read it from info block-jobs rather than assuming.

After cancelling, the guest continues on the source volume with no downtime and no change. What remains is a partial copy on the target, which must be cleaned up:

Destructiveremove the abandoned target volume

VMID=104
DST=ceph-rbd

# Find volumes for this VMID that the config does NOT reference
pvesm list "$DST" | grep "vm-$VMID-"
qm config "$VMID" | grep -E '^(scsi|virtio|sata|ide)[0-9]'

# Remove only the one you have confirmed is unreferenced
VOL="$DST:vm-104-disk-1"
pvesm free "$VOL"

Read the config and the storage listing side by side before freeing anything. pvesm free does not ask twice.

Step 8: Verify after a completed move

Read-only / Safeconfig, uptime, application, latency

VMID=104
GUESTIP=192.0.2.40

qm config "$VMID" | grep -E '^(scsi|virtio|sata|ide)[0-9]'
qm status "$VMID"

# Continuous uptime - a reset means it restarted, not mirrored
qm agent "$VMID" exec -- /usr/bin/uptime -s 2>/dev/null

curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' "http://$GUESTIP:8080/healthz"

qm monitor "$VMID"
#   info block-jobs      -> must be empty
#   quit

Then compare I/O latency against the Step 3 baseline under real load, not immediately after the move while caches are cold. Give it an hour of normal traffic before deciding the new storage is acceptable.

Step 9: Remove the retained source, deliberately

Only after the guest has run on the target long enough that you would not want to go back.

Destructivefree the old volume

VMID=104
SRC=local-zfs

# Confirm the config does not reference it
qm config "$VMID" | grep -E '^(scsi|virtio|sata|ide)[0-9]'
pvesm list "$SRC" | grep "vm-$VMID-"

VOL="$SRC:vm-104-disk-0"
pvesm free "$VOL"
pvesm list "$SRC" | grep "vm-$VMID-" || echo 'source volume removed'

Read-only / Safecluster-wide orphan sweep

for S in $(pvesm status | awk 'NR>1 && $3=="active" {print $1}'); do
echo "== $S"
pvesm list "$S" | awk 'NR>1 {print $1}'
done > /tmp/all-volumes.txt

grep -h -oE 'vm-[0-9]+-disk-[0-9]+' /etc/pve/nodes/*/qemu-server/*.conf | sort -u > /tmp/referenced.txt
wc -l /tmp/all-volumes.txt /tmp/referenced.txt

Rollback

Stage	Rollback
Mirror running	`block-job-cancel`. Guest untouched, no downtime
Mirror `ready`	`block-job-cancel`. Still free - both copies were current
Mirror completed, source retained	Possible, but the source is stale. This is a restore decision
Mirror completed, `--delete 1` used	None. Restore from backup
Guest crashed mid-mirror	It restarts from the source. Remove the partial target volume

Common patterns

Symptom	Likely cause	Resolution
Move fails immediately: no space	Target sized against used, not allocated	Check the configured disk size against target free space
Mirror never reaches `ready`	Guest writes faster than the target absorbs	Pause the workload, or move offline
Target storage fills mid-move	Thin source, thick target	Cancel, free the partial volume, resize the plan
Guest I/O collapses during the move	Uncapped mirror saturating the storage network	`--bwlimit`
Guest boots to EFI shell after the move	`efidisk0` was left on the old storage	Move it too
Orphaned volume on the source after completion	`--delete 0` used and never followed up	Free it, and record the pattern
`pvesm free` refuses	The volume is still referenced, or in use	Re-read the config; do not force it
Guest slower after a successful move	The target is genuinely slower	Compare against baseline; this is a capacity decision

Escalation

Escalate when:

The target storage fills during the move.
A completed move needs reverting to a stale source.
The mirror cannot converge against the guest’s write rate.
Post-move latency is materially worse.
Orphaned volumes exist that predate this operation.

Why this matters

The whole procedure turns on one boundary: before completion the source is the truth, after completion the target is. Every safe action lives on one side of it, and the reason --delete 1 is dangerous is that it removes your ability to be wrong about which side you were on.

There is a second, quieter idea worth taking away. A mirror in ready state has two live copies of the disk, both current, both receiving every write. That is an unusual and rather useful position to be in - it means you can wait, at leisure, for a good moment to switch, and it means cancelling is genuinely free rather than merely cheap. Most storage operations do not offer that, and it is worth using rather than rushing past.

The failure that actually bites in production is duller than either: a guest moved to a target sized against its used space rather than its allocated space, filling a shared storage at 3am and taking out every other guest on it. Check the allocated size. It is the one number that is easy to look at and easy to not look at.