Moving disks between storage backends · Proxmox VE

Choosing a backend is a decision you make once. Moving between backends is the decision you make every time that first one turns out to have been wrong, or the hardware ages out, or the estate grows into something else.

Proxmox VE makes this genuinely easy, which is the source of most of the trouble: the command is short, it usually works, and the three things it does that you did not ask for are all invisible until later.

The commands

For a virtual machine:

qm disk move <vmid> <disk> [<storage>] [OPTIONS]

qm move-disk and qm move_disk are aliases for the same thing. Options that matter: --format (qcow2, raw, vmdk), --delete, --bwlimit, --digest, --target-disk, --target-vmid.

For a container:

pct move-volume <vmid> <volume> [<storage>] [<target-vmid>] [<target-volume>] [OPTIONS]

The volume is a config key — rootfs, mp0, unused3 — rather than a disk name, and the options are --bwlimit, --delete, --digest and --target-digest.

Service impact possiblemove a VM disk to another storage— Copies the disk to the destination storage and rewrites the guest configuration to point at the new volume. Works with the guest running for most backends. The source volume is KEPT unless --delete is given. Set a bandwidth limit so the copy does not saturate the storage everything else is using.

VMID=141
DISK=scsi0
DEST=shared-rbd

qm config "$VMID" | grep "^${DISK}:"

qm disk move "$VMID" "$DISK" "$DEST" --bwlimit 100000

qm config "$VMID" | grep -E "^${DISK}:|^unused"

Read-only / Safe

$ qm config 141 | grep -E '^scsi0:|^unused'

scsi0: shared-rbd:vm-141-disk-0,size=500G,discard=on
unused0: local-lvm:vm-141-disk-0

The default that keeps the old copy

--delete defaults to 0. The pct reference says it plainly: by default the original is kept as an unused volume entry. qm behaves the same way.

That default is right. A move is a copy, and keeping the source means a failed verification costs you a configuration edit rather than a restore. But it has a consequence that surprises people at exactly the wrong moment.

The migration finished and the old storage is still full

A team moved 60 guests off an ageing LVM-thin pool onto Ceph, over three evenings. Every move succeeded. The plan was to decommission the old disks the following week.

The old pool was still at 96%.

Every one of those 60 moves had left the source volume behind as an unusedN entry in the guest configuration. Nothing was orphaned — PVE knew about all of them and would have shown them in the GUI under each guest’s Hardware tab — but nobody had looked at 60 Hardware tabs.

Worse, the pool being nearly full then blocked the last few moves, and the error (“no space left”) pointed at the source storage, which reads like a storage fault on the thing you are trying to leave.

The fix is not to run every move with --delete 1. It is to make the cleanup a deliberate, reviewed second pass:

Complete and verify the moves first, with the sources retained.
Confirm each guest boots and its application answers on the new storage.
Then enumerate the unused volumes and remove them as one scheduled step, with the list written down.

That way you keep the rollback through the risky phase and reclaim the space when you no longer need it.

Read-only / Safefind every unused volume in the cluster— Read-only. Walks every VM and container configuration for unused entries and prints them with their owning guest. This is the list to review before a cleanup pass, and it is worth running periodically regardless — unused volumes also accumulate from failed migrations and hand-edited configs.

pvesh get /cluster/resources --type vm --output-format json \
| jq -r '.[] | [.node, .type, .vmid] | @tsv' \
| while IFS=$'\t' read -r node type vmid; do
    if [ "$type" = "qemu" ]; then cfg="qemu"; else cfg="lxc"; fi
    pvesh get "/nodes/${node}/${cfg}/${vmid}/config" --output-format json \
      | jq -r --arg v "$vmid" 'to_entries[] | select(.key|startswith("unused")) | [$v, .key, .value] | @tsv'
  done

Data-loss riskremove one unused volume, after verification— DELETES DATA. qm unlink removes the unused entry and, with --force, the underlying volume. There is no undo and no confirmation prompt. Run this only against a list you have reviewed, and only after the guest has been verified on its new storage.

VMID=141

qm config "$VMID" | grep '^unused'

qm unlink "$VMID" --idlist unused0 --force 1

pvesm list local-lvm --vmid "$VMID"

Format conversion happens whether you ask or not

A move between storages of different levels converts the image format, because it must.

From	To	What happens
`qcow2` on a directory/NFS store	LVM-thin, RBD, ZFS zvol	Becomes a raw block volume
`raw` on LVM-thin	Directory or NFS	Becomes a file — `raw` by default, `qcow2` if you ask
`qcow2`	`qcow2` on another file store	Stays `qcow2`

--format lets you choose on file-level destinations. On block-level destinations there is nothing to choose: the backend holds raw volumes.

The consequence to plan for is not the format itself but what the format was providing.

Snapshots do not survive the move

A guest’s snapshot history is a property of the storage that holds it — qcow2’s internal snapshots, ZFS snapshots, thin-pool snapshots, RBD snapshots. They are four different mechanisms, and a disk move does not translate between them.

In practice, PVE will refuse to move a disk that has snapshots on backends where the operation cannot be represented, and where it does proceed, the snapshot history is not what arrives.

Decide before the move:

Discard them. Delete the snapshots first, deliberately, so the move is clean and the decision is recorded. This is the usual answer.
Preserve them elsewhere. Take a backup to PBS before deleting the snapshots. A backup is a better long-term retention mechanism than a snapshot anyway, and it is the only form of history that survives a storage change.

What you must not do is discover this at step nine of a change window with a snapshot somebody was relying on as their rollback.

The 500 GB disk that used 40 GB and arrived using 500 GB

A guest with a 500 GB qcow2 disk on a directory store, actually consuming about 40 GB, was moved onto an LVM-thin pool. The pool had 200 GB free. The move failed at roughly 40%.

The team’s first read was that the pool was smaller than they thought. It was not. The copy was transferring the disk’s logical size, and every zero block being written into the thin pool was allocating an extent, because a thin pool allocates on write and a written zero is a write.

Two things prevent this, and both are worth doing before any bulk migration:

Trim inside the guest first. fstrim -av in a Linux guest, with discard=on on the disk and a controller that passes discards through, releases the unused blocks so the copy has less to move and the source reports its real size.

Make sure the destination disk has discard=on in the guest configuration after the move, or the space will never be released again on the new backend either.

The general form of the mistake is worth naming: thin provisioning is a property of a storage, not of a disk. A disk that is thin on one backend arrives at whatever thinness the destination and the copy method can produce, and the honest planning number for a bulk migration is the sum of provisioned sizes, not the sum of used sizes.

Online, offline, and migration with local disks

Online. For most backends a disk move works with the guest running: PVE mirrors the disk to the destination while the guest keeps writing, then switches over. It is the right default and it is why storage migrations can be done outside a maintenance window.

Offline. With the guest stopped, the copy is a straight transfer. Faster, simpler, and the only option where the backend combination does not support mirroring.

Across nodes at the same time. qm migrate has --with-local-disks for live storage migration and --targetstorage for mapping. That moves a guest to another node and its disks to another storage in one operation — useful when retiring a node and its local storage together, and correspondingly more to go wrong. Do it for a small guest first.

Service impact possiblemove a container volume with the container stopped— Copies the volume to the destination storage. Containers have no live storage mirroring, so this requires the container to be stopped. --delete is 0 by default, so the original is kept as an unused entry.

CTID=205
DEST=shared-rbd

pct status "$CTID"
pct shutdown "$CTID" --timeout 120

pct move-volume "$CTID" rootfs "$DEST" --bwlimit 100000

pct start "$CTID"
pct config "$CTID" | grep -E '^rootfs|^unused'

Under the hood

What is a disk move, mechanically?

Three steps, and knowing them explains every failure mode in this lesson.

1. Allocate at the destination. The storage plugin creates a volume of the required size on the target storage. This is where “no space left” appears, and where a thin-to-thick surprise becomes a full destination.

2. Copy. For an offline move this is a straight read-write through the storage layer’s export/import path — the same machinery pvesm export and pvesm import expose, and the same machinery remote migration uses. For an online move, QEMU performs a drive mirror: it copies the existing data while duplicating new writes to both sides, then switches the guest’s I/O to the destination once the mirror has converged.

3. Rewrite the configuration. The guest’s scsi0: line is updated to the new volume ID, and the old volume ID is written back as unusedN — or freed, if --delete was given.

Three things follow.

The copy is at the block level, so the guest is unaware. A guest does not know its disk moved, which is why nothing inside it needs changing.

Sparseness is not preserved by the mechanism itself. Whether a run of zeroes arrives as an allocation or as a hole depends on the source’s ability to report holes and the destination’s ability to skip them. That is why trimming the guest first is the reliable fix rather than a nicety.

--digest protects the configuration, not the data. It refuses the operation if the guest configuration’s SHA1 has changed since you read it, which is how you make a scripted bulk move safe against somebody editing a guest at the same time. Use it in automation; it costs one extra call and removes a whole class of race.

Planning a whole-storage migration

Before

Sum the provisioned sizes, not the used sizes, and check the destination against that number. Then trim the guests so the two converge.
Enumerate snapshots across the guests to be moved, and decide per guest: discard, or back up to PBS first.
Check the destination’s capabilities against what the guests use — snapshots, discard, and whether the destination is shared if the guests are in HA.
Verify a restore of at least one guest from backup, so that the rollback you are relying on is known to work.
Choose a bandwidth limit from what the storage network can spare, and put it in the change record.

During

Move one small, unimportant guest end to end and verify it fully before starting the batch.
Drive the batch from a written list, using --digest in the script.
Leave --delete at its default. The sources are your rollback.
Record the UPID of every move.

After

Verify each guest: it boots, the application answers, and the data is current.
Confirm discard=on on the moved disks where the destination supports it.
Re-check HA membership and backup jobs, which reference guests rather than storages but whose behaviour may change with the storage’s capabilities.
Then run the unused-volume cleanup as a separate reviewed pass, and confirm with pvesm list that the old storage is actually empty before removing its definition.

What happens if a disk move fails part-way?

The guest is fine. This is the good news and it is structural: the source volume is untouched until the switchover, and the configuration is only rewritten on success. A failed move leaves the guest running on its original disk.

The destination is not fine. A partial volume exists on the target storage, allocated and consuming space, and it may or may not be referenced by anything. On a thin backend that can be a very large amount of space.

What to do, in order:

Confirm the guest is healthy and running from the source. qm config and the running state, not an assumption.
Read the task log for the actual error. “No space left” on the destination, an unsupported feature combination, a storage that went inactive mid-copy, and a mirror that failed to converge on a very busy disk are four different problems.
Look for the partial volume. pvesm list <dest-storage> --vmid <vmid> will show volumes belonging to that guest on the destination. Compare against what the guest configuration references; anything on the destination that the configuration does not name is a leftover.
Free it deliberately with pvesm free <volid>, after confirming it is not the live disk. This is a delete with no undo — check twice.
Fix the cause and retry, and if the cause was capacity, re-do the provisioned-size arithmetic rather than retrying into the same wall.

The one to be careful about is a mirror that will not converge because the guest writes faster than the copy proceeds. Raising --bwlimit helps; doing the move offline is the reliable answer. Repeatedly retrying an online move on a write-heavy guest wastes hours and leaves a partial volume each time.

Key takeaways

qm disk move <vmid> <disk> [<storage>] for VMs — qm move-disk is an alias — and pct move-volume <vmid> <volume> [<storage>] for containers, where the volume is a config key such as rootfs or mp0.
--delete defaults to 0, keeping the source as an unusedN entry. That is the right default and it means a migration campaign frees no space until a deliberate cleanup pass.
Remove unused volumes with qm unlink --idlist unusedN --force 1 only after verifying the guest on its new storage. There is no undo.
Format conversion is implicit: anything moving to a block-level backend becomes raw. --format only applies where the destination is file-level.
Snapshots do not travel. Discard them deliberately, or take a PBS backup first — a backup is the only history that survives a storage change.
Thin provisioning is a property of the storage, not the disk. Plan capacity from provisioned sizes, trim guests before moving, and confirm discard=on afterwards.
Online moves use a QEMU drive mirror and need to converge; a write-heavy guest may need an offline move rather than a bigger bandwidth limit.
Use --digest in scripted moves so a concurrent configuration edit aborts the operation instead of racing it.

Knowledge check

Why this matters

The reassuring thing about this lesson is what it means for the previous one: a storage backend decision is not irreversible. Sixty guests can be moved between backends over three evenings, online, without a maintenance window, and the guests never know.

The unreassuring thing is that the ease hides three costs that all land later. The old storage is still full, because the safe default kept your rollback. The thin disk is not thin any more, because thinness belonged to the storage you left. And the snapshot history is gone, because it belonged there too.

None of those are defects. Each is the correct behaviour of a system that copies data between backends with different properties. But each one is discovered rather than planned by most teams, and the discovery usually happens at the point where the space, the rollback or the history was needed.

Plan from provisioned sizes, decide about snapshots in advance, and make the cleanup a step in the plan rather than an assumption about the tool. Then a storage migration is what it appears to be: routine.