Cross-cluster guest migration through PDM · Proxmox VE

Of everything PDM does, this is the feature people repeat to each other. Move a running virtual machine from a cluster in one building to a cluster in another, from a dialog box, without downtime.

It is real. It has been in the product since the first alpha. And it is surrounded by more sharp edges per square metre than anything else in this part, starting with a two-word label in the Proxmox VE command reference that almost nobody who uses the feature has read.

What it is, underneath

PDM’s Migrate action moves a guest to another node, within the same remote (cluster) or across remotes. It is not offered for templates.

Across remotes, PDM is driving the Proxmox VE remote-migration API — the same mechanism reachable on a cluster as qm remote-migrate for a virtual machine and pct remote-migrate for a container. The PDM CLI exposes it directly:

Service impact possiblecross-remote migration from the PDM CLI— Moves VM 141 from site-a to site-b. Both mappings are mandatory. Read the whole lesson before running this against anything you care about.

proxmox-datacenter-manager-client pve qemu remote-migrate site-a 141 site-b \
--map-bridge vmbr0:vmbr0 \
--map-storage local-zfs:ceph-vm \
--target-vmid 3141 \
--online true \
--bwlimit 200000

The container form is pve lxc remote-migrate with the same shape. For a move inside one cluster, the command is pve qemu migrate <remote> <vmid> <target>, which is the ordinary intra-cluster migration you already know from Part IX and carries none of the hazards below.

Proxmox VE labels this EXPERIMENTAL

The qm command reference describes qm remote-migrate as “Migrate virtual machine to a remote cluster. Creates a new migration task. EXPERIMENTAL feature!” The pct reference says the same for containers.

That label is not decoration and it has consequences you should plan around:

Behaviour may change between releases. A runbook written against 9.2 needs re-testing against 9.3, not just re-reading.
The prose documentation does not exist. The Proxmox VE administration guide has a migration chapter, and it covers migration within a cluster. Remote migration appears only in the qm and pct command references. There is no chapter enumerating its prerequisites, its supported configurations or its limitations. This lesson is built from the command reference plus the PDM documentation, and that is all there is to build from.
Anything you cannot find documented, you must test. Not ask about, not infer from a forum post from 2024 — test, on guests that do not matter, on the versions you actually run.

None of this means do not use it. It means use it with a rehearsal, a verification step, and a rollback that does not depend on the feature working.

Prerequisite one: map every bridge

The two clusters are independent. They do not share an SDN configuration, a bridge namespace or a VLAN scheme, and nothing guarantees that vmbr1 on site-A means what vmbr1 means on site-B.

So the mapping is mandatory. In the PDM CLI it is --map-bridge FROM:TO, which may be given more than once — once per source bridge. The PVE-side equivalent, --target-bridge, offers two shorthands worth knowing because they explain the semantics: providing a single bridge ID maps all source bridges to that one bridge, and providing the special value 1 maps each source bridge to itself.

That last shorthand is where estates get hurt. --target-bridge 1 reads as “keep the networking the same” and means “assume the target cluster has a bridge of the same name, and that it is attached to the same layer-2 domain”. The first half of that assumption fails loudly — the migration errors. The second half fails silently: the guest arrives, attaches to a bridge that has the right name and the wrong VLAN, and is unreachable on an IP it still believes it owns.

Prerequisite two: map every storage

Same reasoning, larger consequences. --map-storage FROM:TO, once per source storage. The PVE-side --target-storage has the same two shorthands: a single storage ID maps everything to it, and 1 maps each source storage to itself.

The storage map is where the physics of the operation shows up. Within a cluster with shared storage, migrating a running VM moves memory and leaves the disks where they are — a few gigabytes of RAM over a fast local link. Across clusters there is no shared storage by definition, so every disk is copied over the network.

That changes the arithmetic completely:

	Intra-cluster, shared storage	Cross-cluster
What moves	Memory, plus dirty-page convergence	Every disk, then memory
Dominant cost	RAM size	Disk size divided by link bandwidth
Typical link	Cluster network, 10 GbE and up	Whatever runs between the sites
A 2 TB VM	seconds to a minute	hours

A 2 TB guest over a 1 Gbit/s inter-site link is, at perfect efficiency, about four and a half hours of copying. Real links are not perfectly efficient and are not idle. Plan cross-cluster migrations in hours, size the maintenance window from the disk, and use --bwlimit (in KiB/s) to keep the copy from consuming the link that everything else at both sites depends on.

Prerequisite three: the VMID

Two independent clusters have two independent VMID spaces, and 141 is a popular number. PDM 1.1 added a helpful behaviour here: when migrating a guest to a new cluster, PDM queries the remote cluster for its next free VMID and uses it to pre-fill the migration dialog.

Accept that gratefully and then think about it for ten seconds, because a changed VMID has consequences that outlive the migration:

Backup history does not follow the guest. A PBS backup group is keyed by guest type and ID. VM 141 arriving at site-B as VM 3141 starts a fresh backup chain; the old chain remains under 141 and is now orphaned from the running workload. Your retention policy is quietly protecting a guest that no longer exists while the live one has one backup.
Backup jobs do not follow either. A PVE backup job is cluster configuration, not guest configuration. The job on site-A still names 141; site-B has no job for 3141 until you create one. The window between migration and noticing is a window with no backups.
Monitoring, CMDB and runbooks refer to the old ID. Anything keyed on vmid needs updating, and nothing will tell you which.

The --target-vmid option exists so you can keep the number when it is free at the destination. Prefer that where you can; where you cannot, treat the ID change as a small migration of its own with its own checklist.

The default that surprises people

Read this option description from the qm reference slowly:

--delete <boolean> (default = 0) — Delete the original VM and related data after successful migration. By default the original VM is kept on the source cluster in a stopped state.

A successful cross-cluster migration, run with defaults, leaves you with two copies of the guest: one running on the target, one stopped on the source, with the same hostname, the same configuration, the same static IP addresses, the same service identity and, if it is a cluster member of something, the same node name.

The stopped copy is a loaded weapon

This default is the right one. It is a safety net: if the target turns out to be wrong, the original is still there and still intact, and rollback is starting a VM rather than restoring a backup.

The hazard is what happens to that net afterwards. Nothing cleans it up, nothing labels it, and nothing prevents anyone from starting it. Six weeks later, a stopped VM on site-A gets started by an operator tidying up, or by an HA rule somebody added, or by a bulk power-on after a maintenance window — and now two machines with the same identity and the same IP are running in an estate whose sites are routed to each other.

What that costs depends on the workload, from an IP conflict at the trivial end to a split-brain database at the other. A domain controller is the worst case: starting a stale, previously-migrated copy of one is the classic route to a USN rollback that no amount of subsequent replication will repair.

The handling is procedural because there is no technical control:

Verify the target properly — the guest boots, the application answers, the data is current — before touching the source.
Rename the source guest immediately, to something like MIGRATED-20260812-do-not-start-141. The name is the only thing a future operator will see.
Delete it on a defined schedule, once the target has survived a backup cycle and a reboot. Put the date in the change record.
Do not use --delete true on the first run of a cluster pair you have not migrated between before. The net is worth more than the tidiness.

Online, offline, and what the dialog offers

--online performs a live migration if the guest is running; it is ignored if the guest is stopped. For containers there is also --restart, which uses restart migration.

One honest note about the interface. The roadmap lists, under guest and resource management, “more options in the migration dialog, such as a bandwidth limit and an online/offline preference”. That wording indicates the dialog does not yet offer those two controls, even though the API and the PDM CLI both do.

So: the capability exists, and if you need to set a bandwidth limit or force an offline migration on PDM 1.1.7, reach for the CLI. Verify against your own version before writing it into a runbook — this is precisely the kind of gap that closes in a point release.

Under the hood

How does a migration between two clusters that have never heard of each other authenticate?

The qm remote-migrate target endpoint parameter is the answer, and its shape is instructive:

apitoken=PVEAPIToken=user@realm!token=SECRET, host=ADDRESS [,fingerprint=FINGERPRINT] [,port=PORT]

An API token, a host, and optionally a certificate fingerprint. That is the same three ingredients PDM uses to define a remote, and for the same reason: there is no cluster membership to lean on, no corosync, no shared /etc/pve. The source cluster is simply an API client of the target cluster, authenticating with a bearer credential and pinning a fingerprint if the certificate is not publicly trusted.

Two things follow. First, the trust for a migration is per-migration and point-to-point: two clusters do not become related by migrating between them, which is exactly right. Second, when PDM performs the migration, it is supplying that endpoint — so PDM is handing one cluster a credential for another cluster. Lesson 5 is about what that means.

There is also a lesson in the fingerprint parameter being optional. A remote migration issued by hand, to a cluster with the default self-signed certificate, with no fingerprint supplied, is a disk copy across a WAN with no verification of who is on the other end. PDM at least forces the trust decision at add time.

What this is not a substitute for

Cross-cluster migration is a planned, operator-initiated, both-ends-healthy operation. Four things it is repeatedly mistaken for:

It is not high availability. HA restarts a guest automatically when a node fails, inside one cluster, driven by quorum. Nothing about remote migration is automatic and nothing about it survives the source cluster being down. Part XII is unchanged.

It is not disaster recovery. A migration requires the source cluster to be up, reachable and healthy enough to read every disk. A site that has burned down cannot be migrated from. DR is Part XIV: backups at a second site, replication, a tested restore, and an RPO and RTO somebody has signed. The roadmap acknowledges the gap directly, listing “off-site replication copies of virtual guests for manual recovery on a datacenter failure” as future work, with the parenthetical “(not HA)” attached.

It is not a load balancer. There is no scheduler, no placement policy and no automation. Someone decides, and someone clicks.

It is not a substitute for capacity planning. Being able to move a workload does not create anywhere to move it to. If site-B has no headroom, the migration either fails or succeeds and makes site-B the new problem. Part II and Part XVII still own that question.

Cross-cluster migration checklist

Run this the first time for any cluster pair, and every time for anything that matters.

Before

A current, verified restorable backup of the guest. Not a recent backup — a restore you have tested. This is the rollback that does not depend on the experimental feature working.
A written bridge map for this cluster pair, checked against what the target bridge is actually attached to, not against its name.
A written storage map, and confirmed free space at the destination for every disk.
The target VMID decided: kept, or changed with the consequences listed.
A bandwidth limit chosen from what the inter-site link can spare.
The maintenance window sized from total disk size over that limit, with margin.
Confirmation that the destination cluster has the CPU model, the storage features and the guest-relevant configuration the guest needs.

After

The guest boots and the application answers — from a client, not from the hypervisor.
Data is current: the newest record you wrote before the move is present.
Backups at the destination: a job exists for the new VMID and has run once successfully.
Monitoring, CMDB and DNS updated for the new location and, if it changed, the new ID.
The source guest renamed to something no operator would start by accident, with a deletion date recorded.

What happens if a cross-cluster migration fails halfway?

Enumerate before you fix. A failed migration leaves state at both ends and the first job is to find out which.

At the source, the guest should still exist and, if it was running and the migration was online, may be either still running or stopped depending on where in the sequence it failed. Treat its running state as unknown until you look.

At the destination, there may be a partially created guest and partially copied disks. Those disks are consuming space at the destination whether or not the guest is usable. On a thin-provisioned or ZFS-backed target this can be a large amount of space that nothing will reclaim for you.

The recovery sequence:

Do not immediately re-run the migration. A second attempt against a half-created destination guest is how you get two partial copies and a VMID conflict.
Determine the source guest state and stabilise it. If the workload needs to be running, get it running at the source. That is your working copy.
Read the task log on the destination cluster, not the summary in PDM. The failure reason — out of space, storage feature unsupported, bridge missing, connection dropped — determines everything else.
Clean the destination by hand. Remove the partial guest and confirm the disks are actually gone from the storage, because a removed guest configuration does not always take orphaned volumes with it.
Fix the cause, then rehearse. If the cause was a mapping or a capacity error, correct the map and try again with a guest that does not matter first.

And the reason step 1 in the pre-migration checklist is a verified backup: if the failure damaged the source in some way you did not anticipate, the experimental feature is not the thing you want standing between you and a working workload.

Key takeaways

PDM’s cross-remote migration drives PVE’s qm remote-migrate and pct remote-migrate, which the PVE command reference labels EXPERIMENTAL and which has no prose documentation chapter.
Bridge and storage mappings are mandatory and are not validated for meaning. A bridge with the right name on the wrong network is the silent failure.
Every disk crosses the network. Size the window from disk over bandwidth, not from RAM, and set --bwlimit so the copy does not become the incident.
The default leaves the source guest on the origin cluster in a stopped state. Rename it immediately and delete it on a schedule.
PDM pre-fills the next free VMID at the destination. A changed VMID orphans the PBS backup chain and leaves the guest outside every existing backup job.
Bandwidth limit and online/offline preference are CLI options today; the roadmap lists them as future additions to the migration dialog.
Migration is not HA, not DR, not a load balancer, and not capacity planning.

Knowledge check

Why this matters

The demo is genuinely impressive, and that is the problem. A running virtual machine crossing between two independent clusters from a dialog box looks like the boundary between those clusters has stopped mattering.

It has not. Every reason the second cluster exists — the geography, the failure domain, the upgrade unit, the tenancy line — is still there after the migration, and the migration crossed it by copying two terabytes over a link that other things were using, using an experimental code path, and leaving a duplicate behind.

The right way to hold this feature is as a planned change, in the same category as a storage migration or a major version upgrade: it gets a change record, a window, a rehearsal, a verification and a rollback. Held that way it is excellent, and it removes a class of work that used to mean a backup, a restore, and an outage measured in hours.

Held as a convenience — a thing you do because the button is there — it is the most expensive button in PDM.