Cross-cluster migration from the command line · Proxmox VE

Part XXV covers cross-cluster migration as a Proxmox Datacenter Manager feature: the dialog box, the bandwidth arithmetic, and the duplicate guest the defaults leave behind. This lesson is the layer underneath, which you need for three reasons.

You may not run PDM. You will certainly want this in a script, because a split is a per-guest movement schedule and nobody clicks through two hundred of them. And when a migration fails, the error comes from this layer, not from the one that called it.

The endpoint string, field by field

Everything distinctive about remote migration is in one parameter. Both commands take a <target-endpoint> in this format:

apitoken=PVEAPIToken=user@realm!tokenid=SECRET,host=ADDRESS[,fingerprint=FINGERPRINT][,port=PORT]

Four fields, and each one is a decision:

apitoken — the full PVEAPIToken= header value for a token on the target cluster. Not a password, not an SSH key. The source cluster authenticates to the target as an ordinary API client.

host — an address of any node in the target cluster. There is no cluster VIP concept here; you name a node. If that node is down, the migration does not fail over to a sibling, so pick a node that is healthy at the time and consider which one your script chooses.

fingerprint — optional, and the field most worth caring about. It pins the target’s TLS certificate. Omit it against a cluster using the default self-signed certificate and you are copying disks across a network with no verification of who is receiving them.

port — optional, defaults to the API port, 8006. Present because estates publish the API behind a different port often enough to matter.

Read-only / Safeget the target's certificate fingerprint, on the target— Read-only. Run this on the target node named in host=, and transfer the value out of band. Reading the fingerprint from the source over the same untrusted path you are trying to authenticate defeats the purpose.

openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256

Read-only / Safe

$ openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256

sha256 Fingerprint=AA:BB:CC:DD:EE:FF:00:11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33:44:55:66:77:88:99

The receiving token, at least privilege

The token lives on the target cluster and should not be an administrator. Build it from the privileges the operation actually needs.

Configuration changecreate a receiving identity on the TARGET cluster— Changes access control on the target cluster. Creates a dedicated user and a privilege-separated token, then grants a purpose-built role. The token secret is printed once and never again.

TARGET_USER="migrate-in@pve"
TOKEN_ID="fromsitea"

pveum user add "$TARGET_USER" --comment "Receives remote migrations from site-a"

pveum role add MigrateReceive --privs "VM.Allocate VM.Config.Disk VM.Config.Network VM.Config.Options VM.Config.HWType VM.Config.CPU VM.Config.Memory VM.Config.Cloudinit VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.Audit Sys.Incoming SDN.Use"

pveum user token add "$TARGET_USER" "$TOKEN_ID" --privsep 1

Configuration changescope the token, on the TARGET cluster— Changes access control. A privilege-separated token starts with no permissions of its own; effective permissions are the intersection of the user's and the token's. Grant on the narrowest paths that work — the VM tree, the destination storage, and the node that terminates the stream.

TARGET_USER="migrate-in@pve"
TOKEN_ID="fromsitea"
TOKEN="${TARGET_USER}!${TOKEN_ID}"
DEST_STORAGE="ceph-vm"
DEST_NODE="pve-b1"

pveum acl modify /vms --users "$TARGET_USER" --roles MigrateReceive
pveum acl modify /vms --tokens "$TOKEN" --roles MigrateReceive

pveum acl modify "/storage/$DEST_STORAGE" --users "$TARGET_USER" --roles MigrateReceive
pveum acl modify "/storage/$DEST_STORAGE" --tokens "$TOKEN" --roles MigrateReceive

pveum acl modify "/nodes/$DEST_NODE" --users "$TARGET_USER" --roles MigrateReceive
pveum acl modify "/nodes/$DEST_NODE" --tokens "$TOKEN" --roles MigrateReceive

Sys.Incoming is the one people miss

Sys.Incoming is documented as “allow incoming data streams from other clusters”, and it exists precisely because receiving a guest from a cluster you do not control is a different act from creating a guest locally.

It is a node/system privilege, so it is granted on / or on /nodes/<node>, not on /vms. A token that has VM.Allocate and Datastore.AllocateSpace but no Sys.Incoming will authenticate happily, pass early checks, and fail the migration with a permission error at the point the stream opens — late enough that a large disk copy may already be under way.

The privilege is also a useful mental model for the whole feature: PVE treats an inbound cross-cluster stream as a distinct capability that an administrator must deliberately hand out, rather than as a side effect of being able to make VMs.

Running it

Service impact possiblemigrate VM 141 from this cluster to a remote cluster— Copies every disk of VM 141 across the network to the target cluster and, with --online, live-migrates its memory. The source guest is left in a stopped state. Read the whole lesson, and Part XXV, before running this against anything that matters.

VMID=141
TARGET_VMID=141
TARGET_HOST="192.0.2.60"
TOKEN_SECRET="REPLACE_ME"
FPR="AA:BB:CC:DD:EE:FF:00:11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33:44:55:66:77:88:99"

ENDPOINT="apitoken=PVEAPIToken=migrate-in@pve!fromsitea=${TOKEN_SECRET},host=${TARGET_HOST},fingerprint=${FPR}"

qm remote-migrate "$VMID" "$TARGET_VMID" "$ENDPOINT" \
--target-bridge vmbr0:vmbr2 \
--target-storage local-zfs:ceph-vm \
--online \
--bwlimit 200000

The container form is identical in shape:

Service impact possiblethe container equivalent— Containers have no live migration. --restart shuts the container down, moves it, and starts it at the destination, with --timeout controlling how long the shutdown is given before it is forced.

CTID=205
TARGET_HOST="192.0.2.60"
TOKEN_SECRET="REPLACE_ME"

ENDPOINT="apitoken=PVEAPIToken=migrate-in@pve!fromsitea=${TOKEN_SECRET},host=${TARGET_HOST}"

pct remote-migrate "$CTID" "$CTID" "$ENDPOINT" \
--target-bridge vmbr0:vmbr2 \
--target-storage local-zfs:ceph-ct \
--restart \
--timeout 300

The two mappings are mandatory and their shorthands are covered in Part XXV: a single ID maps everything to it, and the special value 1 maps each source name to itself. The trap is also covered there and is worth repeating in one line: --target-bridge 1 succeeds whenever a same-named bridge exists at the destination, regardless of what network that bridge is attached to.

What does not travel

The guest arrives. A surprising amount of what surrounded it does not, because those things are cluster configuration rather than guest configuration.

Does not travel	Where it lived	What to do
Backup jobs	Cluster config, `/etc/pve/jobs.cfg`	Create a job at the destination and confirm it has run once
PBS backup history	Keyed by guest type and ID on the PBS datastore	Decide whether the old chain is retained, and for how long
Firewall rules at cluster level and security groups	Cluster config	Recreate; guest-level rules in the guest config do travel
HA resource membership, groups and rules	Cluster config	Re-add at the destination if the guest needs HA
Pool membership	Cluster config	Re-add, and re-check the ACLs that depended on it
Replication jobs (`pvesr`)	Cluster config	Recreate against a destination peer, or drop
ACLs granting users access to this VMID	Cluster config	Recreate on the target cluster
Notification matchers referencing the guest	Cluster config	Recreate
Monitoring targets, CMDB entries, DNS	Outside PVE entirely	Update; nothing will tell you
Storage-level snapshots on the source volumes	On the source storage	See below

That last row deserves its own paragraph. Guest snapshots are a storage feature and the destination storage may not support them in the same way — a guest living on ZFS with a snapshot tree, moved onto an LVM-thin or a directory-backed store, will not arrive with an equivalent tree. Treat a snapshot history as something you deliberately abandon or deliberately preserve elsewhere, and decide before the move rather than discovering it after.

Under the hood

What is actually on the wire?

Remote migration is built on the same volume export/import machinery that pvesm export and pvesm import expose on the command line. The source serialises each volume into a stream; the target’s storage plugin materialises it into a new volume on the mapped storage. The pvesm import options tell you what that layer supports — --with-snapshots, --base, --snapshot, --allow-rename, --delete-snapshot — and they are the vocabulary the migration code is speaking.

Two consequences fall out of that design.

First, format conversion happens implicitly. A qcow2 file on a directory store arriving on an RBD or LVM-thin destination becomes a raw block volume, because that is the only thing those backends hold. The guest does not notice; your snapshot strategy does, because the snapshot mechanism has changed from a file format feature to a storage feature.

Second, the transport is the API, on port 8006, not SSH on 22. Intra-cluster migration has an SSH-tunnelled path and a migration_type of secure or insecure in datacenter.cfg; those settings are about traffic inside one cluster. For a remote migration the firewall rule you need between sites is the API port to the node named in host=, and the credential is the token, not a root SSH key. That is a meaningfully better security posture than the alternatives people build by hand, and it is worth knowing so you do not open port 22 between two estates for no reason.

Remote migration or backup and restore?

Both move a guest between clusters. They are not interchangeable.

	`qm remote-migrate`	Backup, copy, restore
Downtime	Near-zero with `--online`, after a long copy	Full: restore duration plus start
Total elapsed time	One disk copy	Backup, transfer, restore — often longer
Source left as	Stopped copy of the guest	Untouched and running
Rollback	Start the stopped source	Do nothing; the source never stopped
Depends on	An experimental code path, both clusters healthy and reachable	Backups you already take
Works when the source is dead	No	Yes, if the backup is off-site
Preserves VMID	With `--target-vmid`, if free	Yes, if free
Scriptable in bulk	Yes	Yes

The honest guidance: use remote migration when downtime is the binding constraint and both clusters are healthy, and use backup/restore when correctness and reversibility matter more than minutes. For a planned estate split done over several weekends, backup/restore is frequently the better tool, and it has the enormous advantage of exercising the restore path you claim to have.

What happens if the link drops at 80% of a 3 TB copy?

You have state at both ends and no resume. Work in this order.

1. Do not re-run the command. A second attempt against a half-created destination guest produces either a VMID conflict or a second set of orphaned volumes.

2. Establish the source guest’s actual state. With --online, where the failure lands in the sequence determines whether the guest is still running. qm status $VMID on the source, and if the workload must be up, start it there. The source is your working copy.

3. Read the task log at the destination. The source’s error is often generic (“connection reset”); the destination’s task log says whether it ran out of space, rejected a storage feature, or lost the stream. pvesh get /nodes/$NODE/tasks --limit 20 on the destination node, then pvesh get /nodes/$NODE/tasks/$UPID/log, gives you the real reason. Part XX covers the UPID model this uses.

4. Clean the destination by hand, and verify. Remove the partial guest, then check the storage directly — a removed guest configuration does not always take orphaned volumes with it:

pvesm list ceph-vm --vmid 141 on the destination should return nothing. If it lists volumes, remove them with pvesm free after confirming what they are. Skipping this step is how a destination cluster silently loses hundreds of gigabytes that no guest owns.

5. Fix the cause before retrying. Out of space at the destination, a storage that does not support the content type, a missing Sys.Incoming, or a link that cannot hold a multi-hour transfer are four different problems. For the last one, set a --bwlimit well under the link’s capacity and consider whether backup/restore over a resumable transfer is simply the better tool for this guest.

6. Confirm nothing partial remains at the source. The source should be unchanged; verify rather than assume, especially the guest’s disk list and its running state.

Scripting a migration campaign

For a split you will run this dozens of times. Five things that separate a campaign that finishes from one that gets abandoned:

Migrate a throwaway guest first, on every cluster pair, and keep it as the smoke test you re-run after either cluster is upgraded.
Drive from a list, not from memory. A file of VMIDs with, per guest, its target VMID, its storage map, its bridge map and its window. Generate it from pvesh get /cluster/resources --type vm --output-format json rather than by hand.
Never loop with --delete true. Leave the stopped sources, rename them in the same loop to something like MIGRATED-do-not-start-141, and delete them as a separate reviewed step once the destination has survived a backup cycle.
Check free space at the destination before each guest, not once at the start. pvesm status --storage ceph-vm is one call and it prevents the failure mode that costs the most time.
Record the UPID of every migration. When something is queried three weeks later, the task log is the only account of what happened, and it is per node.

Key takeaways

The endpoint is apitoken=...,host=...[,fingerprint=...][,port=...]. The fingerprint is optional and pinning it is the difference between an authenticated transfer and a hopeful one. The port defaults to 8006.
The receiving token belongs on the target cluster, privilege-separated, and needs Sys.Incoming — a node-level privilege granted on / or /nodes/<node>, not on /vms. Its absence fails the migration late.
The credential appears in ps and in shell history. Source it from a file or a secret store, and rotate it when the campaign ends.
Bridge and storage maps are mandatory; 1 maps each name to itself and is the silent failure when a same-named bridge is on a different network.
Backup jobs, PBS history, cluster firewall rules, HA membership, pool membership, replication jobs, ACLs and monitoring do not travel. Snapshots may not survive a change of storage backend.
The transport is the API on 8006, not SSH on 22, and it reuses the pvesm export/import volume machinery — which is why formats convert implicitly.
A failed migration has no resume. Stabilise the source, read the destination task log, remove orphaned volumes with pvesm list/pvesm free, then retry.
For a planned split, backup and restore is often the better tool: slower, but reversible, non-experimental, and it exercises your restore path.

Knowledge check

Why this matters

The endpoint string is the whole architecture of this feature compressed into one parameter, and it is worth reading it that way.

There is no cluster-to-cluster trust relationship. There is no federation, no shared secret, no membership. There is a token, a host, and — if you bother — a fingerprint. The source cluster is an API client of the target cluster for the duration of one operation, and when the operation ends the two clusters are exactly as unrelated as they were before.

That is the right design. It means migrating between two clusters does not enlarge either one’s fault domain, which was the entire reason for having two clusters in the first place. It also means every security property of the operation is a property of that one token: who holds it, what it can do, where its secret lives, and when it stops working.

Give it the smallest role that works, pin the fingerprint, keep the secret off the command line, and delete it when the campaign ends. That is a five-minute discipline that keeps the boundary you paid for.