Rehearsing a migration and keeping a way back · Proxmox VE

The previous lesson made importing a virtual machine easy. This one is about the part that stayed hard, which is everything either side of the copy.

Two questions decide whether a migration goes well, and neither is technical. How do we know it worked? and until when can we change our mind?

Rehearse in isolation, or do not rehearse

A rehearsal is a real migration of a real guest, run early, to find out what breaks. Its entire value depends on the copy not being able to talk to anything the original talks to.

The rehearsal guest is a duplicate identity

The imported copy has the source’s hostname, its static addresses, its SSH host keys, its machine ID, its directory computer account, its licence activation, its application cluster membership and its database replication configuration.

Start it on a network that reaches production and every one of those becomes a conflict:

IP conflict, which affects the original as much as the copy, so the outage lands on the system you have not migrated yet.
A directory computer account used by two machines, which in an Active Directory environment breaks the trust for both.
An application cluster that accepts the copy as a member and starts replicating to or from it.
A backup agent that reports in as the same host and overwrites the original’s backup chain.
A monitoring agent that flaps the original’s alerts.
A licence server that sees a second activation and deactivates one.

None of these are hypothetical and several of them damage the source rather than the copy, which is the part that makes this worse than it sounds. The rehearsal’s job is to be safe to get wrong.

Building the isolated network

The mechanism is a network with no uplink. An SDN simple zone is the cleanest way: it creates isolated VNet bridges that are not attached to a physical interface, which is precisely what you want.

Configuration changecreate an isolated rehearsal network— Changes cluster networking configuration. A simple zone creates VNet bridges with no physical uplink, so guests attached to them can reach each other and nothing else. The apply is cluster-wide, so batch your SDN changes and apply once.

pvesh create /cluster/sdn/zones --zone rehearse --type simple

pvesh create /cluster/sdn/vnets --vnet rehearsenet --zone rehearse

pvesh set /cluster/sdn

pvesh get /cluster/sdn/vnets --output-format json | jq -r '.[] | [.vnet, .zone] | @tsv'

Configuration changeattach the rehearsal guest, and belt-and-braces it— Modifies the imported guest. Moves its NIC onto the isolated VNet and sets link_down=1 so the interface is administratively down at first boot. Bring the link up deliberately once you have confirmed from the console which network the guest thinks it is on.

VMID=310

qm config "$VMID" | grep '^net'

qm set "$VMID" --net0 virtio,bridge=rehearsenet,link_down=1

qm start "$VMID"
qm terminal "$VMID"

Success criteria that can fail

“It booted and looks fine” is not a criterion. A criterion is something a rehearsal can fail, written down before it runs, by the people who will be asked whether the migration worked.

A criteria template, per workload

Criterion	Example, made concrete
Boots unattended	Cold boot to login prompt in under 3 minutes, twice, with no console intervention
All services start	The named list of units or services is active after boot, checked automatically
Application answers	A specific transaction succeeds — a login, an API call, a report — from a client, not from the hypervisor
Data is current	A record written on the source before the copy is present on the target
Performance is acceptable	A named operation completes within N% of its measured pre-migration time. Measure the source before migrating, or this criterion is unusable
Licensing survives	Activation state after the virtual hardware change is confirmed, not assumed
Integrations work	Directory join, certificate trust, SMTP relay, log shipping, monitoring check-in — each verified individually
Backups work	A backup job exists at the destination, has run once, and a restore of it has been tested
Rollback works	The documented rollback has been executed once during the rehearsal, end to end

The last row is the one that gets cut for time and is worth more than most of the others. A rollback nobody has performed is a paragraph, not a procedure.

The performance criterion nobody could evaluate

A finance application was migrated over a weekend. On Monday the users said it was slower. The team had no measurement from before the migration, so the conversation had nowhere to go: the users were certain, the engineers could not reproduce it, and nobody could say by how much.

Three weeks were spent tuning storage, CPU type and disk cache. At the end of it the application was measurably faster than the Proxmox baseline taken in week one — and nobody could say whether it had ever been slower than VMware, because that number was never taken.

Measure the source before you migrate it. Not a benchmark suite — the same three operations the users care about, timed, three times, written down. It takes twenty minutes per application and it converts an unfalsifiable complaint into a number.

What a rehearsal proves, and what it cannot

It proves: the guest boots on the new hypervisor; drivers, firmware and machine type are right; the application starts and functions; licensing survives the hardware change; your import procedure is correct and how long it takes.

It cannot prove: that the data will be current at cutover, because the rehearsal copy is already stale; that the cutover will go well under time pressure with an audience; that DNS, routing and firewall changes will work, because those are exactly what the isolation prevented; that integrations against production systems work, for the same reason; or that performance under real concurrent load is acceptable.

Close the gap with a staged cutover, not a longer rehearsal

The things a rehearsal cannot prove are mostly things that can only be tested by touching production. Two techniques get most of the value without pretending otherwise.

A controlled connectivity test. After the isolated rehearsal passes, attach the copy to a real network with a different address and hostname, with a firewall rule allowing only the specific integrations you need to test. You give up “identical to production” and gain the ability to test the directory join, the certificate trust and the log shipping. Change the identity, not the isolation.

A pilot wave. Migrate the least critical real workloads first, for real, with their real identities. That is the only thing that tests the cutover procedure under real conditions, and it is why migration plans should be ordered by consequence rather than by convenience.

The point of no return

Every migration has an hour after which rollback stops being free. Naming it, per workload, is the single most useful thing in the plan.

Before that moment, rollback means: power the source back on, and everyone carries on. After it, rollback means losing whatever was written since.

Under the hood

What actually creates the point of no return?

Data divergence. The moment the first user transaction is written on the migrated system, the source and the target hold different data. Rolling back after that discards those writes, so rollback stops being an operational decision and becomes a data-loss decision that somebody has to own.

This is worth being precise about, because it is not the same moment as the cutover. The point of no return is when the first write lands, which for a system that users reach at 08:00 is 08:00, regardless of when the migration finished at 03:00. Everything between 03:00 and 08:00 is still free rollback, and that window is exactly what verification should be spending.

Irreversible external state. A DNS record with a long TTL that caches elsewhere. A directory computer account reset. A licence reactivated against new hardware, where the vendor permits a limited number. A certificate reissued for a new key. Each of these adds a step to the rollback that is somebody else’s system, and therefore somebody else’s timeline.

Deletion of the source. Obvious, and it should be a separate, scheduled, reviewed decision — not something that happens because a VMware licence expired or someone was reclaiming space.

The practical upshot: the point of no return is a time, computable in advance, and it belongs in the change record next to the window. “Rollback is free until 08:00 Monday; after that a rollback loses the day’s transactions and requires sign-off from the service owner.”

The rollback window

After the point of no return, rollback is expensive but should still be possible for a defined period. That period is the rollback window, and it has a cost that has to be budgeted rather than assumed.

What the window costs	Why
Source capacity	The ESXi hosts and their storage cannot be reclaimed
Source licensing	The hypervisor licence and support contract keep running
Operational attention	The source estate still needs patching, monitoring and backups
Cognitive load	Two places to look during any incident

A rollback window with no end date never ends

The failure mode is not closing the window too early. It is never closing it.

Six months after a migration, an estate is still paying for VMware, still patching hosts nobody uses, and still has a set of powered-off VMs that are now so stale that rolling back to them would be worse than a restore from backup. The window has expired in every sense except the one on the invoice.

Write the closing date into the plan, with a named owner and a defined closing procedure:

Confirm the migrated workload has survived a full business cycle, a backup cycle and a patch cycle.
Confirm backups at the destination have been restored successfully at least once.
Take a final archival backup of the source, retained per the data policy — this is what replaces the rollback.
Record the decision, then decommission.

Step 3 is what makes closing the window safe: you are not giving up the ability to recover, you are moving it from “power the old VM on” to “restore from an archive”, which is slower and enormously cheaper to hold.

Read-only / Safeprove the rollback path before you need it— Read-only. During the rehearsal, verify that the migrated guest has a backup that has actually run and that a restore has been tested. A backup that exists is not a rollback; a restore you have performed is.

VMID=310
STORE=pbs-primary

pvesm list "$STORE" --vmid "$VMID"

pvesh get /cluster/backup --output-format json \
| jq -r '.[] | [.id, (.vmid // "all"), .schedule, .storage] | @tsv'

What happens if you have to roll back after the point of no return?

Work the sequence rather than improvising it, because the expensive mistakes all happen in the first ten minutes.

1. Stop writes on the migrated system first. Not “tell people to stop” — stop it: take the service down, or block access at the load balancer or firewall. Every minute of continued use enlarges the data you are about to strand.

2. Capture the divergent data before you touch anything. Whatever was written since cutover exists only on the new system. Back it up, export it, or snapshot it — you will need it to reconcile, and you will not get a second chance once the source is running and users reconnect.

3. Decide explicitly what happens to that data. Three honest options: replay it into the source, reconcile it manually later, or accept its loss. Someone with authority over the service picks; this is not an engineering decision.

4. Reverse the external state in dependency order. DNS back, with the TTL you should have lowered before cutover. Firewall rules back. Directory and certificate state back. This is usually the slowest part and it involves other teams.

5. Power the source back on, and verify it as thoroughly as you verified the target. It has been off for days or weeks: its certificates may have expired, its agents will check in stale, its clock will be wrong, and its patches are behind.

6. Write down why. A rollback is the most informative event in a migration programme, and the reason is nearly always something the rehearsal criteria would have caught if they had included it. Add it to the criteria before the next wave.

The step people skip is 2, and it is the one that turns a recoverable rollback into a data-loss incident.

Key takeaways

A rehearsal guest is a duplicate identity — hostname, addresses, host keys, directory account, licence, cluster membership. Started on a production network it damages the original as often as the copy.
Rehearse on an isolated network. An SDN simple zone creates VNet bridges with no physical uplink, which is exactly the required property. Boot with link_down=1 and bring the link up deliberately.
Write success criteria that can fail, before the rehearsal, with the people who will judge the migration. Include an executed rollback as a criterion.
Measure the source before migrating it. Without a pre-migration number, a performance complaint is unfalsifiable and unwinnable.
A rehearsal proves boot, drivers, firmware, application function, licensing and procedure timing. It cannot prove data currency, integrations, DNS and routing, or behaviour under real load — those need a controlled connectivity test with a changed identity, and a pilot wave.
The point of no return is the first user write on the migrated system, not the cutover. It is a time you can compute in advance and it belongs in the change record.
The rollback window costs capacity, licensing, patching and attention. Give it a closing date, a named owner and a closing procedure whose third step is a final archival backup — that is what makes closing it safe.
If you must roll back after the point of no return, stop writes and capture the divergent data first. That step is the difference between a rollback and a data-loss incident.

Knowledge check

Why this matters

Migrations are judged on the cutover weekend and decided months earlier, in two documents nobody enjoys writing: the success criteria and the rollback plan.

The reason they matter more than the technical work is that the technical work has a right answer and these do not. Whether a guest boots is a fact. Whether the migration succeeded is a judgement, and if nobody wrote down what success meant, that judgement gets made afterwards by whoever is unhappiest — usually about performance, usually without numbers, usually in a meeting where the engineers have no evidence because nobody thought to take a baseline.

The same applies to the way back. “We can always roll back” is true for a few hours and then quietly stops being true, and the estates that get hurt are the ones where nobody noticed the transition. Naming the hour turns it from a comforting belief into a fact people can plan around.

Write both documents while the migration is still an idea. They take an afternoon and they are the difference between a project that ends and one that lingers.