Domain controllers as virtual machines · Proxmox VE

Everything the hypervisor is good at — snapshot before a change, roll back if it goes wrong, restore last night’s backup when something is broken — is wrong for a domain controller. Not discouraged. Wrong, in a way that can turn a single-server problem into a forest-wide one, and in a way that Microsoft documents explicitly enough that “we did not know” is not available afterwards.

Active Directory is a multi-master replicated database. Every domain controller is a writable copy that reconciles with the others by exchanging change sequences, and the entire reconciliation protocol depends on one assumption: a change sequence number, once issued by a DC, is never issued again by that DC. A hypervisor rollback breaks exactly that assumption, and it breaks it silently.

This lesson is about what protects you, what does not, and which of those two categories the operation you are about to perform falls into.

USN rollback, in the shape it actually occurs

Microsoft’s own walkthrough of the mechanism is the clearest available, and it is worth following once rather than summarising:

At T1, a snapshot is taken of DC1. DC1 has a highest committed USN of 100 and an invocation ID of A.

At T2, 100 users are created on DC1. Each consumes a USN, so DC1 moves to 200. These replicate to DC2, which records its high watermark as: everything from DC1, in the context of invocation ID A, through USN 200.

At T3, the snapshot from T1 is applied to DC1. Its USN rolls back to 100. It now believes the next update it makes should be USN 101.

Now consider what happens with no protection. DC1 creates a new user and assigns it USN 101, in the context of invocation ID A. DC2 already believes it has everything from DC1 up to USN 200 under invocation ID A, so it does not ask for it. The object exists on DC1 and does not exist on DC2, and neither machine has any reason to think anything is wrong. Multiply that by every change made on DC1 between USN 101 and 200 and you have a permanently divergent directory — a USN bubble — that replication will never repair because replication does not believe there is anything to repair.

That is USN rollback. It does not announce itself, and its symptoms — a user that exists when you authenticate against one DC and not another, a group membership that works in one office — arrive weeks later and look like anything except a restore that was performed in a hurry.

What vmgenid does about it

The protection is a value the hypervisor exposes to the guest. On the Proxmox side it is a first-class VM option, documented tersely:

--vmgenid <UUID> — Set VM Generation ID. Use 1 to autogenerate on create or update, pass 0 to disable explicitly.

Proxmox VE generates one automatically for new VMs and stores it in the VM configuration, and it is designed to change on the events that constitute a time shift for the guest — snapshot rollback, backup restore, and cloning a whole VM.

On the Active Directory side, the behaviour is precise:

AD DS relies on the hypervisor platform to expose an identifier called VM-Generation ID to detect the snapshot restore of a virtual machine. AD DS initially stores the value of this identifier in its database (NTDS.DIT) during domain controller promotion. When an administrator restores the virtual machine from a previous snapshot, the current value of the VM-Generation ID from the virtual machine is compared against the value in the database. If the two values are different, the domain controller resets the Invocation ID and discards the RID pool, thereby preventing USN reuse or the potential creation of duplicate security-principals.

Return to T3 with that in place. DC1 comes up, notices that the generation ID from the hypervisor no longer matches the one in its database, and does two things before committing anything. It resets its invocation ID from A to B, and it discards its RID pool.

The invocation ID reset is the repair. DC1 now issues USNs 101 onward under invocation ID B, about which DC2 knows nothing, so DC2 requests all of it. And because DC2 still holds the changes DC1 made between USN 101 and 200 under invocation ID A, those replicate back into DC1. The directory converges.

Discarding the RID pool is the second repair, addressing a different hazard: a rolled-back DC would otherwise hand out relative identifiers it had already issued, producing two security principals with the same SID.

Check that your DC VMs actually have a vmgenid

A VM created on a modern Proxmox VE gets one automatically. A VM that predates the feature, was built by an older automation path, or was imported from another platform may not — and a DC with no generation ID has none of the protection above.

Read-only / Safeaudit the generation ID on every DC guest

# qm config 120 | grep -E '^vmgenid:' || echo 'NO VMGENID - unprotected'

vmgenid: 6f0f1c5e-8e2a-4c3d-9f1b-2b7a5d4e0c11

Illustrative output

Adding one to an existing DC is qm set VMID --vmgenid 1, and it is safe — but understand what happens at the next boot. The guest compares the value the hypervisor now presents against the null value stored in its database, finds they differ, and triggers the safeguards: invocation ID reset, RID pool discarded, non-authoritative inbound replication. That is the protection working exactly as designed, on a healthy DC, and it will produce replication activity and a burst of events that look alarming if you were not expecting them.

Do it deliberately, on one DC at a time, during a window, with a healthy replication partner online — not on every DC in the domain on a Friday afternoon.

What the safeguards explicitly do not cover

Here is the part that matters most on Proxmox, and it is the part that people who have read about vmgenid usually have not read.

Microsoft lists two restore methods that virtualized domain controllers do not support safe restore for:

VHD and VHDX files manually copied over existing VHD files

VHD and VHDX files restored using file backup or full disk backup software

Neither of these operations is covered under VM-GenerationID semantics and therefore don’t change the VM-Generation ID. Restoring domain controllers using these methods could either result in a USN rollback and either quarantine the domain controller or introduce lingering objects and the need for forest wide cleanup operations.

A vzdump restore is full-disk backup software

vzdump takes a full-disk image of the guest. Restoring it writes that image back. From Active Directory’s point of view that is the second bullet above, and the consequence named is not theoretical — USN rollback, DC quarantine, or lingering objects requiring forest-wide cleanup.

Whether your particular restore changes the generation ID is a question about your hypervisor’s behaviour, not about Microsoft’s protection model, and it is one you must verify rather than assume. The value is visible in the VM configuration, which makes the test simple and worth doing before you need it:

Record vmgenid from qm config for a test DC.
Restore that guest from a vzdump archive, on an isolated network.
Read vmgenid again.
Boot it and check the directory service event log for the safeguard events — the invocation ID reset and the RID pool discard.

If the value did not change, or the guest did not trigger safeguards, then a restore of that guest is an unprotected USN rollback and must never be used as an AD recovery mechanism. If it did, you have protection — and you still should not rely on it, for the reason in the next section.

Do this test on a lab domain. Never on production, and never on a network where the restored DC can reach a real one.

Even when it works, it is not a recovery strategy

The safeguards are a seatbelt, not a reason to drive into walls. Microsoft is unusually direct about their scope:

Virtualized domain controller safe restore is not a replacement for system state backups and the AD DS Recycle Bin.

After restoring a snapshot, the deltas of previously un-replicated changes originating from that domain controller after the snapshot are permanently lost. Safe restore implements automated non-authoritative restoration to prevent accidental domain controller quarantine only.

Read the last four words again. The feature exists to stop a rolled-back DC from poisoning the directory. It does not bring back what was on it.

And it comes with requirements that are easy to violate during an incident, which is precisely when they will be violated:

A restored domain controller must be able to contact a writable DC

All domain controllers in a domain must not be restored simultaneously

Any changes originating from a restored domain controller that have not yet replicated outbound since the snapshot was taken are lost forever

The middle one is the incident-shaped failure. If a storage problem takes out several DCs and someone restores all of them from backups, Active Directory replication will recover but SYSVOL replication halts: every DC marks itself non-authoritative and then looks for an authoritative partner, and there is not one. Group Policy and logon scripts stop being served domain-wide, and the recovery is a manual authoritative-sync procedure on one DC.

'Just restore the DC from last night's backup'

A domain controller has a problem — a failed update, a corrupt profile service, a disk that filled. It is 22:00. The instinct is overwhelming and it is what the hypervisor is for: restore last night’s backup, be done in fifteen minutes.

The first diagnosis is that this is a server problem. It is not. A domain controller is not a server that holds data; it is one replica of a distributed database that other replicas are actively reconciling against. Restoring it does not restore a server, it inserts a stale replica into a live replication topology.

What you get, in the good case, is safeguards firing: invocation ID reset, RID pool discarded, a burst of inbound replication, and the permanent loss of anything that originated on that DC and had not replicated out. Password changes made against it in the last few hours. Accounts created by a helpdesk that happened to bind to it. Its own computer account password rotation.

In the bad case — the restore did not change the generation ID, or the DC is running a version without safeguards, or it cannot reach a writable partner — you get a USN bubble that nothing will repair and that surfaces weeks later as inconsistent authentication.

The correct move is almost always the opposite one. A domain with more than one DC does not need this DC. Seize its FSMO roles if it held any, forcibly remove it from the directory, clean up its metadata, and build a new DC. That is slower on the night and it is finished when it is finished, with no possibility of a bubble.

The corollary is the design rule: run enough domain controllers that losing one is a rebuild rather than a recovery. Two is the minimum for a production domain. That is not redundancy for uptime; it is what makes the safe option available at 22:00.

What a DC backup strategy looks like instead

You still need backups. You need different ones, and you need to be clear about what each is for.

Backup	What it is for	What it is not for
System state backup inside the guest, taken with Windows Server Backup or an AD-aware product	Authoritative restore of deleted objects; forest recovery; the supported AD recovery path	Routine “put the server back”
AD Recycle Bin	Recovering deleted objects without any restore at all	Anything structural
`vzdump`/PBS backup of the DC VM	Rebuilding the machine after a disaster in which the whole site is gone and there is no surviving DC to replicate from	Fixing one broken DC in a healthy domain
Additional domain controllers	Everything else	Nothing; this is the primary mechanism

The distinction that resolves most arguments: a healthy domain repairs a broken DC by replication, not by restore. Hypervisor backups of DCs exist for the case where there is nothing left to replicate from, and in that case you are running a forest recovery procedure, not a VM restore.

If you do take hypervisor backups of DCs, take them properly. The guest agent is what makes the difference:

Configuration changeensure a DC guest is backed up with a quiesced filesystem— The agent is what lets the backup ask Windows to freeze the filesystem. Without it, the backup is crash-consistent.

VMID=120

qm set "$VMID" --agent enabled=1
qm agent "$VMID" ping && echo "agent responding"

# fsfreeze-status should report thawed on a healthy guest
qm agent "$VMID" fsfreeze-status

The Proxmox guest agent documentation gives the purpose directly: the agent exists to shut the guest down properly rather than relying on ACPI, and “To freeze the guest file system when making a backup/snapshot” — using VSS on Windows. A vzdump of a Windows DC without a working agent is a crash-consistent image of a database mid-write. It may restore. It is not a thing to plan around.

Do not run every domain controller on one hypervisor host

Microsoft states it plainly:

Don’t run all domain controllers in a forest or domain on the same hypervisor host. That introduces a single point of failure that debilitates AD DS, Exchange, SQL, and other enterprise operations each time the hypervisor goes offline.

On Proxmox this is an anti-affinity rule, and Part XII gives you the mechanism: HA rules that keep the DCs on separate nodes. Set it explicitly. Left to the scheduler, two DCs will end up on the same node eventually, and the first evidence will be a node reboot that takes authentication with it.

For a multi-site estate, extend the rule: at least one DC per site that has users in it, on hardware that does not share a failure domain with the others. Part II covers failure domains; this is the workload that most repays taking them seriously.

Time

Active Directory authentication is Kerberos, and Kerberos rejects tickets whose timestamps fall outside a tolerance window — five minutes by default. A domain controller with a wrong clock does not degrade; it stops authenticating, and the error messages point everywhere except at the clock.

The Windows Time Service hierarchy has one shape and it is not negotiable:

The PDC emulator of the forest root domain is the authoritative source for the forest. It should be configured to synchronise with an external time source — your network’s NTP servers, or an internet pool.
Every other DC synchronises from a DC higher in the hierarchy. Microsoft describes the default behaviour of a DC forcing “NT5DS (Windows NTP) time synchronization with another domain controller (in a default Windows Time Service hierarchy, this means using the PDCE)”.
Domain members synchronise from a DC.

The virtualisation-specific rule follows from that: a domain-joined guest must not have a second time source competing with the domain hierarchy. On hypervisors that inject host time into the guest, that injection has to be disabled on DCs. On KVM there is no always-on host time injection to disable, which removes the classic version of this problem — but three Proxmox-specific points remain:

localtime is on by default for a Windows ostype, and off for everything else. A Windows DC created with ostype: other (Lesson 1) has a hardware clock in UTC and a wall clock offset by your timezone. Kerberos will notice.
A guest that has been paused, suspended or migrated may resume with a stale clock. It will correct itself against the hierarchy, but the interval before it does is an interval in which authentication may fail. Watch for it after maintenance.
The PDC emulator’s external source has to actually work. If it is unreachable, the whole forest drifts together, which is worse than one machine drifting, because nothing looks inconsistent internally until something external — a certificate, a federated login, a backup system — disagrees.

Read-only / Safeverify the time hierarchy from inside a DC

PS C:\> w32tm /query /status ; w32tm /query /source ; w32tm /monitor

Leap Indicator: 0(no warning)
Stratum: 3 (secondary reference - syncd by (S)NTP)
Precision: -23 (119.209ns per tick)
Source: dc01.example.com
Poll Interval: 10 (1024s)

dc01.example.com[192.0.2.10:123]:
  ICMP: 0ms delay
  NTP: +0.0011870s offset from local clock
dc02.example.com[192.0.2.11:123]:
  ICMP: 1ms delay
  NTP: -0.0004410s offset from local clock

Illustrative output

An offset in the low milliseconds is healthy. An offset in the tens of seconds is an incident that has not started yet.

Under the hood

Why does the invocation ID reset fix the problem when a USN reset does not?

Replication state is not tracked as “DC2 has seen changes up to number N from DC1”. It is tracked as a pair: (invocation ID, USN). DC2’s up-to-dateness vector says “I have everything originated under invocation ID A up to USN 200”.

The invocation ID identifies a specific instance of the directory database, not the domain controller. That distinction is the whole design. A DC keeps its name, its computer account, its GUID and its FSMO roles across a database instance change; what changes is the identity of the sequence of updates it produces.

So when the rolled-back DC1 adopts invocation ID B, it has not claimed to be a new domain controller. It has claimed that the change sequence it is about to produce is a new sequence, unrelated to the one DC2 has records of. DC2 has no watermark for (B, anything), so it requests everything from the beginning of that sequence. No gap can form, because there is no stale watermark to skip over.

And the changes DC2 holds under (A, 101-200) are still valid records of things that happened. They replicate back into DC1 by the ordinary rules. The directory converges on the union rather than on either machine’s view.

This is also why simply advancing DC1’s USN past 200 would not work, even if something could calculate the right number. The USNs between 101 and 200 have already been used under invocation ID A for a specific set of changes; reissuing them under the same invocation ID for different changes is the corruption, not the numbering. Only a new sequence identity removes the ambiguity.

The RID pool discard addresses the parallel problem in a different namespace. RIDs are allocated to a DC in blocks by the RID master, and the DC hands them out to new security principals. A rolled-back DC believes it still holds RIDs it has already issued. Discarding the pool and requesting a fresh block from the RID master is the only way to guarantee it does not mint a second principal with an SID that already exists — a duplicate that no cleanup tool can safely resolve, because both objects are legitimate as far as every ACL referencing them is concerned.

What happens if a DC is restored and you need to check whether damage was done?

Suppose the restore has already happened — someone rolled back a snapshot, or restored a backup, before reading any of this. The question is whether the safeguards fired.

Check the directory service event log on the restored DC first. A safeguard event will record the invocation ID reset and the RID pool invalidation. If those events are present, the protection worked and your remaining problem is data loss, not divergence: anything originated on that DC and not yet replicated outbound is gone permanently.

If those events are absent, treat the DC as suspect. Take it off the network before it can originate any further changes, and check replication from a partner rather than from the suspect machine:

Read-only / Safeask a healthy partner what it thinks it is missing

C:\> repadmin /showrepl dc02.example.com /repsto

Default-First-Site-Name\DC02
DSA Options: IS_GC
DSA object GUID: 5d083398-4bd3-48a4-a80d-fb2ebafb984f
DSA invocationID: 730fafec-b6d4-4911-88f2-5b64e48fc2f1

==== OUTBOUND NEIGHBORS FOR CHANGE NOTIFICATIONS ============

DC=example,DC=com
  Default-First-Site-Name\DC01 via RPC
      DSA object GUID: f62978a8-fcf7-40b5-ac00-40aa9c4f5ad3
      Last attempt @ 2026-08-12 21:14:03 was successful.

Illustrative output

The invocation ID in that output is the thing to compare. If the suspect DC is still presenting the invocation ID it had before the restore, no safeguard fired and you are looking at a potential USN bubble.

repadmin /showchanges PARTNER GUID NAMING-CONTEXT /statistics gives the count of changes a partner believes it has not received, which is the closest thing to a direct measurement of the gap.

The safe resolution for an unprotected rollback is removal, not repair. Take the DC out of the directory, clean its metadata, and rebuild it. A USN bubble cannot be repaired by replication, by definition — replication is the thing that does not know it exists.

And this same procedure is why you run the test in the callout above before an incident. Establishing whether your restore path changes the generation ID is a twenty-minute lab exercise on a quiet afternoon, or a forensic question at midnight.

Domain controllers on Proxmox: the rules

Design

At least two DCs per domain, so that losing one is a rebuild rather than a recovery.
Never all DCs on one hypervisor host. Enforce with HA rules, do not leave it to the scheduler.
At least one DC per site with users, in a separate failure domain.
ostype set to a Windows value, so localtime is correct.
vmgenid present in the VM configuration. Audit it; do not assume it.
Guest agent installed and answering, so backups can quiesce with VSS.

Operations

Do not snapshot a DC as a routine change-control step. Use a second DC as your rollback.
Do not restore a DC to fix a problem in a domain that has another healthy DC. Remove and rebuild instead.
Never restore all DCs simultaneously. SYSVOL replication will halt and need a manual authoritative sync.
Never clone a DC VM without the supported cloning procedure and its configuration file. An unprepared copy that reaches the network is a duplicate DC.
Time: PDC emulator of the forest root syncs externally; everything else syncs down the hierarchy; nothing else competes.

Backups

System state backups inside the guest, on a schedule, tested. That is the supported AD recovery path.
AD Recycle Bin enabled, so object deletion never needs a restore at all.
Hypervisor backups of DCs kept for the site is gone case only, clearly labelled as forest-recovery input rather than as a restore you can perform casually.
A tested restore procedure, run in a lab, that establishes what your restore path does to vmgenid.

Key takeaways

USN rollback happens because replication tracks (invocation ID, USN) pairs and a rolled-back DC reissues numbers a partner believes it already has. The divergence is permanent and silent.
vmgenid is a Proxmox VM option — 1 to autogenerate, 0 to disable — that AD compares against a value in its database. A mismatch triggers an invocation ID reset and a RID pool discard.
Microsoft explicitly excludes disks restored by full disk backup software from VM-GenerationID semantics. Verify what your restore path does to vmgenid, in a lab, before you need to know.
Safeguards prevent quarantine only. Un-replicated changes originating on the restored DC are lost forever, and the feature is explicitly not a replacement for system state backups or the AD Recycle Bin.
A healthy domain repairs a broken DC by replication. Remove and rebuild beats restore, which is why two DCs is a minimum rather than a nicety.
Never restore all DCs at once — SYSVOL replication halts — and never run them all on one hypervisor host.
Kerberos has a five-minute default tolerance. The forest root PDC emulator syncs externally; everything else syncs down the hierarchy; nothing competes.

Knowledge check

Why this matters

Virtualisation made one operation cheap that used to be expensive: putting a machine back the way it was. That single capability is why snapshots feel like safety, and it is correct for almost every guest in your estate.

Domain controllers are the exception, and they are the exception in a particularly cruel way: the operation is not merely ineffective, it converts a local problem into a distributed one, and it does so without any error at the time. The estate keeps working. Authentication keeps succeeding. The divergence surfaces later, in a form that does not resemble its cause, and by then the change record that would explain it is three months old.

The engineering response is not to be careful. Careful people restore domain controllers at 22:00 because the alternative looks slower and the risk is invisible. The response is structural: run enough domain controllers that the safe option is also the available one, keep them on separate hosts so that one failure cannot force the unsafe choice, take the backups that Active Directory actually supports, and test — once, in a lab, in daylight — what your restore path does to a generation ID.

Do that and the 22:00 decision stops being a decision. There is a second DC, the first one gets removed and rebuilt, and nobody has to be careful about anything.