Keeping clusters identical: baselines, drift and version skew

Drift does not announce itself. It shows up as a sentence in an incident channel: “that works on the other cluster”. By then it has usually been true for months.

The previous two lessons were about deciding to have a second cluster and moving guests to it. This one is about the bill that arrives afterwards, which is not hardware. It is that every cluster-scoped setting now exists twice, with nothing keeping the copies in agreement.

The surface, named

/etc/pve is the whole cluster control plane, and its file map is documented. These are the entries that are cluster-scoped — one copy per cluster, shared by every node — and therefore the entries that can diverge between clusters:

Node-scoped entries — nodes/<NAME>/config, the per-node certificates, and the guest configuration files under nodes/<NAME>/qemu-server/ and nodes/<NAME>/lxc/ — are a different problem and mostly not one you want to make identical.

The naming contract

Before automating anything, agree on names. Three of them do disproportionate work, because cross-cluster operations are name-based and nothing validates meaning.

Storage IDs mean the same thing everywhere

--target-storage local-zfs:ceph-vm is a string-to-string map, and every migration, every script and every runbook that crosses the boundary carries one. If fast-nvme is a local ZFS pool on cluster A and an iSCSI LUN on cluster B, every map has to be written by someone who knows that, and it will eventually be written by someone who does not.

The contract worth adopting is that a storage ID encodes capability and locality, not hardware:

Then --target-storage shared-rbd:shared-rbd is a sentence that means something on both clusters, and 1 — the shorthand that maps each source storage to itself — becomes safe rather than a gamble.

Bridge and VNet names mean the same network

Same argument, worse consequence, because a wrong storage map fails and a wrong bridge map succeeds. Part XXV describes the failure in full: vmbr0 exists on both clusters, means production on one and out-of-band management on the other, and the guest arrives on the wrong VLAN with a static address it still believes in.

Write down, per cluster, what each vmbrN and each VNet is attached to, and treat that document as part of the cluster’s definition. If the two clusters cannot agree on names, agree on a mapping table instead and put it in the runbook — but agree on it once, centrally, not per migration.

VMID ranges do not overlap

This one is free and almost nobody does it. datacenter.cfg has a next-id option with lower and upper bounds — default 100 and 1000000 — controlling the range from which the GUI and API pick the next free ID.

pvesh set /cluster/options --next-id lower=100,upper=99999

Cluster A takes 100–99999, cluster B takes 100000–199999, cluster C the next band. What that buys:

• A VMID is globally unique across the estate, so --target-vmid can always keep the original number and a remote migration never collides.
• A guest ID identifies its home cluster on sight, which shortens every incident conversation and every ticket.
• PBS backup groups, which are keyed by guest type and ID, stay unambiguous across a shared backup server.

Retrofitting is harder than starting this way, but even retrofitting only the allocation range is worth it: new guests stop colliding immediately, and the existing overlap becomes a finite known list rather than a growing one.

Detecting drift

You cannot diff two clusters by reading files, for the reason in the callout above. You can diff them through the API, and pvesh returns JSON on request.

CLUSTER="site-a"
OUT="/var/lib/pve-baseline/${CLUSTER}/$(date +%F)"
mkdir -p "$OUT"

pvesh get /cluster/options       --output-format json > "$OUT/options.json"
pvesh get /storage               --output-format json > "$OUT/storage.json"
pvesh get /access/roles          --output-format json > "$OUT/roles.json"
pvesh get /access/users          --output-format json > "$OUT/users.json"
pvesh get /access/groups         --output-format json > "$OUT/groups.json"
pvesh get /access/acl            --output-format json > "$OUT/acl.json"
pvesh get /access/domains        --output-format json > "$OUT/domains.json"
pvesh get /cluster/firewall/groups  --output-format json > "$OUT/fw-groups.json"
pvesh get /cluster/firewall/rules   --output-format json > "$OUT/fw-rules.json"
pvesh get /cluster/firewall/options --output-format json > "$OUT/fw-options.json"
pvesh get /cluster/sdn/zones     --output-format json > "$OUT/sdn-zones.json"
pvesh get /cluster/sdn/vnets     --output-format json > "$OUT/sdn-vnets.json"
pvesh get /cluster/backup        --output-format json > "$OUT/backup-jobs.json"
pvesh get /pools                 --output-format json > "$OUT/pools.json"

ls -l "$OUT"

Then compare. The only trick is that these lists are unordered and carry per-cluster noise (digests, node names, counts), so normalise before diffing.

A="/var/lib/pve-baseline/site-a/2026-08-12/storage.json"
B="/var/lib/pve-baseline/site-b/2026-08-12/storage.json"

norm() {
jq -S 'map({storage, type, content, shared, nodes, disable})
       | sort_by(.storage)' "$1"
}

diff -u <(norm "$A") <(norm "$B") || true

$ diff -u <(norm "$A") <(norm "$B")

   {
   "content": "images,rootdir",
   "disable": null,
-    "nodes": null,
+    "nodes": "pve-b1,pve-b2",
   "shared": 1,
   "storage": "shared-rbd",
   "type": "rbd"
 },
 {
   "content": "backup",
-    "shared": 1,
-    "storage": "pbs-primary",
-    "type": "pbs"
+    "shared": 1,
+    "storage": "pbs-secondary",
+    "type": "pbs"
 }

Version skew

Two different rules, frequently conflated.

Inside a cluster, versions must match. The Cluster Manager chapter is explicit that all nodes should run the same version for HA deployments, and the mixed-version state is a transient condition during a rolling upgrade, not a configuration.

Across clusters, versions do not have to match, and insisting they do throws away one of the main reasons for splitting: being able to validate an upgrade somewhere before production meets it.

What you need instead is a written skew policy. A workable one:

• One minor version of skew, maximum, and only in one direction. The validation cluster runs ahead; production follows within a defined window. Nothing runs behind by more than one minor release.
• Remote migration only between clusters at the same version, until you have tested otherwise on guests that do not matter. The feature is experimental and cross-version behaviour is not documented.
• A hard stop on major versions. Do not leave two clusters on different major releases for longer than the migration project takes.
• The skew is recorded and has an owner and an end date, or it is not a policy, it is a backlog.

pveversion -v

pvesh get /nodes --output-format json | jq -r '.[].node' | while read -r n; do
printf '%s\t' "$n"
pvesh get "/nodes/$n/version" --output-format json | jq -r '"\(.release) \(.version)"'
done

What to automate, and in what order

Not everything deserves a playbook. Order by blast radius of getting it wrong, and by how often it changes.

Automate first — high value, low risk, changes rarely, and the drift hurts:

1. Repositories and pinning. Two clusters on different repository sets is how version skew becomes accidental rather than chosen.
2. datacenter.cfg options, especially migration, bwlimit, ha (shutdown_policy) and next-id.
3. Roles and ACL structure. The role definitions and the shape of the permission tree, not the individual user list.
4. Storage definitions, once the naming contract exists.
5. Firewall security groups, IPsets and aliases. Rules that reference them can stay hand-managed; the objects they reference should not.
6. Notification targets and matchers, so an alert is not loud on one cluster and silent on the other.

Automate later, carefully — SDN objects (an apply is disruptive), backup jobs (windows genuinely differ per site), HA rules (per-guest and rarely transferable).

Do not automate — anything under priv/, certificates, corosync.conf, node network configuration. These are per-cluster by nature, and a playbook that templates them is a playbook that will one day render cluster A’s identity onto cluster B.

Part XX covers the tooling: xx-cli-ansible-proxmox for the fleet mechanics and xx-automation-idempotency for the discipline that stops a playbook from being a second source of drift.

Key takeaways

• Cluster-scoped configuration lives in /etc/pve: datacenter.cfg, storage.cfg, user.cfg, domains.cfg, firewall/cluster.fw, ha/*, sdn/*, status.cfg, vzdump.cron, ceph.conf, corosync.conf. Every one of them can diverge.
• Never copy /etc/pve between clusters. It carries each cluster’s CA key, ticket authkey, node certificates, token secrets and corosync membership. The unit of synchronisation is the API call, not the file.
• Storage IDs should encode capability and locality, not hardware, so that a storage map is meaningful on both sides — and so the 1 shorthand is safe.
• Bridge and VNet names must mean the same network on both clusters, or the mapping table must be written down centrally. A wrong bridge map succeeds.
• Allocate non-overlapping VMID ranges with pvesh set /cluster/options --next-id lower=,upper=. It makes IDs globally unique, keeps --target-vmid usable, and disambiguates PBS backup groups.
• Build the drift report from pvesh ... --output-format json, normalise with jq, diff peer-to-peer and against yesterday, and keep an allow-list of intended differences so the report is empty when the estate is right.
• Versions must match inside a cluster; across clusters a written skew policy is better than enforced uniformity, because being able to validate an upgrade is one of the reasons the second cluster exists.
• Automate repositories, datacenter options, roles, storage and firewall objects first. Never automate priv/, certificates, corosync.conf or node networking.

Knowledge check