CRS and dynamic load balancing (PVE 9.2) · Proxmox VE

Every time the HA stack needs to decide where a guest should run, something makes that decision. That something is the Cluster Resource Scheduler, and until recently it was easy to ignore because its default answer was crude and its scope was narrow: recovery after a node failure, and nothing else.

PVE 9.2 changes both halves. There is a dynamic scheduler mode that reads actual load, and there is an automatic rebalancer that can act on it while nothing has failed at all. If you upgrade a cluster and later find migrations in the task log that nobody ordered, this is the lesson that explains them.

The three modes

CRS is configured in /etc/pve/datacenter.cfg under a single crs key with several sub-options. The scheduler mode is ha=:

Mode	Decides on	Behaviour
`basic`	Guest counts and HA rules only	The historical default. Spreads services numerically.
`static`	Configured CPU and memory of each guest	Packs by what guests were allocated, not what they use.
`dynamic`	Observed node load	Places by what the nodes are actually doing.

The distinction between static and dynamic is the one that matters, and the word “static” is a little misleading. static is not a fixed placement - it is a scheduler whose inputs are static: the cores and memory numbers in each guest’s config. It does not care whether a VM configured with 32 GiB is using 2 GiB or 31 GiB.

dynamic uses observed load, so it converges on something closer to the truth

and it also means the scheduler’s decisions change over time on a cluster whose configuration has not changed at all.

Read-only / Safewhat is CRS set to right now— Read-only. Run this before assuming anything about a cluster you inherited - an absent crs line means the basic default, which is not what a recent installer necessarily leaves behind.

set -euo pipefail

grep -E '^crs:' /etc/pve/datacenter.cfg || echo 'crs not set - default ha=basic'

# The full parsed view, including defaults the file does not spell out.
pvesh get /cluster/options --output-format json-pretty | head -40

Cluster-wide riskchange the scheduler mode— Affects placement decisions for every HA-managed guest on every node. It does not move anything by itself - but it changes where the next recovery, and every rebalance, will place guests.

set -euo pipefail

# Observe first. Do not enable rebalancing in the same change.
pvesh set /cluster/options --crs 'ha=dynamic'

grep -E '^crs:' /etc/pve/datacenter.cfg

Automatic rebalancing: the part that moves things

Setting ha=dynamic changes where new placements go. It does not, on its own, move a running guest. That is a separate switch, and it is off by default:

Key	Default	What it controls
`ha-auto-rebalance`	`0`	Whether the cluster rebalances running guests at all
`ha-auto-rebalance-threshold`	`30`	Node imbalance percentage that triggers a rebalance
`ha-auto-rebalance-margin`	`10`	Minimum improvement percentage required to commit a migration
`ha-auto-rebalance-hold-duration`	`3`	HA rounds the condition must persist before acting
`ha-auto-rebalance-method`	`bruteforce`	Scoring approach: `bruteforce` or `topsis`
`ha-rebalance-on-start`	`0`	Use CRS when an HA service goes from stopped to started

Read those four numeric keys as one sentence: the cluster acts when nodes differ by more than 30%, the move must improve things by at least 10%, and the condition must hold for 3 consecutive HA rounds first.

Three brakes, all adjustable, all in the direction of not thrashing.

Cluster-wide riskenable automatic rebalancing, conservatively— From this point the cluster live-migrates running guests without operator action. Announce it, and enable it in a window you are watching.

set -euo pipefail

pvesh set /cluster/options --crs \
'ha=dynamic,ha-auto-rebalance=1,ha-auto-rebalance-threshold=40,ha-auto-rebalance-margin=15,ha-auto-rebalance-hold-duration=6'

grep -E '^crs:' /etc/pve/datacenter.cfg

Those numbers are deliberately more conservative than the defaults - a wider imbalance tolerated, a larger improvement demanded, and twice as long for the condition to persist. That is the right starting posture for a first enablement: you want to see the mechanism act a few times before you let it act often.

The migration nobody ordered

The symptom. The task log shows live migrations of production VMs at 03:40. Nobody was working. No node failed. Change management has no record. The first assumption in the room is that someone has credentials they should not have, and a security investigation starts.

What happened. ha-auto-rebalance=1 was enabled - during an upgrade, by a colleague testing something, or because a template config was applied cluster-wide - and the nightly backup window pushed one node’s observed load past the imbalance threshold for long enough to satisfy the hold duration. The scheduler did exactly what it was told.

Why the first diagnosis is wrong. Everyone searches the audit log for a user who initiated a migration, and there is none, which makes the security theory look stronger. The task’s initiator is the HA stack itself. You will not find a person because there was not one.

How to confirm it in one command:

grep -E '^crs:' /etc/pve/datacenter.cfg
ha-manager status --verbose | head -30

If ha-auto-rebalance=1 appears there, that is your answer, and the whole question becomes a documentation problem rather than a security one.

The lasting fix is not to disable it. It is to record in the cluster’s operational documentation that the scheduler is authorised to migrate guests, and to make that visible to whoever reads the task log at 04:00. An automated action that the on-call engineer has never heard of is an incident regardless of whether it was correct.

Why there is a hold duration and a margin, and not just a threshold

A scheduler with one threshold and no hysteresis oscillates. This is a general property of feedback control, and it shows up here in a specific and expensive form.

Suppose the rule were simply “if node A is more than 30% busier than node B, move a guest from A to B”. Migrating a guest is not free: it consumes CPU on both nodes, saturates the migration network, and dirties memory pages that must be re-sent. So the act of rebalancing temporarily raises load on both ends. The scheduler observes the new imbalance, decides again, and can end up migrating the same guest back and forth indefinitely - each migration causing the load spike that justifies the next one.

The three brakes each cut one part of that loop:

ha-auto-rebalance-hold-duration requires the imbalance to persist for several HA rounds, so a transient spike - a backup job, a batch run, a compile - never triggers a migration at all. This is the one that matters most in practice, because most load imbalance is transient.
ha-auto-rebalance-margin requires the predicted post-move state to be meaningfully better, not merely different. Without it the scheduler would act on noise, since two nodes are never exactly equal.
ha-auto-rebalance-threshold sets how unequal the cluster has to be before anyone cares. It answers “is this worth an outage-shaped event”, and the honest answer on most clusters is “not at 5%”.

The ha-auto-rebalance-method key chooses how candidate moves are scored. bruteforce evaluates candidate placements directly. topsis uses a multi-criteria ranking method, which weighs several dimensions - CPU, memory, and so on - against an ideal rather than optimising one of them. On a cluster with heterogeneous nodes the multi-criteria approach usually produces less surprising results; on a uniform cluster the difference is small. Change it only with a reason, and change one key at a time so you can attribute the effect.

Constraining the scheduler

Three levels of control, from narrowest to broadest.

Exclude one guest

Every HA resource has an auto-rebalance property, documented as “HA resource may be migrated during automatic rebalancing”, defaulting to 1.

Configuration changeopt a single guest out of automatic rebalancing— Changes one HA resource. The guest can still be migrated manually and will still be recovered by HA after a node failure - only the automatic rebalancer leaves it alone.

set -euo pipefail
VMID=101

ha-manager set "vm:$VMID" --auto-rebalance 0
ha-manager config --type vm

Reach for this on anything with a reason not to move: a latency-sensitive database whose replication would notice, a guest with device passthrough that is pinned to specific hardware, a licensing arrangement tied to a socket count.

Constrain where it may move things

Node-affinity rules bound the placement search for every scheduler mode. The scheduler optimises within the rules; it does not override them.

Configuration changekeep a tier on the nodes it belongs on— Creates a node-affinity rule. The scheduler may rebalance among the listed nodes and nowhere else.

set -euo pipefail

ha-manager rules add node-affinity db-tier \
--resources 'vm:101,vm:102' \
--nodes 'pve-01:2,pve-02:1' \
--strict 1

ha-manager rules config

Negative resource-affinity is the other useful shape here: two guests that must never share a node - a database primary and its replica, two halves of a quorum - stay separated no matter what the load numbers say.

Turn rebalancing off, keep dynamic placement

Cluster-wide riskstop automatic migrations without reverting to basic scheduling— Disables the rebalancer. Recovery placement after a node failure still uses the dynamic scheduler, which is usually what you want.

set -euo pipefail

pvesh set /cluster/options --crs 'ha=dynamic,ha-auto-rebalance=0'
grep -E '^crs:' /etc/pve/datacenter.cfg

This is the configuration most production clusters should sit at for their first months on 9.2: better recovery placement, no unattended migrations.

Observing a decision

Read-only / Safewatch the CRM decide— Read-only. The CRM master's log is where placement decisions are recorded; run this on the node holding the manager lock.

set -euo pipefail

# Which node is the CRM master right now.
ha-manager status | grep -i master

# Its reasoning, live.
journalctl -u pve-ha-crm -f --no-pager

# Migrations that the HA stack initiated, rather than a user.
pvesh get /cluster/tasks --output-format json-pretty \
| grep -B2 -A4 'qmigrate' | head -40

Read-only / Safewhat a rebalance looks like in the CRM log

# journalctl -u pve-ha-crm --since '30 min ago' --no-pager | tail -8

pve-ha-crm[2411]: node 'pve-01' load 0.81, 'pve-02' load 0.34, 'pve-03' load 0.39
pve-ha-crm[2411]: imbalance 47% exceeds threshold 40%, round 6 of hold 6
pve-ha-crm[2411]: crs: considering vm:104 pve-01 -> pve-02, predicted improvement 22%
pve-ha-crm[2411]: service 'vm:104': migrate to node 'pve-02'
pve-ha-crm[2411]: migrate service 'vm:104' to node 'pve-02' (running)
pve-ha-crm[2411]: service 'vm:104': state changed from 'started' to 'migrate'
pve-ha-crm[2411]: service 'vm:104': state changed from 'migrate' to 'started'

Illustrative output

Every input to the decision is in those lines: the observed loads, the imbalance against the threshold, the hold-duration counter, and the predicted improvement against the margin. When a migration surprises you, this is the record that explains it - and when the scheduler is not doing something you expected, it is where you find which brake stopped it.

What happens if you enable dynamic CRS on a cluster with no shared storage?

Nothing, and the nothing is quiet.

CRS is part of the HA stack, and HA cannot place a guest on a node that cannot open its disks. A guest on local storage has exactly one viable node, so the scheduler’s placement search returns that node regardless of mode, threshold or margin. Changing ha=basic to ha=dynamic on such a cluster changes no outcome at all.

This produces one of the more frustrating support conversations: an operator enables dynamic scheduling and rebalancing, watches for a week, sees no migrations, and concludes the feature is broken. It is working perfectly; it has no choices to make.

The check that tells you which situation you are in:

pvesm status
ha-manager config

Cross-reference the storages backing each HA resource against the shared column. Every guest whose storage is not shared is invisible to the scheduler. The prerequisite matrix in what HA requires is the systematic version of that check.

The same reasoning explains a subtler case: a cluster with mixed storage, where the scheduler appears to rebalance some guests aggressively and ignore others entirely. It is not playing favourites. The ignored ones have nowhere to go.

Rolling it out without an incident

A staged sequence, because every step here is observable and the temptation is to do them all at once.

Record the current setting and put it in the change ticket. grep crs: /etc/pve/datacenter.cfg, including the case where it is absent.
Move to ha=static first if you were on basic. Placement now accounts for guest size, which is an improvement on counting, and nothing moves on its own.
Move to ha=dynamic. Still nothing moves on its own. Watch recovery placement across a planned node reboot and confirm it lands sensibly.
Opt out the guests that must not move with --auto-rebalance 0, and put node-affinity rules on the tiers that have placement constraints. Do this before the next step, not after.
Enable ha-auto-rebalance=1 with a raised threshold and hold duration. Watch the CRM log for a week.
Tighten towards the defaults only if the observed behaviour is boring.

The step people skip is 4, and it is the one that turns step 5 from a configuration change into an incident.

Knowledge check

Why this matters

Automatic rebalancing is the first Proxmox feature that will move a production workload without a human deciding to. That is a genuine step change, and it is worth treating as one rather than as another tuning parameter.

The technical risk is modest. The brakes are well chosen, the defaults are conservative, and live migration is a mature operation. The operational risk is entirely about attribution: a migration in the task log with no initiator, seen by an engineer who does not know the feature exists, is indistinguishable from a compromise until someone reads a config file. That is an expensive hour and an avoidable one.

So the work here is half configuration and half communication. Decide which guests must not move and say so in the config, not in someone’s memory. Write down that the cluster is authorised to migrate guests, where the on-call engineer will find it. And enable the rebalancer separately from the scheduler mode, in a window you are watching, so that the first few decisions are ones you saw happen rather than ones you found afterwards.