HA operational procedures: enable, monitor, disarm · Proxmox VE

Why this matters in production

HA needs day-2 operations: enabling HA on VMs, monitoring it, and disarming it deliberately during maintenance. Each operation has gotchas.

Enabling HA on a VM

GUI: VM → HA → Activate. CLI:

ha-manager add vm:100 --state started

Configuring HA state

ha-manager set vm:100 --state started
ha-manager set vm:100 --state stopped
ha-manager set vm:100 --state disabled
ha-manager set vm:100 --state ignored

The requested states you set, and what each means:

State	Behaviour
`started`	HA keeps the VM running, restarting and relocating it as needed
`stopped`	HA keeps the VM stopped, but still relocates it on node failure
`disabled`	Stopped, and not relocated on failure
`ignored`	Removed from HA management entirely while the resource entry stays

The distinction between stopped and disabled is the one that surprises people. A stopped resource is still HA’s business: if its node fails, HA moves the resource’s ownership to a survivor and keeps it stopped there. A disabled resource is left alone. If you are stopping a guest for a week and do not want the HA stack tracking it through a node failure, disabled is the one you want.

Separately, the CRM maintains internal states you will see in ha-manager status and cannot set directly:

State	Meaning
`fence`	Waiting for the node holding this resource to be fenced
`recovery`	Node fenced; waiting for a viable target
`error`	The LRM reported a failure. Manual intervention required.

Reading those three correctly during an incident is most of the diagnosis. A service sitting in fence for minutes means fencing is not completing - see anatomy of a bad fence. A service in recovery that does not progress means no node can run it, which is usually storage - see what HA requires.

Configuration changethe other resource options, and when they matter— Changes one HA resource. All of these are per-resource and none of them affect anything else.

set -euo pipefail
VMID=100

ha-manager set "vm:$VMID" \
--max_restart 2 \
--max_relocate 1 \
--failback 0 \
--auto-rebalance 0 \
--comment 'payments-db: pinned, do not auto-move'

ha-manager config --type vm

Option	Default	Reach for it when
`max_restart`	1	A guest that sometimes fails its first start on a cold node
`max_relocate`	1	You want HA to try more than one alternative before giving up
`failback`	1	A guest should stay where it was recovered rather than moving back
`auto-rebalance`	1	The guest must not be moved by the CRS rebalancer
`comment`	—	Always. It is the only place the why survives.

failback 0 is under-used. With the default, a guest recovered onto a survivor migrates back to its preferred node as soon as that node returns - which means a node failure costs you two service interruptions rather than one, the second at an unpredictable time when the node happens to come back.

Removing HA from a VM

ha-manager remove vm:100

Moving HA-managed guests by hand

Ask the CRM to do it, rather than migrating behind its back. ha-manager migrate and ha-manager relocate are aliases for the corresponding crm-command forms.

# Online migration, guest keeps running (VMs).
ha-manager migrate vm:100 pve-02

# Stop on the old node, start on the new one.
ha-manager relocate vm:100 pve-02

Node maintenance

For work on one node - firmware, a reboot, a NIC swap - PVE has a purpose- built mode. It marks the node unavailable for placement and migrates its HA resources away, while leaving HA fully armed everywhere else.

Service impact possiblenode maintenance mode— Queues a CRM command that migrates HA resources off the node. VMs migrate online; containers restart. The LRM watchdog stays active until the migrations complete.

set -euo pipefail
NODE=pve-03

ha-manager crm-command node-maintenance enable "$NODE"

# Wait until nothing HA-managed is still assigned to that node.
ha-manager status | grep -E "service .*\($NODE," || echo 'node is clear'

# ... perform the maintenance, reboot as needed ...

ha-manager crm-command node-maintenance disable "$NODE"
ha-manager status

Services that were on the node migrate back when maintenance mode is disabled. If you do not want that, set --failback 0 on the resources concerned before you start.

Disarming the whole stack

For work that affects every node - a corosync network change, replacing the switch that carries all cluster links - there is a documented cluster-wide disarm. It releases the watchdogs everywhere, so there is no automatic failover at all while it is in effect.

Cluster-wide riskdisarm and re-arm HA cluster-wide— While disarmed there is no automatic failover anywhere. Open this window deliberately, keep it short, and close it explicitly. Confirm the state in ha-manager status before and after.

set -euo pipefail

# freeze: leave services running and untouched.
# ignore: release HA's claim on them entirely.
ha-manager crm-command disarm-ha freeze

ha-manager status | grep -iE 'fencing|watchdog|disarm'

# ... the disruptive work ...

ha-manager crm-command arm-ha
ha-manager status

Two commands that do not exist

ha-manager disable --scope node and ha-manager watchdog appear in a lot of blog posts and in at least one earlier revision of this course. Neither is a real subcommand. ha-manager has no disable, enable or watchdog verb at all, and running them gets you a usage error, not a disarmed cluster.

The real commands are the two above:

One node, planned work: ha-manager crm-command node-maintenance enable|disable NODENAME
Whole cluster, disruptive work: ha-manager crm-command disarm-ha freeze|ignore, then ha-manager crm-command arm-ha

The distinction matters beyond the syntax. Maintenance mode leaves fencing armed everywhere else, so the rest of the cluster is still protected while you work on one node. Disarming removes that protection cluster-wide. Reaching for the second when you needed the first leaves an entire cluster with no automatic failover for the length of a firmware update.

Monitoring HA state

ha-manager status --verbose

Read-only / Safewhat a healthy cluster looks like, and where to look when it is not

# ha-manager status

quorum OK
master pve-01 (active, Wed Aug 12 09:41:02 2026)
lrm pve-01 (active, Wed Aug 12 09:41:05 2026)
lrm pve-02 (active, Wed Aug 12 09:41:04 2026)
lrm pve-03 (idle, Wed Aug 12 09:41:03 2026)
service ct:210 (pve-02, started)
service vm:100 (pve-01, started)
service vm:104 (pve-03, error)

Illustrative output

Reading it:

quorum - anything other than OK and nothing else on the page matters.
master - which node holds the manager lock. A timestamp that is not advancing, or a master that changes repeatedly, is a cluster problem.
lrm - active means it is managing resources; idle means it has none assigned, which is normal on a node with no HA guests and abnormal on one that should have some. old timestamp - dead? is a node the cluster has lost.
service - the node it is assigned to, and its state.

The watchdog state appears in the verbose output as armed, standby, disarming or disarmed. There is no ha-manager watchdog command; to check that a node has a watchdog device at all - which is a different and more important question - test for the device:

for NODE in pve-01 pve-02 pve-03; do
  printf '%-10s ' "$NODE"
  ssh -o BatchMode=yes "root@$NODE" \
    'test -c /dev/watchdog && echo present || echo "MISSING - cannot self-fence"'
done

Recovering from error state

If a resource enters error state, HA has stopped trying and is waiting for a human. The recovery is three steps and the middle one is the whole job.

Service impact possibleclear an error state - after fixing the cause— Setting the resource to disabled clears the error flag. Doing it before the cause is fixed simply produces the error again, and each cycle costs another set of failed start attempts.

set -euo pipefail
VMID=100

# 1. Find out WHY before touching anything. The CRM master's log names it.
ha-manager status | grep "vm:$VMID"
journalctl -u pve-ha-crm --since '1 hour ago' --no-pager | grep -i "vm:$VMID"
journalctl -u pve-ha-lrm --since '1 hour ago' --no-pager | grep -i "vm:$VMID"

# 2. Fix the cause: storage offline, no capacity, missing bridge, an
#    unsatisfiable strict rule, a guest config referring to a device this
#    node does not have.

# 3. Only then clear the flag and re-enable.
ha-manager set "vm:$VMID" --state disabled
ha-manager set "vm:$VMID" --state started
ha-manager status | grep "vm:$VMID"

The error state that comes straight back

The symptom. A service is in error. Someone runs the disabled/started pair, it starts, and forty seconds later it is in error again. They repeat it. On the third attempt someone suggests removing the guest from HA.

What is happening. The disabled/started cycle clears the flag, not the fault. HA immediately tries again, hits the same problem, exhausts max_restart and max_relocate, and returns to error. The commands worked exactly as documented.

Why the first instinct is wrong. The instinct is to treat error as a stuck state that needs unsticking, because that is what it looks like. It is not stuck; it is a considered decision by the CRM that further automatic attempts are not useful, and it is right.

Where the answer actually is - one of these four, in this order of likelihood:

VMID=100
pvesm status                                    # storage offline?
ha-manager rules config --resource "vm:$VMID"   # unsatisfiable strict rule?
pvecm status | grep -E 'Quorate|Expected'       # quorum?
journalctl -u pve-ha-lrm --since '2 hours ago' | grep -iE 'error|failed'

And the thing not to do. Removing the guest from HA makes the error disappear because nothing is managing the guest any more. The guest is still broken and now nothing will restart it if its node fails. That is not a fix, it is the removal of the alarm.

Why max_restart and max_relocate exist, and what happens when they run out

When a resource fails to start, the CRM does not retry forever, and the two counters describe a deliberate escalation.

max_restart is how many times the LRM will retry the start on the same node. The reasoning: some failures are local and transient - a storage path that had not finished coming up, a bridge that appeared a second later, an address still held by something shutting down.

max_relocate is how many times the CRM will then try a different node. The reasoning: some failures are properties of the node, not the guest.

When both are exhausted the resource goes to error and everything stops. That final state is the important design decision. The alternative - retry indefinitely - would produce a guest cycling through start attempts across every node in the cluster forever, generating load, filling logs, and hiding the underlying fault behind an infinite stream of near-identical failures.

So error is not a bug report. It is the CRM saying: I have tried this the number of times you configured, on the number of nodes you configured, and something needs a person. Raising the counters to make the state go away raises the number of attempts before that message, and nothing else.

The counters default to 1 each, which is deliberately low. Raise max_restart only where you have observed a genuinely transient local failure, and raise max_relocate only on a cluster where nodes differ enough for the second attempt to have a different outcome.

What happens if you migrate an HA-managed guest with qm migrate instead of ha-manager migrate?

It works, and the HA stack notices and adjusts - but the route through the CRM is still the correct one, and the reason is about who is in charge rather than about breakage.

ha-manager migrate places the request into the CRM’s command queue. The CRM decides when to act on it, checks it against the rules in force, and records the decision in its own log. The resulting migration is part of the state machine that is also handling recovery, rebalancing and fencing.

A direct qm migrate moves the guest, and the CRM reconciles afterwards. On a quiet cluster the difference is invisible. It stops being invisible when something else is happening at the same time:

A node-affinity rule forbids the destination. The CRM will move the guest back, so your migration is undone by a system you did not know was disagreeing with you.
The CRS rebalancer is enabled. Your placement is one input among several and may not survive.
A recovery is in progress. Two things are now moving guests, and the interleaving is nobody’s design.

The habit worth having: on an HA-managed guest, use ha-manager migrate and ha-manager relocate. They are aliases for crm-command forms, they cost nothing extra, and they leave a record in the place where the next person will look when they are trying to work out why a guest is where it is.

Production considerations

Common mistakes

Enabling HA without verifying the prerequisites.
Forgetting to disarm HA before maintenance.
Leaving VMs in disabled state after recovery.

Key takeaways

Use ha-manager add to enable, ha-manager set to change state, ha-manager remove to disable.
Disarm HA before node maintenance.
Watch the master election.

Knowledge check

Why this matters

The commands in this lesson are simple. What is not simple is that two of the ones most widely repeated do not exist, and that the state model has three pairs of terms that look interchangeable and are not: stopped versus disabled, maintenance mode versus disarming, clearing an error versus fixing one.

Every one of those distinctions costs something when it is got wrong, and the cost is always paid at an awkward time. Disarming the cluster when you meant to maintain one node leaves every guest unprotected during a firmware update. Setting stopped when you meant disabled moves a guest you thought was parked. Cycling a service through disabled and started without reading the log turns a ten-minute storage diagnosis into an hour of watching the same failure.

The habit that covers all of them is the same: before running a state-changing HA command, read ha-manager status and the CRM log, and be able to say in one sentence what you expect the command to change. If you cannot, the command is not the next step - the log is.