Skip to main content
RunBook Academy

← All runbooks in Proxmox VE

medium riskcluster affecting~60 min

Put a node into maintenance

1 · Prerequisites

Confirm every item is in place before any state change.

  • Cluster has capacity for the node to be evacuated
  • HA is configured to allow disarm
  • Maintenance window approved
  • Communication sent to stakeholders

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · HA status shows the node has VMs that can be migrated
  • · pvesh get /cluster/resources --output-format json lists VM distribution

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Communicate to stakeholders
  2. 2Prefer per-node maintenance mode where the work is confined to one node: ha-manager crm-command node-maintenance enable NODENAME. It migrates HA guests away and leaves fencing armed everywhere else
  3. 3Only if the work is genuinely cluster-wide, disarm instead: ha-manager crm-command disarm-ha freeze. There is no ha-manager disable or ha-manager enable subcommand
  4. 4Verify the state changed rather than assuming it: ha-manager status | grep -iE "fencing|watchdog|disarm" must print a line containing disarming or disarmed. Empty output means the command did not do what you think
  5. 5Migrate VMs off the node. Set VMID and TARGET first, then: qm migrate "$VMID" "$TARGET" --online (repeat for each VM)
  6. 6Verify the node has no running guests: pvesh get /nodes/NODENAME/qemu --output-format json
  7. 7Begin maintenance (patching, hardware work)
  8. 8After maintenance, re-arm: ha-manager crm-command arm-ha (or node-maintenance disable NODENAME if you used maintenance mode)
  9. 9Verify HA is armed again: ha-manager status shows the watchdog armed, not disarmed

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓After maintenance, all nodes are reachable
  • ✓HA is armed cluster-wide
  • ✓Previously migrated VMs are still running

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If maintenance goes wrong, restore from PBS
  • ↶If HA re-arm fails, manually enable per-resource

6 · Escalation

When the runbook isn't enough, contact:

  • · Cluster lead if maintenance exceeds window
  • · Vendor support if hardware issues found

Put a node into maintenance

This runbook covers taking a node out of service for planned maintenance.

When to use

  • Firmware updates
  • Hardware replacement
  • Kernel upgrades
  • Cluster rebalancing

Prerequisites

  • Maintenance window approved by stakeholders
  • Cluster has N+1 capacity (surviving nodes can host evacuated VMs)
  • OOB management accessible (iDRAC/iLO/IPMI) in case of network issues
  • All VMs on the node have a valid migration target

Pre-checks

pvesh get /cluster/resources --output-format json | jq -r '.data[] | select(.node=="<node>") | .vmid'
# List VMs on the node
ha-manager status
# Confirm HA state

Procedure

Step 1: Communicate

Notify stakeholders:

  • Application owners (impact on their workloads).
  • Support teams (alerting may fire during the operation).
  • Leadership (high-risk changes).

Step 2: Take HA out of the way

Two different operations, and the cheaper one is usually correct.

Work confined to one node — put that node into maintenance. HA migrates its managed guests away and keeps fencing armed everywhere else, so the rest of the cluster stays protected while you work:

NODE=pve-02
ha-manager crm-command node-maintenance enable "$NODE"

Work that is genuinely cluster-wide — disarm, which releases every watchdog in the cluster. There is no automatic failover anywhere while this window is open, so open it deliberately and close it explicitly:

# freeze: leave HA guests running and untouched.
# ignore: release HA's claim on them entirely.
ha-manager crm-command disarm-ha freeze

Wait for the LRMs to release their watchdogs, then verify that the state actually changed:

ha-manager status | grep -iE 'fencing|watchdog|disarm'
# Expect a line reporting the watchdog as `disarming` or `disarmed`.

Step 3: Migrate VMs

For each VM on the node:

# Substitute your own values before running:
VMID=100
TARGET_NODE=pve-02

qm migrate "$VMID" "$TARGET_NODE" --online

For VMs that cannot live-migrate:

# Substitute your own values before running:
VMID=100
TARGET_NODE=pve-02

qm shutdown "$VMID" --timeout 60
qm migrate "$VMID" "$TARGET_NODE"
qm start "$VMID"

Step 4: Verify the node is empty

# Substitute the node you are draining:
NODE=pve-01

qm list --node "$NODE"
# Should be empty

Step 5: Perform maintenance

Apply patches, reboot, replace hardware, etc.

Step 6: Re-arm HA

After maintenance, close the window explicitly — whichever one you opened in Step 2:

# If you used per-node maintenance mode:
NODE=pve-02
ha-manager crm-command node-maintenance disable "$NODE"

# If you disarmed cluster-wide:
ha-manager crm-command arm-ha

Confirm the cluster is protected again before you consider the window closed:

ha-manager status | grep -iE 'fencing|watchdog|disarm'
# Expect the watchdog reported as `armed`, and no node still `disarmed`.

Step 7: Monitor

Watch for issues for at least 30 minutes. Verify previously migrated VMs are reachable.

Verification checklist

  • All VMs are reachable
  • HA is armed cluster-wide
  • Cluster has quorum
  • No new health warnings
  • No unexpected reboots in monitoring

Rollback

If maintenance goes wrong:

# Substitute your own values before running. The archive name comes from
# `pvesm list pbs-main`:
VMID=100
ARCHIVE=pbs-main:backup/vzdump-qemu-100-2026_08_01-00_00_00.vma.zst
STORAGE=local-zfs

# Re-arm HA if needed
ha-manager crm-command arm-ha

# Restore any VMs that failed to migrate
qmrestore "$ARCHIVE" "$VMID" --storage "$STORAGE"

Escalation

  • Cluster lead: if maintenance exceeds the window
  • Vendor support: if hardware issues are found
  • Security team: if OOB credentials need to be used

References

  1. Proxmox VE — HA disarm