medium riskservice affecting~60 min
Evacuate a node for planned maintenance
1 · Prerequisites
Confirm every item is in place before any state change.
- A cluster with at least 2 nodes
- Shared storage (Ceph, NFS, or iSCSI) accessible from all nodes
- HA configured if you want automatic recovery
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · Verify no live-migration jobs are currently running
- · Confirm the destination nodes have capacity for the migrating VMs
- · Verify shared storage is reachable from all candidate destinations
- · Confirm HA-managed VMs will relocate on failure (ha-manager status)
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1List VMs on the target node: qm list | grep <node>
- 2Migrate each VM to a destination node: qm migrate <vmid> <destination> --online (if the VM is running) or --with-local-disks false
- 3For LXC: pct migrate <ctid> <destination> --online
- 4Verify the node is empty of workloads: qm list | grep <node> should return nothing
- 5Stop corosync on the node: systemctl stop pve-cluster corosync
- 6Or, for shorter windows, set the node to maintenance in the GUI: Datacenter → HA → <node> → Maintenance
- 7Perform the maintenance (firmware update, hardware swap, etc.)
- 8Bring the node back: systemctl start pve-cluster corosync (or remove from maintenance)
- 9Verify: pvecm status, ha-manager status
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓All workloads have been migrated off the node
- ✓qm list on the target node returns no workloads
- ✓After maintenance, node rejoins the cluster and pvecm status shows it as online
- ✓HA resources can run on the node again
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶If a VM migration fails, check shared storage reachability and retry
- ↶If the node cannot rejoin the cluster, do not force it — investigate the failure first
6 · Escalation
When the runbook isn't enough, contact:
- · Cluster admin if corosync refuses to start after maintenance
- · Storage admin if shared storage is unreachable
Evacuate a node for planned maintenance
The goal is zero-downtime maintenance. Live migration requires shared storage; without it, the VM must be stopped and started on the new node.
Two modes
- Drain (with HA): set the node to maintenance in the HA panel. HA will migrate all managed VMs off automatically. Wait until the cluster dashboard shows no resources on the node.
- Manual drain: migrate each VM one by one. Use this when you want control over the order, or when VMs are not HA-managed.
Common mistakes
- Forgetting to migrate local-storage VMs (they will not migrate to a node that can’t see the storage)
- Leaving the node in maintenance mode after the work is done
- Not verifying that HA-managed resources have actually relocated before stopping services on the node