Rolling maintenance: node evacuation and HA workflows

Why this matters in production

Taking a node down for maintenance without following the right procedure causes HA failovers or service interruptions. The maintenance workflow protects the cluster from the maintenance itself.

The maintenance workflow

flowchart LR
  A[Decide maintenance] --> B[Communicate]
  B --> C[Disarm HA]
  C --> D[Migrate VMs]
  D --> E[Verify cluster health]
  E --> F[Begin maintenance]
  F --> G[Verify node health]
  G --> H[Re-arm HA]
  H --> I[Monitor for issues]

Step 1: Communicate

Notify stakeholders: application owners, support teams, leadership.

Step 2: Take HA out of the way

For a single node, maintenance mode is the right tool: it migrates that node’s HA guests away and leaves fencing armed on every other node.

NODE=pve-02
ha-manager crm-command node-maintenance enable "$NODE"

Reach for the cluster-wide disarm only when the work affects the whole cluster, because it releases every watchdog and there is no automatic failover anywhere until you re-arm:

ha-manager crm-command disarm-ha freeze

Note that ha-manager has no disable, enable or watchdog subcommand. ha-manager disable --scope cluster is a usage error, not a disarmed cluster — and the error is easy to miss, because the verification below then greps for a state that was never entered.

Wait for all LRMs to release their watchdogs. Verify with:

ha-manager status | grep -i fencing

Step 3: Migrate VMs off the node

GUI: right-click VMs → Migrate → Online migration. CLI:

for vmid in $(qm list | awk '/running/ {print $1}'); do qm migrate $vmid pve-02 --online; done

Step 4: Verify cluster health

qm list | grep -E 'running|stopped' | head && pvecm status

Step 5: Begin maintenance

Apply updates, reboot, hardware work — whatever the maintenance requires.

Step 6: Verify node health post-maintenance

pvecm status && pvesh get /nodes/pve-01/status && ceph -s 2>/dev/null || true

Step 7: Re-arm HA

Close whichever window you opened in Step 2:

# per-node maintenance mode
NODE=pve-02
ha-manager crm-command node-maintenance disable "$NODE"

# or, if you disarmed cluster-wide
ha-manager crm-command arm-ha

Step 8: Monitor

Watch for issues for at least 30 minutes after maintenance. Verify that previously-running VMs are reachable.

Production considerations

Common mistakes

• Forgetting to re-arm HA.
• Migrating VMs to a node that doesn’t have capacity.
• Leaving a maintenance mode active overnight.

Key takeaways

• Communicate, disarm, migrate, verify, maintain, re-arm, monitor.
• Verify capacity before starting.
• Don’t forget to re-arm HA.

Knowledge check