Maintenance on a hyper-converged node

Why this matters in production

Both layers must be prepared, in the right order, and the mistakes produce either unnecessary data movement or stopped VMs.

Readiness check

# the cluster must be healthy before starting
ceph -s
ceph health detail

# and able to absorb the node
ceph osd df tree | grep -A15 "$(hostname -s)"
pvecm status
ha-manager status

Do not start if:
  cluster health is not HEALTH_OK
  any PG is degraded or undersized
  another node is already out
  the surviving nodes cannot run this node's VMs

The sequence

# the node that will host the VMs during the maintenance:
TARGET_NODE=pve-02

# 1. move the VMs off first
for v in $(qm list | awk 'NR>1 && $3=="running" {print $1}'); do
  qm migrate "$v" "$TARGET_NODE" --online
done

# 2. verify no VMs remain
qm list
ha-manager status | grep "$(hostname -s)"

# 3. prepare Ceph — noout for a brief outage
ceph osd set noout
# or, for a longer one, enter maintenance mode
ceph orch host maintenance enter "$(hostname -s)"

# 4. stop the OSDs cleanly
systemctl stop ceph-osd.target
ceph -s

# 5. do the work, then reverse

VMs first, then Ceph. Stopping the OSDs while VMs are still running on the node means those VMs lose their storage path momentarily and may see I/O errors.

Choosing flags versus maintenance mode

ceph orch host maintenance enter pve-02
# ... work ...
ceph orch host maintenance exit pve-02

Maintenance mode handles the flags and daemon stops together and records the host’s state, which is why it is preferable for anything beyond a reboot.

Returning to service

# 1. bring Ceph back
ceph orch host maintenance exit "$(hostname -s)"
# or
systemctl start ceph-osd.target
ceph osd unset noout

# 2. wait for health before returning VMs
watch -n 30 'ceph -s | grep -E "health|pgs:"'

# 3. return the VMs
VMIDS=vmids
for v in ${VMIDS}; do qm migrate "$v" "$(hostname -s)" --online; done

Waiting for HEALTH_OK before migrating VMs back means the node is not serving guest I/O while its OSDs are still catching up.

Quiz

Production discipline

Migrate VMs off before stopping OSDs and wait for HEALTH_OK before migrating them back — both orderings avoid the node serving guest I/O while its storage path is unstable. Use ceph orch host maintenance for anything longer than a reboot; it handles the flags and daemons together and records the state.

Cross-course references

• Kubernetes: draining a node before touching its storage daemons is the same order
• Linux: quiescing consumers before stopping a service is standard practice