Coordinating Proxmox and Ceph maintenance

Why this matters in production

Work touching both layers has an ordering that matters, and the failure modes of getting it wrong are self-inflicted incidents during planned work.

The ordering rules

1. Never begin from anything other than HEALTH_OK
2. Move compute off before touching storage on a node
3. One node at a time
4. Wait for health between nodes, not just between steps
5. Return compute only after storage is healthy
6. Have a stop condition and use it

Rule 4 is the one most often skipped: proceeding to the next node when the previous one’s OSDs are still catching up compounds the reduction.

A worked sequence

# readiness
ceph -s | grep -E 'health|pgs:'
pvecm status
ha-manager status

# per node
NODE=pve-02

# 1. compute off
for v in $(qm list | awk 'NR>1 && $3=="running" {print $1}'); do
  qm migrate "$v" pve-03 --online
done

# 2. verify empty
qm list | awk 'NR>1 && $3=="running"' | wc -l

# 3. storage into maintenance
ceph orch host maintenance enter "$NODE"

# 4. do the work

# 5. storage back
ceph orch host maintenance exit "$NODE"

# 6. wait for health — not just for the daemons to start
while ! ceph health | grep -q HEALTH_OK; do sleep 30; done

# 7. compute back

Verifying readiness at each step

# a guard that prevents proceeding early
ceph health | grep -q HEALTH_OK || { echo "NOT READY"; exit 1; }

Scripting the guard is what makes the rule effective rather than aspirational.

When maintenance overruns

Planned: 30 minutes. Actual: approaching 2 hours.
  the OSDs have been down long enough to be marked out
  recovery has started and is moving that node's data
  returning the node now means backfilling it back

# check what happened
ceph -s | grep -E 'misplaced|degraded'
ceph osd tree | grep -A3 pve-02

The mistake is returning a node mid-recovery, which produces movement in both directions simultaneously.

Quiz

Production discipline

Script the HEALTH_OK guard between nodes rather than relying on judgement — proceeding while the previous node’s OSDs are still catching up compounds the reduction. If a window overruns past the down-out interval, let the recovery finish before returning the node; a partial reversal moves far more data.

Cross-course references

• Kubernetes: rolling updates wait for readiness between nodes for the same reason
• Linux: sequential maintenance on redundant systems always requires a health gate