Cold-starting a cluster from a full shutdown

Most cluster procedures assume a cluster that is running. Cold start is the one that assumes nothing: every node dark, /etc/pve unreachable because there is no pmxcfs to serve it, no quorum because there is nobody to vote, and a room full of hardware that is about to come back in an order nobody chose.

It happens for two reasons, and they need different preparation:

• Planned. A UPS replacement, a generator test, an electrical inspection, a data-centre move, a chilled-water outage announced two weeks ahead.
• Unplanned. The power failed and the UPS ran out.

The planned case is the one to get right, because a cluster that has been shut down cleanly comes back cleanly. The unplanned case is survivable largely to the extent that you had already rehearsed the planned one.

The fact that shapes everything

A cluster is not quorate when all nodes are offline. That is a tautology, but its consequences are not obvious until you are standing in front of it:

• On boot, pve-cluster starts and pmxcfs mounts /etc/pve read-only, because a single node in a three-node cluster has one vote out of three.
• pve-guests, the unit responsible for autostarting guests, waits for quorum before it starts anything. Guests with onboot: 1 do not start on the first node, or the second — they start when the majority is present.
• The HA stack behaves the same way. The local resource manager cannot reset its watchdog without quorum, and the cluster resource manager makes no placement decisions until the cluster is quorate.

So the first node up looks broken. It is not broken. It is waiting, exactly as designed, and the correct response is to bring up the next node.

Planned shutdown: the order that makes the restart boring

Shutting down is where the work happens. Everything you do here is something you do not have to reason about at 04:00 on the way back up.

set -euo pipefail

# What is HA currently managing? Capture it - this is your restart checklist.
ha-manager status > /root/cold-start-ha-state.txt
ha-manager config >> /root/cold-start-ha-state.txt
cat /root/cold-start-ha-state.txt

# Capture the full guest inventory and its run state, on every node.
# This is what you will verify against tomorrow. Take it now, while it is true.
for node in $(pvecm nodes | awk 'NR>3 {print $3}'); do
echo "== $node"
ssh "$node" 'qm list; pct list'
done > /root/cold-start-inventory.txt

# Disable replication jobs so none fire mid-shutdown.
pvesr list

set -euo pipefail

# The startup option controls both directions. order is the sequence,
# up is the delay after starting before the next guest, down is how long
# to wait for this guest to shut down before moving on.
# Lower order numbers start FIRST and shut down LAST.
#
#   qm set 101 --startup order=1,up=30,down=120   # domain controller / DNS
#   qm set 110 --startup order=2,up=15            # database
#   qm set 120 --startup order=3                  # application servers
#
# If ordering is unset across the estate, drive the shutdown manually,
# most-dependent first:
VMID=120
qm shutdown "$VMID" --timeout 300

# A guest that will not shut down cleanly within its timeout is a decision,
# not an automatic stop. Find out why before you reach for qm stop.
qm status "$VMID"

set -euo pipefail

# Only on a cluster running Ceph. Confirm health BEFORE you set flags -
# shutting down a cluster that is already degraded is a different procedure.
ceph -s

# Stop Ceph reacting to OSDs that are about to disappear on purpose.
ceph osd set noout
ceph osd set norebalance
ceph osd set norecover

# Confirm the flags are set before proceeding.
ceph osd stat

set -euo pipefail

# Shut down nodes one at a time. On each, confirm no guests are running
# before pulling it out from under anything.
qm list
pct list

# Then, on that node:
shutdown -h now

# Keep the last node with quorum until the end. On a three-node cluster you
# will lose quorum when the second node goes down; that is expected and
# harmless once every guest is already stopped.

Bringing it back up

The ordering is dictated by dependencies, not by convenience. Each layer needs the one below it.

set -euo pipefail

# Poll until quorate. If it never becomes quorate, the answer is on the
# nodes that have not appeared, not on this one.
pvecm status

# Which nodes are present at the corosync layer right now?
corosync-cfgtool -n

# Count what you have against what you need. Expected votes comes from
# corosync.conf; total votes is what is actually here.
pvecm status | grep -E 'Quorate|Total votes|Expected votes|Activity blocked'

set -euo pipefail

# Only once every OSD you expect is up and in.
ceph osd stat
ceph -s

ceph osd unset norecover
ceph osd unset norebalance
ceph osd unset noout

# Watch recovery settle rather than assuming it does.
ceph -s

Verifying against an inventory, not a dashboard

The dashboard shows what is running. It cannot show what is missing, because a guest that never started does not draw attention to itself. The inventory you captured in phase 1 is what closes that gap.

set -euo pipefail

# Regenerate the same view you captured before the shutdown.
for node in $(pvecm nodes | awk 'NR>3 {print $3}'); do
echo "== $node"
ssh "$node" 'qm list; pct list'
done > /root/cold-start-inventory-after.txt

diff /root/cold-start-inventory.txt /root/cold-start-inventory-after.txt || true

# Guests configured to autostart that are nevertheless stopped.
for vmid in $(qm list | awk 'NR>1 {print $1}'); do
onboot=$(qm config "$vmid" | awk '/^onboot:/ {print $2}')
state=$(qm status "$vmid" | awk '{print $2}')
if [ "$onboot" = "1" ] && [ "$state" != "running" ]; then
  printf '%-6s onboot-but-stopped\n' "$vmid"
fi
done

# HA should be back to the state you captured, not to a state full of errors.
ha-manager status

A break/fix exercise

Common mistakes

• Forcing quorum because the nodes are slow to boot. Boot takes minutes; a modern server with a lot of RAM can spend five of them in POST alone. Waiting is free.
• Leaving noout set after the restart. It is a maintenance flag, not a configuration setting.
• Never having captured the inventory. Without a before list, “everything is back” is an assertion, not a finding.
• Assuming HA will sort it out. HA cannot act until the cluster is quorate, which is the exact condition you are trying to reach.
• Powering on the whole rack at once after a long outage. Inrush current on a rack full of servers spinning up disks simultaneously can trip the breaker you just got back. Stagger it.
• Treating cold start as a procedure you can write during the incident. It is the one procedure that has to exist on paper, off-cluster, because the wiki that documents it may be running on the cluster.

Key takeaways

• All nodes down means no quorum by definition; the first node up will look broken and is not.
• The dangerous state is a partially powered cluster, and the dangerous action is forcing quorum inside it.
• Order on the way down: HA frozen, guests stopped by dependency, Ceph flagged, nodes off. Order on the way up: network, storage, quorum, Ceph, storage verified, guests, HA.
• startup order on the guests is the durable record of your dependency graph. Set it before you need it.
• Verify against the inventory you captured, not against the dashboard.

Knowledge check