Cluster expansion lab

This lab adds a 4th node to a 3-node cluster and migrates VMs to it.

Steps

1. Prepare the new host

# On the new host (pve-04)
apt update && apt full-upgrade -y
# Make sure the version matches the existing cluster
pveversion -v

2. Add the host to /etc/hosts on every cluster node

On all 4 nodes (and the cluster hosts file):

cat >> /etc/hosts << 'EOF'
10.0.0.14  pve-04.lab.local  pve-04
EOF

3. Join the cluster from the new host

# On pve-04
pvecm add 10.0.0.11
# Provide the root password when prompted

This joins the new host to the cluster, syncs the cluster config, and starts corosync.

4. Verify the join

# From any node
pvecm status
# Expected: 4 nodes, quorum, all online

pvecm nodes
# Expected: pve-01 pve-02 pve-03 pve-04 listed

5. Configure storage on the new node

The new host needs to access the shared storage (Ceph, NFS, or iSCSI):

# For Ceph: it joins automatically
ceph -s
# Expected: new host appears as an OSD-host candidate

# For NFS: mount it
mount -t nfs <nfs-server>:/srv/nfs-vm /mnt/nfs-vm
# Or use pvesm to add it

6. Migrate a VM to the new host

qm migrate 100 pve-04 --online

Watch progress:

pvesh get /cluster/resources --type vm | jq '.[] | select(.vmid==100) | {vmid, node, status}'

7. Verify HA (if HA is enabled)

If the migrated VM is HA-managed:

ha-manager status
# Expected: VM 100 listed, state "started" on pve-04

If HA groups have node priorities, the VM may move back to the preferred node. Override with pve-node pinning or HA group config.

8. Test failover across all 4 nodes

Run a quick HA failover test:

# From a different node than the one running VM 100
ssh root@pve-01 'echo c > /proc/sysrq-trigger'
# Wait 90 seconds; VM should be running on another node
qm status 100

Verification

• 4-node cluster has quorum
• Storage is accessible from the new node
• Live migration to the new node completes cleanly
• HA failover works with the larger cluster

Notes

• For 4 nodes with default corosync settings, quorum is still

  50% (3 of 4 votes needed). No qdevice required.

• For 5 nodes, qdevice is optional but recommended for stretch clusters.
• Adding nodes is always safer than removing them. Removal changes quorum math; do it during a maintenance window.

Cleanup

To remove the new node:

# On pve-04: leave the cluster
pvecm leave

# On pve-01: remove from cluster config (if necessary)
pvecm delnode pve-04

This is destructive — only do it if you’re decommissioning pve-04.