Scaling a hyper-converged cluster

Intermediate⏱ ~17 mincephpvecmpveceph

What you'll learn

Describe what adding a node changes
Sequence a node addition safely
Recognise the scaling limits
Plan growth that does not waste capacity

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Adding a node adds compute and storage together, which is convenient when both are needed and wasteful when only one is.

What a node addition changes

Layer	Effect
Compute	more CPU and memory for VMs
Storage capacity	more raw capacity
Storage performance	more devices, more parallelism
Failure absorption	1/N falls, improving efficiency
Failure domain count	more hosts, better CRUSH placement
Recovery	a rebalance of roughly the new node’s share

The failure absorption improvement is often overlooked: going from four nodes to five reduces the reserve from 25% to 20% of usable capacity, which frees capacity in addition to what the node adds.

Sequencing the addition

# 1. join the Proxmox cluster
EXISTING_NODE_IP=10.20.0.11
pvecm add ${EXISTING_NODE_IP}
pvecm status

# 2. verify networking before adding storage
CEPH_NODE=stor-04
ping -M do -s 8972 -c 2 ${CEPH_NODE}
iperf3 -c ${CEPH_NODE} -t 10

# 3. install Ceph and add the host
pveceph install
ceph orch host add pve-05 10.0.1.15
ceph orch host label add pve-05 osd

# 4. add OSDs, ideally all at once
ceph orch daemon add osd pve-05:/dev/sdb
ceph orch daemon add osd pve-05:/dev/sdc
# ...

# 5. watch the rebalance
watch -n 30 'ceph -s | grep misplaced'

Adding all the node’s OSDs together produces one rebalance rather than one per device.

The scaling limits

Node count	Characteristic
3	minimum; no recovery from a host failure with `size=3`
4–6	works well; absorption reserve is large
7–12	the sweet spot
13–20	contention management becomes significant work
20+	dedicated is usually better

The limits come from:
  coordination cost of maintenance across many coupled nodes
  contention management growing with density
  the fixed compute-to-storage ratio diverging from demand
  failure domain design becoming harder with mixed roles

Growth that does not waste capacity

Situation	Approach
Both needed	add a balanced node
Compute only	add a compute-only Proxmox node, no OSDs
Storage only	add drives to existing nodes, or a storage-only node
Neither balanced	consider migrating to dedicated

# a compute-only node: join Proxmox, do not add OSDs
IP=10.20.0.11
pvecm add ${IP}
# do not run pveceph osd create

Mixed node roles within one cluster are entirely supported and are what prevents the fixed-ratio waste.

Quiz

Knowledge check · 4 questions

Q1. Why does adding a node improve usable capacity by more than the drives it contains?

The host failure absorption reserve is roughly 1/N, so increasing N releases previously reserved capacity Ceph compresses data more efficiently once more OSDs are present, since BlueStore assembles larger compression blocks More PGs are created as the cluster grows, and the finer placement grid packs objects with less wasted space The balancer moves existing objects onto the new host, which frees the space their previous copies occupied
Q2. Every node added to a hyper-converged cluster must run both VMs and OSDs.

True False
Q3. Plan growth for a hyper-converged cluster.

A six-node hyper-converged cluster is at 45% storage utilisation and 85% CPU utilisation. VM demand is growing steadily; storage demand is flat.
Q4. At what node count does hyper-convergence stop scaling well, and why?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Add nodes with only the role that is needed — a Proxmox cluster supports compute-only nodes and Ceph hosts without VMs, and always adding balanced nodes is what makes hyper-convergence expensive at scale. Add all of a new node’s OSDs together so the addition produces one rebalance.

Cross-course references

Kubernetes: node pools with distinct roles avoid the same fixed-ratio waste
Linux: heterogeneous cluster membership is normal and often more efficient