Hyper-converged Proxmox and Ceph

Why this matters in production

Hyper-convergence removes the cost of separate storage hosts and replaces it with a resource contention problem. Both parts are real.

The architecture

flowchart TD
  subgraph Node1[Node 1]
    V1[VMs] --- O1[OSDs]
  end
  subgraph Node2[Node 2]
    V2[VMs] --- O2[OSDs]
  end
  subgraph Node3[Node 3]
    V3[VMs] --- O3[OSDs]
  end
  O1 <--> O2
  O2 <--> O3
  O1 <--> O3

Every node runs both hypervisor and storage. A VM’s I/O may be served by an OSD on its own host or on any other.

pveceph install
pveceph init --network 10.0.1.0/24
pveceph mon create
pveceph osd create /dev/sdb

Configuring resource separation

# bound OSD memory so it does not compete with VMs
ceph config set osd osd_memory_target 4294967296     # 4 GiB per OSD

Node with 128 GiB RAM, 6 OSDs:
  OSDs:            6 × 4 GiB  = 24 GiB
  Host OS:                    =  8 GiB
  Available for VMs:          = 96 GiB

Over-committing VM memory against this leaves the OSDs to be squeezed or the host to swap, and an OSD host that swaps performs catastrophically.

# CPU: reserve cores for OSDs where the platform supports it
systemctl set-property ceph-osd@0.service AllowedCPUs=0-3

Proxmox VMs can be pinned away from the cores serving OSDs, which prevents a busy VM starving the storage layer.

The operational consequences

Deciding whether it suits

Quiz

Production discipline

Subtract the OSD memory commitment — osd_memory_target times the OSD count — before allocating any VM memory; an OSD that swaps degrades storage for the entire cluster, not just its own node. Pin VMs away from the cores serving OSDs where the platform allows it.

Cross-course references

• Kubernetes: running storage and workloads on the same nodes has identical coupling
• Linux: any co-located service needs explicit resource reservation