A dedicated Ceph cluster for Proxmox

Intermediate⏱ ~17 mincephpvesm

What you'll learn

Describe the dedicated architecture
Compare it against hyper-converged on specific criteria
Configure Proxmox against an external cluster
Plan the network between them

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Separating the layers removes the contention and the coupling, at the cost of dedicated hardware and a network between them. For most production deployments the trade is favourable.

The architecture

flowchart TD
  subgraph PVE[Proxmox nodes]
    V1[VMs] & V2[VMs] & V3[VMs]
  end
  subgraph CEPH[Ceph nodes]
    O1[OSDs] & O2[OSDs] & O3[OSDs]
  end
  PVE -->|storage network| CEPH
  O1 <-->|cluster network| O2
  O2 <--> O3

Each layer scales independently, is maintained independently, and fails independently.

The comparison

Criterion	Hyper-converged	Dedicated
Hardware cost	lower	higher
Resource contention	present	absent
Failure coupling	node loses both	independent
Maintenance coordination	both layers together	independent
Scaling flexibility	fixed ratio	independent
Network requirement	internal	a storage network
Operational complexity	one cluster to run	two
Latency predictability	variable	consistent

Configuring Proxmox against an external cluster

# on the Ceph cluster
ceph auth get-or-create client.pve \
  mon 'profile rbd' \
  osd 'profile rbd pool=pve-vms' \
  mgr 'profile rbd pool=pve-vms' > /tmp/pve.keyring

# on each Proxmox node
mkdir -p /etc/pve/priv/ceph
cp /tmp/pve.keyring /etc/pve/priv/ceph/ceph-vms.keyring
chmod 600 /etc/pve/priv/ceph/ceph-vms.keyring

pvesm add rbd ceph-vms \
  --pool pve-vms \
  --monhost '10.0.2.11,10.0.2.12,10.0.2.13' \
  --username pve \
  --content images,rootdir

/etc/pve/ is replicated across the Proxmox cluster, so the keyring placed on one node appears on all.

Planning the network

Network	Carries	Sizing
Proxmox cluster network	corosync	low bandwidth, low latency critical
VM network	guest traffic	per workload
Storage network	Proxmox to Ceph	sized for aggregate VM I/O
Ceph public network	client to OSD	same as the storage network
Ceph cluster network	OSD to OSD	sized for replication and recovery

# on the Ceph side
ceph config set global public_network 10.0.2.0/24
ceph config set global cluster_network 10.0.3.0/24

The storage network is the Ceph public network from Ceph’s perspective; naming it separately in the Proxmox context avoids confusion about which network the hypervisors use.

# verify path and MTU from each Proxmox node
for m in 10.0.2.11 10.0.2.12 10.0.2.13; do
  ping -M do -s 8972 -c 2 -W 2 "$m" >/dev/null 2>&1 && echo "$m OK" || echo "$m FAIL"
done

Quiz

Knowledge check · 4 questions

Q1. Why should corosync traffic be kept off the storage network?

Corosync and Ceph use incompatible protocols on one segment, and a switch cannot prioritise both of them correctly Corosync is sensitive to latency and jitter, and storage recovery bursts can delay its packets enough to trigger node fencing Corosync requires multicast, which storage switches commonly disable, so the membership traffic is dropped silently Bandwidth contention would slow storage, since corosync sends a full copy of the cluster configuration on every token round trip
Q2. Losing a hyper-converged node makes the cluster reschedule workloads and recover data at the same moment, on the capacity that remains.

True False
Q3. Plan networks for a dedicated deployment.

A Proxmox cluster will connect to a dedicated Ceph cluster. The team plans to use one 25 GbE network for everything to simplify the design.
Q4. Why does `/etc/pve/priv/ceph/` only need the keyring placed on one node?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Keep corosync off the storage network — a recovery burst that delays its packets can fence nodes, turning a storage rebalance into a compute outage. Weigh the dedicated architecture on failure independence rather than on contention; contention is manageable and coupled failures are not.

Cross-course references

Kubernetes: separating control plane and storage traffic follows the same reasoning
Linux: cluster heartbeat networks are always separated from bulk traffic