Network design for Proxmox with Ceph

Advanced⏱ ~17 mincephiperf3corosync-cfgtool

What you'll learn

Enumerate the networks in a Proxmox and Ceph deployment
Size each appropriately
Recognise the consequences of insufficient separation
Verify the design before production

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Four distinct traffic types with different requirements share the same physical infrastructure. The failure modes of merging them are specific and severe.

The networks

Network	Carries	Latency sensitivity	Bandwidth
Corosync	Proxmox cluster membership	critical	negligible
Ceph public	client to OSD	high	high
Ceph cluster	OSD to OSD replication and recovery	moderate	very high
VM network	guest traffic	per workload	per workload
Management	administration	low	low

Corosync is the outlier: almost no bandwidth, and the most sensitive to delay of any of them.

Sizing

Ceph public:  aggregate VM I/O × peak factor
Ceph cluster: public × (size - 1) for writes, plus recovery

100 VMs averaging 20 MB/s, peak factor 3:
  public  ≈ 6 GB/s ≈ 48 Gb/s → 2× 25 GbE minimum
  cluster ≈ public × 2 for size=3 → 4× 25 GbE, or 100 GbE

The cluster network carries more than the public network on a replicated pool, which is the sizing mistake most often made in the other direction.

# measure actual usage
ceph -s | grep client
sar -n DEV 1 10 | grep -E 'ens|bond'

Consequences of insufficient separation

Merged	Consequence
Corosync with storage	recovery bursts cause node fencing
Corosync with VM traffic	a busy guest causes node fencing
Ceph public with cluster	recovery starves client I/O
Ceph with VM traffic	guest traffic affects storage latency
Everything on one link	any event affects everything

The corosync failure mode in detail:
  recovery saturates the shared link
  → corosync token delayed beyond its timeout
  → the node is considered lost by the others
  → it is fenced, its VMs are stopped
  → those VMs restart elsewhere, adding load
  → the recovery continues, delaying corosync again

That loop has taken down entire Proxmox clusters during a routine disk replacement.

Verifying before production

# corosync latency under storage load
corosync-cfgtool -s
# start a heavy recovery or benchmark, then:
corosync-cfgtool -s     # check for retransmits and link status
journalctl -u corosync --since '10 min ago' | grep -i retransmit

# storage path throughput and MTU
# storage-network address of a Ceph node:
CEPH_NODE=192.0.2.11

for m in $(grep monhost /etc/pve/storage.cfg | tr ',' ' '); do
  ping -M do -s 8972 -c 2 -W 2 "$m" >/dev/null 2>&1 && echo "$m MTU OK"
done
iperf3 -c "$CEPH_NODE" -t 30 -P 8

Running a full recovery while watching corosync is the test that matters, and it must be done before production rather than discovered during one.

Quiz

Knowledge check · 4 questions

Q1. Why does the Ceph cluster network carry more traffic than the public network?

It carries monitor traffic as well as replication, and each OSD map change is broadcast to every OSD over that network Recovery and backfill run over the cluster network alone, and recovery is the only traffic either network carries in bulk Each client write arrives once on the public network and is then sent to each replica over the cluster network The public network compresses client payloads, so the same data occupies less on the wire than it does between the OSDs
Q2. Corosync needs high bandwidth because it carries cluster state.

True False
Q3. Investigate node fencing during storage maintenance.

A Proxmox cluster fenced two nodes during a routine disk replacement in the Ceph cluster. VMs were stopped and restarted elsewhere. Corosync and storage share a network.
Q4. What is the test that must be run before a Proxmox and Ceph deployment goes into production?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Size the Ceph cluster network larger than the public network — it carries size - 1 copies of every write plus all recovery traffic. Run a full recovery while monitoring corosync before going into production; if tokens are delayed under storage load, the first real recovery will fence nodes.

Cross-course references

Kubernetes: control plane traffic separated from data plane for the same reason
Linux: cluster heartbeats are always isolated from bulk data paths