Ceph hardware and network sizing

Why this matters in production

Ceph is not free. Every OSD consumes CPU, RAM, network, and storage. A cluster sized for the workload, not for the marketing brochure, is the difference between a smooth Ceph deployment and a constant operational struggle.

The four resources

CPU

Each OSD process needs CPU. Modern enterprise NVMe SSDs can saturate multiple cores.

Plus: 1 core for MON, 1 core for MGR, 1 core for MDS (if using CephFS).

For a 4-node cluster with 6 NVMe OSDs per node: ~24 OSD cores + 3 MON + 3 MGR + spare = ~32 cores reserved for Ceph.

Memory

Each OSD process requires 4 GB by default; Ceph recommends 8 GB for production. Each MON and MGR consumes ~1–2 GB.

For 24 OSDs + 3 MON + 3 MGR:

24 OSDs × 8 GB = 192 GB
 3 MON × 2 GB = 6 GB
 3 MGR × 2 GB = 6 GB
 Ceph buffers = ~10 GB
 Total Ceph RAM ≈ 214 GB

Plus VM RAM. A 256 GB node running 24 OSDs and meaningful VM workloads is tight.

Network

The Ceph network has two channels:

In a hyperconverged cluster, the cluster network is critical: every write replicates across it. Insufficient cluster bandwidth means slow writes.

flowchart LR
  C[Client I/O] --> P[Public network 10 GbE]
  P --> O[OSD]
  O --> R[Cluster network 25 GbE]
  R --> O2[OSD peer]
  O2 --> R
  R --> O3[OSD peer]

Storage

Disks: enterprise SSDs with PLP. SATA SSDs acceptable for bulk; NVMe required for high-IOPS VM workloads.

DB/WAL: a small, fast NVMe device per OSD (or shared across a few OSDs) for the BlueStore metadata. This dramatically improves small random write performance.

Network reference architecture

For a 3-node hyperconverged cluster:

Five NICs is heavy but matches the workload classes cleanly. Smaller deployments combine NICs (e.g. 3 + 4 as one LACP bond) at the cost of some isolation.

Capacity planning

Three numbers to plan:

Raw capacity  = N OSDs × size per OSD
Usable (rep)  = raw × (1/replication factor) × (1 - overhead)

For 12 × 4 TB SSDs with size=3 replication:

Raw = 48 TB
Usable = 48 / 3 ≈ 16 TB
Reserve 30 % for nearfull headroom ≈ 11 TB practical

Keep capacity utilisation below 70 % (nearfull_ratio=0.85). Beyond that, backfilling during recovery blocks.

The number the capacity plan usually gets wrong

Dividing raw by the replication factor gives you the capacity of a cluster with nothing wrong with it. That is not the number you can fill.

When a node fails, its placement groups are rebuilt on the surviving nodes. On a three-node cluster that means two nodes must absorb the third node’s share — so the cluster has to hold all its data on two thirds of its OSDs before it can return to active+clean.

ceph df
ceph osd df tree
ceph osd tree

Worked through for a three-node cluster with 4 × 4 TB per node, size 3:

Raw                       48 TB
Usable at size 3          16 TB
One node fails            32 TB raw remains
Data still to be held     the same as before
Utilisation after loss    1.5x what it was before

A cluster at 60% before the failure is at 90% after it, which is past nearfull_ratio — so backfill blocks and the cluster cannot heal. A cluster at 45% before is at 67% after and recovers normally.

That is where the 70% guideline comes from, and it is why it is a guideline about N+1 self-healing rather than about disk economics. A four-node cluster survives a node with less headroom than a three-node one, because the surviving three absorb the load rather than two.

Production considerations

Common mistakes

• Using consumer SSDs.
• Single network for everything.
• Forgetting the RAM cost of MON/MGR/OSD daemons.
• Trusting capacity math without accounting for replication overhead and nearfull headroom.

Key takeaways

• CPU: 1+ core per OSD; more for high-IOPS NVMe.
• RAM: 8 GB per OSD + 2 GB per MON/MGR.
• Network: separate public and cluster networks.
• Disks: enterprise SSD with PLP.

Knowledge check