Replication — the cost of surviving a node failure

Foundation⏱ ~15 min

What you'll learn

Describe the three costs replication imposes
Trace the path of a replicated write and identify the latency contributors
Compare replication factors against their capacity and durability effects
Explain why replication cost scales with cluster write volume

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Replication is conceptually the simplest durability strategy: keep N copies, survive N-1 failures. Its costs are equally simple and are usually underestimated in exactly the same three places.

The three costs

Space. size 3 means usable capacity is one third of raw. A 300 TB raw cluster provides 100 TB usable, and less after reserving headroom for failure and recovery.

Write bandwidth. Every client write becomes size device writes and size - 1 network transfers. A cluster ingesting 1 GB/s of client data is moving 3 GB/s to devices and 2 GB/s across the cluster network.

Latency. A write is not acknowledged until the primary and the required replicas have committed. The client therefore waits for the slowest of them, not the average.

The write path

sequenceDiagram
  participant C as Client
  participant P as Primary OSD
  participant R1 as Replica 1
  participant R2 as Replica 2
  C->>P: write object
  P->>R1: replicate
  P->>R2: replicate
  R1-->>P: committed
  R2-->>P: committed
  P-->>C: ack

The client talks only to the primary. The primary fans out, waits for the replicas, then acknowledges. So client-observed latency is approximately:

network(client→primary)
+ max(primary commit, replica commits including their network hops)
+ network(primary→client)

Choosing the replication factor

size	Usable	Tolerates	Typical use
2	50%	1 failure, with risk	never for production data
3	33%	2 failures	the standard
4	25%	3 failures	very large clusters, critical data

size 2 deserves the blunt statement: with min_size 2 any single failure blocks writes, and with min_size 1 every degraded write goes to a single copy. Neither is acceptable for data you intend to keep. The apparent capacity saving is real and the risk is disproportionate.

Why replication cost scales with writes and not with capacity

Space cost is a one-time multiplier: three copies of a petabyte is three petabytes, regardless of how often it is read.

Bandwidth and latency costs are per-write and therefore scale with workload rather than data volume. Two clusters holding the same petabyte can have completely different replication costs: an archive written once and read occasionally pays the space cost and almost no bandwidth cost, while a VM fleet with continuous small writes pays the bandwidth cost continuously.

This is why the replication-versus-erasure-coding decision is made per pool rather than per cluster. An archive pool gains most from erasure coding’s space efficiency and is barely affected by its higher write cost; a VM pool is the reverse. Mixing both in one cluster with separate pools is the normal production answer, not a compromise.

Sizing from the workload

Start from the write rate, not the capacity:

device write bandwidth needed = client write rate x size
cluster network bandwidth     = client write rate x (size - 1)
usable capacity               = raw / size, minus headroom

Then check the devices and the network can supply those figures at the required latency. A design that satisfies capacity and fails bandwidth is common, and it presents as a cluster that fills up fine and gets slower as it does.

Quiz

Knowledge check · 4 questions

Q1. A cluster ingests 1 GB/s of client writes into a size 3 pool. What load does replication place on the cluster network?

1 GB/s, since the client sends the data once 2 GB/s, since the primary forwards a copy to each of two replicas 3 GB/s, matching the total device write volume It depends on the PG count and cannot be estimated from these figures
Q2. On a 200-OSD cluster with 4096 placement groups, one degraded device can raise p99 write latency across several percent of a pool while health stays OK.

True False
Q3. A cluster sized for 500 TB usable with size 3 meets its capacity target but write latency degrades as it fills. Diagnose the design.

1.5 PB raw across 90 HDD OSDs with BlueStore WAL and DB colocated on the spindles. Pool size 3, min_size 2. Design targeted 500 TB usable and reached it. Workload is VM disks with sustained small random writes, currently around 400 MB/s of client writes. Cluster network is a single 10 GbE per host. Latency was acceptable at 30% full and is poor at 70%.
Q4. Explain why the replication versus erasure coding decision belongs per pool rather than per cluster.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Size from the write rate first: device bandwidth scales with size, cluster network bandwidth with size - 1, and a design validated only against capacity will meet its capacity target and disappoint on latency. Keep size 3 as the production floor and treat size 2 as unsuitable for data you intend to keep, whichever min_size accompanies it. And make the replication decision per pool, because the costs scale with workload rather than with stored bytes.

Cross-course references

Ceph: Part XXIII (Replication) for the write path in operational depth.
Ceph: Part XXVII (Replication vs Erasure Coding) for the per-pool decision.
Ceph: Part LXVIII (OSD Latency) for finding the slow member of an acting set.