What replication adds to every RBD write

Intermediate⏱ ~16 mincephfio

What you'll learn

Compute the device and network cost of replication
Explain why write IOPS scale differently from read IOPS
Size a cluster against a write IOPS requirement
Assess the cost of raising size

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Cluster sizing calculations based on aggregate device IOPS routinely overstate write capability by the replication factor. The correction is simple arithmetic and it changes procurement decisions.

The device cost

A client write to a size-3 pool becomes three device writes:

client writes 1,000 IOPS
  → primary OSD:  1,000 device writes
  → replica 1:    1,000 device writes
  → replica 2:    1,000 device writes
  = 3,000 device IOPS consumed

Reads consume one device operation each, served by the primary.

Pool	Device IOPS per client write IOPS	Per client read IOPS
replicated size 2	2	1
replicated size 3	3	1
replicated size 4	4	1
EC 4+2	6 (one per chunk)	4 (k chunks)

Note the EC row: an EC write touches k+m devices, more than replication, even though it moves fewer bytes.

Sizing from a requirement

requirement: 50,000 client write IOPS, size 3
device IOPS needed: 150,000
per-device capability (SATA SSD, mixed): ~20,000
devices needed: at least 8, realistically 12+ for headroom

Doing this arithmetic before procurement is the difference between meeting a requirement and discovering the shortfall in production.

Measuring the ratio

# client-visible
ceph osd pool stats rbd-vms

# device-level, on an OSD host
iostat -x 5

The ratio of device write IOPS across the cluster to client write IOPS should approximate the pool’s size. A materially higher ratio means something else is consuming device I/O — recovery, scrubbing, compaction, or copy-on-write from snapshots.

The cost of raising size

ceph osd pool set rbd-vms size 4

Two costs, both immediate:

Capacity: every existing object gains a copy, so a large data movement begins
Write IOPS: every subsequent write consumes one more device operation, permanently reducing the pool’s write capability by a quarter

The second is often forgotten and is the one users feel.

Quiz

Knowledge check · 4 questions

Q1. A workload requires 30,000 client write IOPS on a size-3 pool. How many device write IOPS must the cluster provide?

30,000 90,000 60,000 10,000
Q2. An EC 4+2 write consumes more device operations than a size-3 replicated write, despite moving fewer bytes.

True False
Q3. Size a cluster for a stated write requirement.

A new deployment must sustain 80,000 write IOPS with 4 KiB blocks on a size-3 replicated pool. Candidate devices are SATA SSDs rated at 25,000 mixed IOPS each. A vendor has proposed 12 devices, citing 300,000 aggregate IOPS.
Q4. What immediate cost does raising a pool's size from 3 to 4 impose on write performance?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Apply the replication multiplier explicitly in any sizing calculation and show the working, since aggregate device IOPS is the figure vendors quote and it overstates write capability by the replication factor. State the write-IOPS capability to stakeholders rather than the aggregate, and include the utilisation ceiling in the number.

Cross-course references

Kubernetes: replicated stateful workloads multiply write load on the underlying storage identically
Linux: RAID 1 and RAID 10 write penalties follow the same arithmetic