Write amplification on EC pools

Advanced⏱ ~17 minceph

What you'll learn

Compute the OSD operations produced by an EC write
Distinguish full-stripe from partial-stripe writes
Explain the read-modify-write penalty precisely
Align application I/O to reduce amplification

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Capacity efficiency and write efficiency pull in opposite directions on EC. The overhead figure everyone quotes — 1.5× for 4+2 — describes bytes stored. The operation count is a different number, and for misaligned small writes it is dramatically worse.

Full-stripe writes

A stripe is k × stripe_unit bytes. Write exactly one aligned stripe:

4+2, stripe_unit 4 KiB → stripe = 16 KiB
client writes 16 KiB aligned:
  split into 4 × 4 KiB data chunks
  compute 2 × 4 KiB coding chunks
  write 6 chunks to 6 OSDs

Six OSD operations, 24 KiB written for 16 KiB of data. Byte amplification 1.5× — matching the capacity figure. Operation amplification 6×, against 3× for a size-3 replicated pool.

Partial-stripe writes

Now write 4 KiB into the middle of an existing stripe:

1. read the affected stripe's other data chunks (k−1 reads)
2. merge the new 4 KiB
3. recompute both coding chunks
4. write the modified data chunk and both coding chunks

That is 3 reads plus 3 writes across 6 OSDs for a 4 KiB client write — and every one of them is a network round trip to a different OSD, so the latency is bounded by the slowest of six rather than the slowest of three.

The same write on a size-3 replicated pool is three writes of 4 KiB and no reads.

The comparison

Client write	Replicated size 3	EC 4+2 aligned	EC 4+2 misaligned
OSD ops	3 writes	6 writes	3 reads + 3 writes
Bytes moved	3×	1.5×	~2× plus reads
OSDs touched	3	6	6
Latency bound	slowest of 3	slowest of 6	slowest of 6, two phases

Note the inversion: EC moves fewer bytes than replication but performs more operations against more devices. On throughput-bound workloads EC wins; on IOPS-bound or latency-bound workloads it loses.

Alignment as a tuning lever

ceph osd erasure-code-profile set ec42 k=4 m=2 stripe_unit=4096
ceph osd pool get ec-pool erasure_code_profile

For RBD, matching the image object size and the application’s block size to the stripe geometry converts partial writes into full-stripe writes. For RGW, large objects are already written sequentially and mostly align naturally.

Quiz

Knowledge check · 4 questions

Q1. A 4 KiB write lands in the middle of an existing stripe on a 4+2 EC pool with a 16 KiB stripe. What does the OSD layer do?

Writes the 4 KiB to the one OSD holding that chunk and marks the coding chunks stale, leaving the next deep scrub to recompute them Writes the full 4 KiB to all six OSDs, so any k of them can serve the stripe without decoding Rejects the write as misaligned and returns EINVAL, which is what allow_ec_overwrites suppresses Reads the rest of the stripe, recomputes both coding chunks, and writes the modified data chunk plus both coding chunks
Q2. An aligned full-stripe write on a 4+2 EC pool moves fewer bytes than the same write on a size-3 replicated pool.

True False
Q3. Explain unexpectedly poor EC performance under a real workload.

An EC 4+2 pool benchmarked at 2.1 GB/s sequential write during acceptance testing. In production, the same pool serves an application doing 8 KiB random writes and delivers a fraction of the expected IOPS with p99 latency five times the replicated pool it replaced.
Q4. Why can a partial-stripe write not update the coding chunks incrementally without reading?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Record the stripe geometry alongside the EC profile in the pool documentation, and state the application block sizes the pool is intended to serve. When a workload is proposed for an EC pool, ask for its write size distribution first — it is the single fact that predicts whether the pool will perform acceptably.

Cross-course references

Kubernetes: this is why database StatefulSets get a different StorageClass from bulk workloads
Linux: the RAID 5/6 read-modify-write penalty is the identical mechanism with the same alignment advice