Flash wear and write amplification

Advanced⏱ ~18 mincephnvme

What you'll learn

Explain write amplification in a Ceph context
Compute the multiplier for a given configuration
Estimate drive lifetime from the workload
Reduce amplification where possible

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A drive rated for 1 DWPD in a cluster writing 1 DWPD of client data will not last its rated life, because the client write is not the only write that reaches the NAND.

Where the multiplication happens

client writes 1 GiB
  × replication (size=3)              → 3 GiB across the cluster
  × BlueStore metadata (RocksDB)      → +10–30% depending on object size
  × RocksDB compaction                → +1–3× on the DB device
  × device-internal write amplification → ×1.1–4 depending on the drive
= NAND writes considerably above 3 GiB

Factor	Typical multiplier
Replication `size=3`	3.0
Erasure coding `k=4 m=2`	1.5 (but read-modify-write on partial writes)
BlueStore metadata	1.1–1.3
RocksDB compaction (DB device)	2–4 on that device specifically
Device internal	1.1–4

The RocksDB compaction term applies to the DB device rather than the data device, which is why a shared DB NVMe wears far faster than the data drives it serves.

Computing lifetime

# what the drive has absorbed
nvme smart-log /dev/nvme0n1 | grep -iE 'data_units_written|percentage_used'

data_units_written is in 1000×512-byte units, so:

nvme smart-log /dev/nvme0n1 -o json | python3 -c '
import sys,json; d=json.load(sys.stdin)
tb = d["data_units_written"] * 512 * 1000 / 1e12
print(f"{tb:.1f} TB written, {d["percentage_used"]}% of endurance used")'

drive rating: 1 DWPD × 3.84 TB × 5 years = 7008 TBW
observed:     1240 TB written in 11 months
projection:   1353 TB/year → 5.2 years   ← within rating

Estimating from the workload

client writes:  2 TB/day
size=3:         6 TB/day cluster-wide
across 24 OSDs: 250 GB/day per OSD
BlueStore ×1.2: 300 GB/day per OSD
device ×1.5:    450 GB/day per OSD to NAND
on a 3.84 TB drive: 0.12 DWPD    ← comfortably within a 1 DWPD rating

Doing this before purchase is what prevents buying read-intensive drives for a write-heavy pool.

Reducing amplification

Change	Effect
Erasure coding instead of replication	halves the cluster-level multiplier
Larger objects	reduces the metadata proportion
Avoiding partial-stripe writes on EC	avoids read-modify-write
A separate DB device	isolates compaction wear from the data drives
Larger `osd_memory_target`	fewer metadata reads, marginal on writes

ceph config get osd bluestore_min_alloc_size_ssd
ceph df detail    # compare STORED against USED

Quiz

Knowledge check · 4 questions

Q1. Why does a shared DB NVMe wear faster than the data drives it serves?

DB devices are normally specified as read-intensive drives, so the same bytes consume more of their rated endurance RocksDB compaction rewrites metadata repeatedly, and one device absorbs the metadata churn of every OSD it serves It serves every client read for the OSDs behind it, and read disturb consumes NAND endurance as writes do It holds a full copy of every object written to the OSDs it serves, so it absorbs their combined volume
Q2. Erasure coding always reduces flash wear compared with replication.

True False
Q3. Size drives for a write-heavy pool.

A new all-flash pool will absorb 2 TB/day of client writes at size=3 across 24 OSDs. The proposal specifies read-intensive 0.3 DWPD drives on cost grounds.
Q4. Name the factors that multiply a client write before it reaches the NAND.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Compute per-OSD DWPD from the client write rate, the durability multiplier, and device amplification before purchasing flash; the calculation takes minutes and the alternative is discovering the endurance shortfall in production. Rate shared DB devices separately — they absorb compaction wear from every OSD they serve.

Cross-course references

Kubernetes: etcd write amplification drives the same disk endurance planning
Linux: RAID5 write amplification on flash presents the identical concern